If you delete a card, every viewer who had it loses it, and it can't be recovered.
If you only want to take a card out for a while, hide it instead: go to Manage cards, select the card and click Hide. A hidden card stops coming out of packs and can't be redeemed either, but viewers keep it, and when you publish it again they can redeem it.
If you're going to remove several cards your viewers own, let them know on stream beforehand. A discount coupon is a good way to make up for it.