Wow, that is tough. The channel changing is so passive agressive. He is trying to bait you. I think the best thing to do is keep it easy breezy and don't give him a reaction. Either something like, "oh,thanks! I was dying to watch this but I was too lazy to change the channel." Or what about just getting up and smiling and inviting the kids to have some ice cream with you, or something? Or go out yourself -- with a smile as if that had been your plan all along.
It will frustrate and confuse him when he gets the opposite reaction he's expecting...and make you feel empowered.