You most certainly don't have to engage in R talk if you don't want to. If he says "stupid things," say "Now you're just being stupid. Goodnight." and roll over and go to sleep. If he says HURTFUL things, say "You're being hurtful -- please stop it. Goodnight," and roll over and go to sleep.
You cannot engage someone who won't engage.
If he tries to initiate intimacy, you can refuse him, and you don't need to give a reason. He doesn't deserve one. "Because I don't want to," or "Because I choose not to" should suffice, but you could always add "Because I cannot be intimate with someone who is being intimate with someone outside of my marriage, that's why, and I can't believe you would even have to question that. Goodnight."
I know this is hard, but until you refuse all R talks, and end being intimate with him, you're allowing him to cake-eat, and only adding to your own frustration.
