The problem is your H like most men interprets your attitude towards the marriage and your relations with him through sex

If you have sex, he will think everything's ok... you are ok with him doing what he's doing and all is well

if you dont', that sends a clear message to him that something's wrong...

Yes, i hate to admit it, but men do think with thier penis.