Could it possibly be that then he felt he had to earn it from you? Is that so bad? We already know that he has a history of denying himself....it's not such a stretch to me to think that he may also think he has to earn it. When you turned your back on him...that might have been the switch to....ah, now I have to chase her a bit.
