I understand your situation about trying to address opposite prior complaints in marriage. If one of the complaints about a period in the relationship was not feeling cared for, it is difficult to feel like it's a great idea to stop showing affection now.
