Is the frustration because you both have different parenting styles? Is there anyway for both of you to accept that you each have your own unique thing to offer your children as parents?

Too bad the mood was not right with you when he tried to express a little bit of affection.