Either way - if you did something or didn't do something right - a normal healthy partner would have come to you to talk. The fact that he took other action tells you what has to change.
I say this to give you some perspective. Walking away or having an A or whatever, is not a normal response to marital problems. Work on building communication. Stop taking responsibility for what belongs to your H and just work on your responsibility to encourage, support and facilitate a healthy relationship.