Phoebe, it's hard when you're accused by the person you let get closer than anyone else. It's easy to blow off criticism from a stranger, but the one you've showed your true self to for years has unique access to your most vulnerable spots.
Today, I was accused (again) of alienating WH's kids from him. A very unfair accusation that we've been over before and SD said was not true at all. His ex actually worked very hard to alienate them, while I supported their relationship and was their caretaker for years.
I finally told him that maybe they were less tense now because they didn't have to keep his secret about his affair anymore.
I'm working very much the same way on the same things you are. I wish our WH's would do the same.