First of all, it's GREAT that you ended your A. If I were you, I'd use my "affair mode thinking" to try to figure out where your H is coming from. Use what you were thinking to 'justify' your A when you were in it, to try to make sense of what your H is using to justify his.
It's sadly comical that your H points the finger at you, conveniently forgetting that he has three pointing back at himself. Two 'wrongs' don't make a right, so what he's doing HAS NO JUSTIFICATION, no matter how he tries to 'spin it.'
As you've said, he remains an intregal part of your life. Spending holidays and family occassions with you. I think that's an excellent sign that he's kept the door to reconciliation open.
I hear ya about the 'sex' part...but having sex with him while he may or may not be having sex with her....doesn't make you the winner.
Have you had any conversations about reconciliation? It's been two years, so I don't think the subject shoud be considered taboo.
How's the honesty level between the two of you now? Can you talk about the past relationship that you two shared with each other without being judgemental or his getting defensive? Believe me, he knows as well as you do what went wrong in your M before either of you had affairs, and maybe it's time you both talked about that.