Your situation sounds a lot like mine....I was LD for most of the marriage and my H had a lot of anger once I became a sexual being again. Making matters worse, my H was never open sexually and didn't communicate his needs well...but he still wanted sex! My wanting him in that way made him feel desired...I never really understood this need until the shoe was on the other foot.
When we started counseling, the therapist pointed out to me that my H was one angry man. I never found any evidence of an affair, but like NOPs points out, it's a path that conflict-avoiders can take rather than deal with the issues at hand. I hope your H is not involved with someone...that is going to make it an uphill battle.
Everyone talks about having good communication, that it's all about communication, blah,blah, blah...but I have come to realize that it is true---you have to feel comfortable talking about this stuff, not once, but many, many times. Til this day, after all the work we have done, my H will tell me he can't deal when I am being "negative". We both have had such high expectations of the other...the counseling really helped us understand what a realistic marriage is all about.
I want to mention that in the counseling my H admitted to feeling jealous about my doting on the kids and not him...it's not just about sex, it's about affection as well. Also, my H was awkward about coming forward and coming up with ideas to make our sex life exciting: to me, it became routine and about him, and it lead to the LD. I am learning to take more responsiblity here and assert myself, and my H is now less self-absorbed.
I think the counseling is a great idea...I know my H was able to hear things from the therapist that he could not have otherwise.