I'm probably not the best person to be giving advice right now, but I can offer understanding.
I don't think it is easy for our Hs to just walk away. If it was, they'd be gone and never look back. Of course that doesn't make things any easier. It probably makes things harder for us--to see them and know they don't want to be part of our lives.
I know just how you feel about wanting to do things with your H during the summer. It makes me sad and angry to think of all the fun things we could be doing together if only my H would get his head out of his a@$.