I don't think you need to meet. The whole point of mediation is to have a supervised conversation to keep things fair and above board, and to have someone who is objective. They can suggest alternatives to sticking points. My thought would be that any meeting is going to cause an argument - it is not going to be a fun conversation.
If you are serious, I would stand by your statement. Personally, I think you have him on the run, and honestly, why would you do him any favors?
I don't suggest being unkind, but LBS get beat up so many times because they give in.
Just my opinion - you know what your heart is capable of, but to me, it's funny that H drops you, but then comes warbling back when the grass isn't quite as green on the other side.
I know how it feels to be scared you'll commit the "final push" to make them go away, but if something like this does that - especially with him not bothering to contact you in so long, then he was never really there.