Personally after getting legal advice about where you stand I agree with Mila.
Let him do the hard work if he wants a divorce. As you know a lot of MLC men threaten divorce as a way of being happy and even go through with it. If you do it it will be more justification for him staying away as you divorced him!
I am sure if he divorces you for unreasonable behaviour your legal advice may advise you go and counter claim the adultery.
With the recent history of better behaviour from your H he may just be swirling and ping ponging around trying to sort his life out and find himself happiness. Who knows what tricks OW may be pulling in the background with your H and she may also have an agenda.