I can see why you would decline the invitation, it looks like more of the same. You'd have to be detached from reality to think that a 'family day' would create any real movement in your situation, and that doesn't appear to be the case with you.
My guess, & it's just based on a feeling I can't pin down, is part of the reason he may not come around fully is because he doesn't want the OW to believe he wants to R with you. Even though he doesn't want her, it takes a strong person & direct person to come out with the truth & most people opt for the cowardly way out. So, he hands out just enough to keep you hanging in there & not enough of what it would take to move forward.
I may have forgotten & he did actually tell her he wanted to give your M another try. I may also be projecting my own situation onto yours.
As far as your part in this, it's difficult, if not impossible to figure out what came 1st. At the end of the day it probably doesn't matter much, it's like chasing your tail & only serves to make you crazy.
You are in a position to make whatever choice you want, or none at all right now. It's almost easier to have the choice made for you, isn't it? Almost all WAS's seem to have someone waiting in the wings to give them that push in deciding to leave, until they find out it's just a short term solution.
Which ever way you go, you have grown in leaps & bounds from the "love killer" you may have been at one time, into someone worthy of being shown incredible amounts of loving.