What came across loud and clear was a *boundary*. (And you DO have proof in that you KNOW he has talked to the coworker on messenger because she popped up on his recents list, yes?)
No, don't worry about coming up with statements to make. Everything you say should be genuine and timely. Just own your power. Set firm and fair boundaries. Otherwise, be cordial.
Instead of saying things like "it isn't worth my time," consider rephrasing to leave all the "victim-y" stuff out. Say, instead, "I won't": "I won't go to joint counseling until I'm sure you're not planning to use it as justification for your bad behavior. When you want to work on our M with me, let me know. Then I might have a change of heart." Something like that.
If your H wants to go with you and the boys this weekend, don't stop him! You can even say: "The boys and I are going to x at 2 p.m. If you'd like to go, just be ready by 1:30." Something like that. And don't bring it up again. Don't ask him if he's going. He's a big boy and you are his wife, not his momma. He can keep a schedule. If he isn't ready or available when it's time to leave, do not text him or call him or ask him a thing. Don't say goodbye if you don't want to. Just pack up and - #byefelicia - scoot on down the road!
If he's ready, slap on your game face, sister. These are opportunities to show him that you are accepting whatever happens and you're moving right on!
M: 40 H: 44 Married 14 years S11 & D6; D20 & D19 from previous M 2BDs/PAs, 8 years apart Piecing: April 2014