You see how controlling that was? He's packing and leaving unless you give up your wants and needs? Unless you shoulder responsibility for all that's wrong in the relationship? You sacrificed in order for him to pursue his athletic dreams and what you have to show for it is three kids and no assets.
Look at the things he wasn't. He wasn't the guy who wanted to rush to marry you and create a stable home for his family. He wasn't the guy who put aside his dreams to be a good family man who put his family's future first. I'm willing to bet you did the majority of compromising and accommodating in this relationship.
You didn't push him out. You stopped letting him play his controlling game.
It may be worth knowing who the OW is IF you can control yourself. Just so you know who she is in future interactions.
You could tell your H he's free to go, but in leaving his family and his loving partner he's leaving the best thing he ever had, just for a piece of a**. And that you hope he's at peace with the fact that another man will likely end up raising his children.