Sorry to hear of your troubles. I think you've come to the place (and we all come to it many times in the DB process) where you have to sit back, take some deep breaths, and decide how much is too much for you to take any more.
Your making the decision to pursue a D may force your H to have an "ah ha" moment and he may come to his senses. He also may get angry and see it as proof that he's right to have chosen the OW for now. You really have to think about what feels right to YOU and move accordingly. You have to decide whether you are ready to accept whatever outcome may result here.
I'm in a similar sitch, like we seem to get along fine and we can have some fun together, do the house logistics, and care for the boys but then when it's time for romance, love and all that other good stuff it's the OM that gets it and I've decided I've had enough. I told my W as much last night and I will move onto the D process now. I don't think it will change her mind about OM but I can't take the pain anymore and I don't want to wait for her to make up her mind anymore. She makes up her mind every day when she chooses him over me, when she arranges her work and life schedule so that they can have long weekend romantic getaways instead of spending her free time with me and our sons 4, & 6. I told her she's made her decision as far as I'm concerned and that I'm done.
It's not an ultimatum; if she chooses later to try to work on our R I can think about it. You can always make the same decision w/ your H. Divorce doesn't have to be forever; I mean aren't we all here because when we said forever and for better or worse we actually meant it?
So go with your gut, your heart and your mind (if you can get them to line up, I know mine hardly ever do). Do what will work best for you right now, your H is doing just fine taking care of himself. Keep coming here to vent and let us know how you're doing.