According to Dr. Laura (you can take this info with a grain of salt if needed ) men view sports as an outlet for their competitive nature...they need to excel at something. Apparently your h gets a big ego boost from winning at VB. His job is not providing him with the necessary boost to his ego or winning attitude. I seriously do not think it is about the OW as a reason for him playing.
He wants you...otherwise why come back? He loves you. He chose you over OW.
IMHO, you should not make this VB an issue about OW. It's just your h's competitive nature...you mention that this has not changed since his single days, right? Even the A with OW may have been his attempt to feed his ego not cause he didn't love you.
He wants to play and does not want you to make it an issue...so don't. What can you do to support him? Maybe turning the tables and being supportive will make him do it less? Or maybe you can join something and have him watch the kids one night?
I'd say do the opposite of what you normally do when faced with this and see if you can't turn this around to your advantage.
I know it is hard...I face the same thing with my h and his constant going ons with single friends. It's hard and I understand your discouragement...keep going forward with your m...cause you know the alternative (divorce) is not the solution (or at least poses more drawbacks).