In my defense, I've told him over and over again that I wish for him to be happy. I have never and will never ask him to leave her for me. If he needs to work on his relationship with his girlfriend so that the can stay together, I will not stop him - I will support his choice 100%.
Do you realize that this "belief" is usually included in most affairs? In fact, many wayward spouses will tell the betrayed spouse how much their affair partner wishes them and their marriage well - IF that's what they want.
You're avoiding making the appropriate choice by throwing it back onto him. And in doing so you are making the choice to keep the affair going. And you're couching it in "I won't stop you", but you certainly aren't helping either.
Remove yourself entirely. Give the mother of his child a fighting chance.