When a spouse leaves for another person they most always are leaving in the fantasy of it all. More often than not, the OP is "less" than their spouse (less good looking, less work ethic, less emotional stability, etc.). Realization always kicks in, even if they never admit it. In fact, I have seen them work all the rest of their lives trying to justify and "make better" their decision in whatever way they can.

I think being happy is the best thing for everyone.

You have to do what you think is right. Only you know your spouse and what has worked. But, if your spouse doesn't have you to fight with/argue with/confide in, etc. he may realize that he doesn't have to justify anything and can just feel what he feels. Don't underestimate how he feels about you.