Your GF is having an affair with a married man, so I don't think most people (that I know anyway) would be thinking badly of you. If you GAL, work on yourself, I think most people will tell you stuff like they tell me: my H is crazy, an idiot, going through a MLC, etc.
I think not being there might be best for you, but if your stuff is also there, would she possibly take some of it? When my H moved out he took some things that were not solely his (mutually owned stuff)without asking me and you never know maybe she would take something of yours? If you're not worried about that, then I think it would be fine not to be there of course.
It sounds like a lot of your problems are when your GF is drinking (maybe she has a drinking problem?). Maybe you should try not to be around her when she is drinking as she sounds pretty unpleasant (from what you describe) while drinking. That isn't good for your self confidence or PMA. Also, at some point in my R, my H and I started a "6 month rule" that we could only bring up problems in our R for 6 months and then we couldn't bring it up after the 6 months. In a long-term R, after 10 or 15 years if you didn't do something like that, we could be bringing up bad stuff we had done around the clock probably!!! Karen
Karen