Tell him you won't live in an open marriage. Don't let him use your house to get ready for activities with his OW. He's cake eating big time, and your putting the slices on his plate. Stand up for yourself.
And the arguments? You're trying to get a reaction, any reaction. It's passive aggressive; you want some sort of attention from him, and arguing is the easiest way to get it. But it's not going to work the way you think.