I totally understand the benefit the kids get from your X visiting. However, I should have clarified - do you think your X wife will be visiting if and when you have a new love of your life living with you?
My plan was to allow my X to come in to visit while I am still single, but after this last weekend of not respecting me when she was invited into my home, I am seriously considering putting a stop to her visits once the D is final. I am walking on eggshells right now and dont want to rock the boat, but I need to make my personal life more private from my X. When the kids are with me they follow my rules and not W's rules. I feel it best that W not witness how the kids and I operate at home as she then wants to exert too much control to fit her method.
Your situation is a bit more like my fathers in that the kids live with you pretty much full time. My father never allowed our mother to visit - only to pick us up.