I didn't think you would (agree), but I felt compelled to speak my mind. What you're doing is a recipe for "limbo," and he's absolutely fence-sitting. You seem okay with that, and that's fine -- different people move at different speeds. When you're NOT okay with it, then it will be time to try something different.
I personally could never abide continued contact of my wife with another man, and "only" 2-3x/year visits. I value myself more than that.
You cannot "control" the situation -- only what YOU are willing to (or not willing to) put up with. Boundaries.
