I find it very doubtful that OW came into the picture only after you gave him space. You probably only discovered something that was there before. It's unusual for guys not to line up another woman before committing to leaving (or expressing the desire to leave), unless you were miserable to be with (which I doubt since otherwise he wouldn't come around now).
You cannot be lonely if you like the person you're alone with. Dr. Wayne Dyer