He sounds like he's still in a sense "sneaking around", even if he isn't good at it. He probably is just looking for a young piece and you are letting him work on that by pretending that there is no problem. He might break this whole thing off himself if you tell him you know, show some outrage, and indicate that you will NOT tolerate it (you decide on consequences for yourself).
You cannot be lonely if you like the person you're alone with. Dr. Wayne Dyer