This is not the situation that everyone else has. You are the walk away wife. He just needs to know that he can trust you. I don't think you push him away. You don't have to chase him so hard that you drive him away, but I would continue to accept invites and continue to get together. I don't see him pushing you away. If he was done with you he wouldn't be doing anything with you.
You cannot be lonely if you like the person you're alone with. Dr. Wayne Dyer