Someone once suggested on here taking ONE thing you used to do with/for him daily, and replace it with one thing you do FOR YOURSELF.
Yes, I would recommend you pull back on going to the gym with him, and when you DO go together, make it at YOUR initiation. If you're going to turn down his, don't say "I'm not up for it," as you did above, but maybe something like "Thanks, but I have plans."
