But I'd like him to know I am here and do care - that I backed off to work on myself (and probably to avoid being hurt more, but I can't tell him that)
The problem with this is that he more than likely sees this as pressure. Sometimes the sound of our voice can be pressure. Best to speak with ACTIONS not Words. Just like you believe nothing he says and 1/2 of what he does. Listen to his ACTIONS not his words. So do your actions show him that you care?
Enjoy your small group tonight. Keep trying to find new activities to expand your world.