I think I'll stick to the dinner as a family when he comes over for awhile.
He did also complain about not having time for him as in not visiting him at his job, bringing him lunch...that's why I didn't think of that as pursuing, just thoughtful. He does work fairly close to me and if I present it in a way that I am just being thoughtful then I wouldn't think of it as pursuing, just thoughtful.
I used to work out during my lunch hour and if he called and wanted to go to lunch or to bring him lunch I would tell him I was busy or didn't want to take the time either, I wanted to exercise, that was more important