lc4, I consider my advice crack advice, so I guess you get what you pay for.
I think it's good to say no to those things you don't enjoy and kind of weed those things out, so to speak. That allows you to focus on what's really important.
I was just thinking about this earlier tonight. Do you have ability to just go away by yourself for a weekend or week? Maybe to decompress and really get in touch with what some of those personal goals may be?
Perhaps for some of the evenings you could use those as an opportunity, too, to just get out of the house. Your situation is a little atypical in terms of a WAS as you would not usually have to be concerned too much about them contact you too frequently or being around too much. I suggest that you just play it by ear for now, make adjustments in the form of baby steps, just depending on what direction you see fit. It sounds like you're doing a pretty good job of not being overly eager or anything. Maybe over time you can ease into some more boundaries, like having him plan to come over, for example i.e. must making a little less easy for him to come and go as he pleases.
