I FEEL for you!! I have distant memories of when H was living at home. I remember times of him doing nothing but watch TV and I would feel like if he can do it then so can I. It was hard to get things done when H was around doing nothing. I find myself much more productive and doing what I want and need to do when H is not around.
Anyway, why don't you tell H that you need some time away from the kids and ask him to come up with a day and time every week that you can have some LL time. This puts the ball in his court without being controlling.