Glam this is just a thought so please ignore if you have already done this or you feel it would not be right for you but along the lines of 180's. Have you tried not asking if he is coming over or if he asks if you would like him too instead of always saying "of course" maybe say you have plans or he is welcome to come when you get back. I know this would involve some planning as you have young children so maybe you could arrange for friends to come over and just say oh xyz will be here but you are welcome to join us.
As I said just a thought that maybe doing 180 would just make H realise that your life is no longer revolving around his and you are doing other stuff at times. What a shame you have to start your working week on a Sunday. I guess we all have to be thankful we do have a job to go to whatever the day. Enjoy the week end.
