Sounds like things are comfortable between the two of you. I think your patience is inspiring! You have the best of both worlds right now, space for you and your H has his space and the connection of family. It's like you two are dating.
Just curious what is your H like? What kind of personality? What are his hobbies? Has he changed any since he left?
Sorry for all the questions, but I like to get a feel for the H's.