Instead of focusing on this past visit, you and H could discuss how things could be better for him on the next go-around. Myabe he would feel more secure if you both planned some couples time, if only a few hrs, just to reconnect. H and I have each felt abandoned by the other when around family...it's best to come up with a preventative plan.