His statement tells a lot. You have a much more profound influence over him than you give yourself credit for. He doesn't show his worry and concerns in the way you want to see it, but it certainly seems that he is feels them and it bothers him.
I am going to agree with Corri that you should convince him to go, but for a slightly different reason. Assure him, as Corri suggests. Then if you get the slightest hint of an improper or controversial statement concerning your H from any of your family members, I think you need to directly confront them on it. Tell them you do not appreciate their statements, you are married to your H and intend to stay that way, and that you will not allow them to come between you two any more. I strongly suggest you do this in front of your H. Leave no uncertainty about where your loyalties lie. This could go a long way toward repairing the bond with him.
