You are right, I wouldn't invite my babysitter. I will go by myself and enjoy yummy BBQ with coworkers.
As for cooking, you don't think it will be too rude not to offer if H is there? I cook for my kids and myself, not especially for him. I'm a little confused about "neighboury" attitude. I should be polite and not ignore him but not too overly friendly is what I understand...
