I just don't know what to say. I am so sorry. This is totally alien to me. He seems like he's lost it.
I don't believe he was asking you along just for nanny services. That's awful to think, and really, it would have been easier to just leave you both at home. I still want to believe it was because he likes your company.
What is your gut instinct? For me, it is almost always right.