Advice Needed- setting boundaries

[Sam3]

Here's a backstory his grandmother & my grandmothers sister live in the same old people apartment place.

My grandma had already called me today to tell me the two 80 yr old ladies got into in argument about us.

His grandma was bragging about that he brought our 2 older kids over on Sunday to see her. And that husband comes home every weekend & I just take off with The baby. And that i never was very friendly. He hasn't been happy for years.
My great aunt tells her "well sam is quiet but she's a very good mother. And maybe husband ran off with some other
woman." (I'm not sure if she really knows or is just assuming)

So his grandma called him today.

He asks to speak to me tonight after the kids talked to him & he's mad.
He tells me "I don't know what your telling your family but my kids are the most important thing to me & I don't need them hearing this story. I didnt run off with some other woman. I have told you I'm not in a relationship. He says i told my grandma one day the truth will come out. Your freaking grandma thinks I'm having an affair. You need to patch things up with them."

Seriously? I need to patch things up?

Why is that my responsibility I'm not the one who had a man staying in a hotel room with me & sending me text messages.

I feel like He can deal with the fallout. Why is it my problem? And like I have control over 2 80 yr old ladies arguing.
My grandmother does know about OW. Because we live in a very small town & they live about 5 minutes away. I have taken the baby
over to her house when I've been trying to stay away from the house while he's here with big kids.

Do I validate his concerns?
Do I let him deal with the fallout & tell him it's not my side of the street to clean?