I think any calculated response to his comments about your weight will show attachment.
That’s probably why he said it. He knows it’s a powerful bread crumb.
Personally, I think you should just turn around and walk straight out the room when he says it.
Give zero indications that you’ll accept any hooks, no matter how powerful. A spiteful comment that others have noticed - you might think this is a good idea, but it’s still taking his bait.