You told him exactly what needed to be said. And frankly, when it is your day to have her, he doesn't have any right to say whether you have a guy around her or not (although, to be fair, I don't think kids should meet the people their parents are dating until they have been dating at least six months and it is clear it is going to be a stable long-term relationship. It's not fair to the kids to get attached to people who come and go.)