I would use this as an opportunity to educate him and let him experience the consequences of his actions long-term, rather than shutting him down.
I would just say given that he has been away much of her life and she is very young, that you would prefer that the initial solo visits with him be shorter and nearby in case there are issues and that as she ages and you see how she handles things, that you could discuss lengthening those times.
