Initially I was in the 'No' camp but Ginger1 makes a good and considered point.
Thinking about it a little more, by saying 'no' you are trying to control your H and his relationship with the kids. If there are no safety concerns why not give it a trial? This might be an opportunity for your H to learn something... e.g. he still lives too far away!
Also if this trial works out, there could be a chance to have some more time for 'you', providing you GAL it wisely.
Me 50, ExW 49 T21, M13+ S15, S13 BD #1: 25-Jan-2016 (EA confirmed & ILYBINILWY) Sept-2016 Mediated Sep. starts Oct-2016 W petitions for D Jan-2017 R w OM admitted/confirmed Jun-2018 D'd