Just saw your bat signal and saw that many good folks have already swung by here talking you off the ledge. Yeah, they all are correct in their view of your true underlying reason for saying "no"....however painfully difficult it may have been for you.
I say let them have the kids as LONG as there's no real danger in any of the activities they partake in together.
Keep going...one foot in front of the other...eyes straight ahead.
