It's Father's Day, he wants to spend times with his kids... I think you should make that available. It's your parents house, their rules, but I'd let them if he can't spend time there with the kids then it will be just you coming over, without the kids.
Spending times with your/his kids is never cake eating.
M - 9 1/2 years 5/5/16 - Bomb drop - 3 week EA 10/31/16 - We sold house 01/10/18 - D Finalized