No apologies necessary. And I still think you need to be more firm regarding his visits. Don't tell him he can't visit because you are busy, because you schedule activities for T/TH. Simply state that his visit days are MWF, so he can visit on those days. Period, no explanations or apologies.
"You can't stop the waves, but you can learn to surf." Jon Kabat-Zinn
