I agree w/asitis on setting a schedule for visitation. I would encourage your h to take your son out for the day. A day in the park, museum, etc. Your home shouldn't be the community center for meet ups all of the time. This will change if you should divorce. He should be responsible for your son, which means, taking him to his place and spending time w/him.
Setting visitation times allows you time to do the thing that you want to do while your son is w/his father. It gives you some time to yourself. After all, your h has plenty of time to himself during the week while you have your son 24/7. It's time your h learn what being a father is and that means taking care of his son on his own in his own environment or finding ways to entertain his son.
Your h may get a bit miffed w/the changes in visitation settings, but that's too bad. He's the one that walked and he's the one that needs to figure things out, especially if a divorce should happen. One you set your boundaries, do not change them unless you think they are unreasonable. Don't waiver/waffle on them. If you waffle, he'll known he can get by w/stuff. Stick to them.
