You can do what you want, but I get the impression it's not about wanting to come there to see you. It's more about keeping you right where he left you and dangling that old carrot by showing just a wee bit of interest so that you won't rock his world by setting boundaries and making demands on him or moving on w/your life.
If I were in your shoes, I'd drop your son off on Wednesday afternoons and this allows your space to be yours. The longer you allow him to come in and visit w/you while waiting on your son, the harder it will be for you to set a boundary. You are the one that has to initiate boundaries. The man doesn't get it and when he comes over, he can then say to others "gee, wife and I are the best of friends and we visit and chat up a storm". Don't be his Plan B unless that is where you want to be.
You need to remember that he fired you as his wife, companion and lover and you are most definitely not his next door neighbor, i.e., whereby he can drop in for a nice chat up on a weekly basis.
What you posted to your h about being willing to drop your son off at his place was actually giving him the option of either you doing what you suggested or him still coming to the house. Of course, he's going to want to come to the house to check to see if there are any changes in the house and to see if you've changed anything about yourself. He wants to make sure you are still there waiting and available for him. Of course, he's never going to miss home or you if he gets to come by there to pick up his son because he's still able to get his "feel good" kibbles each and every time he comes there. He can't miss something if it's always available to him.
If I were in your shoes, I would consider dropping your son off at his father's place periodically. Does your son have a key to his father's place? If not, dear old dad should make arrangements to provide his son w/a key. Your son can also be ready to go and meet his father at the door or even in the driveway when he pulls up. Your h doesn't necessarily need to come in the house.
If you give him an option, he will always go w/the one that gives him the most benefit. If you give him an inch, he'll take a mile. They do like to test boundaries to see if we will waffle and allow them to do what they want at the end of the day.
At the end of the day, you will need to decide what you want to do. If you decide to set a boundary, you will need to adhere to the boundary and not give in. If you give in, he'll know that you are a push over and will ignore other boundaries that you may set in the future.
