Stay strong, it is your home and you DO have the final say on who visits. His behaviour is outrageous, don't tolerate it and don't argue with him, if he challenges you again repeat your position and say you will not continue discussing it. I get the feeling he has acted like a bully for a long time, this is a great chance for you to be assertive now because you are going to need these skills in the future!
