Setting YOUR boundaries, which is FOR YOU... A boundary is something YOU can control.
You can't control him moving out, so it is not setting a boundary of telling him to leave if he won't get his head out of his butt.
What you can do is set a boundary that you will not tolerate any him speaking down at you or disrespecting you, therefore you will leave the conversation if he does that.