Is it a punishment? I think it is what you decide it is.
He is telling you he is uncomfortable with you there. now you can choose what to do about that. you can
look at it as punishment. he is punishing you. It is all negative. you don't dessrve this. you should flee this. escape.
conclude he is immature, unable to handle reality. Look at him as an incompetent wimp. He is spineless, weak-kneed, he is not a real man if he has to direct things this way. he is not worth having. why would you want this jerk?
decide that he is asking you to do something, and you can choose to honor the request or not. it is neither punishment of you, nor an indictment of him. it is a request. you can honor it and see where it goes, or you can rejectit and see how that works.
These are just some of the options I see - ways to see the situation based on what I read. maybe there are other options too. The point is, it is up to you.
