I don't think that you come accross as controlling or harsh at all. But I would ask one of the guys to chime in to see how it sounds from a man's perspective...
How about
I am happy that you want to spend time with S____.S ______also misses you and will feel devastated if you only see S _______. Then continue as you set out above.
Just a thought. Let's see how the others feel.....
