In all fairness FY, what you say is absolutely correct.

We forget that people aren't possessions...

They never WERE or ARE ours.

I see that phrase more as a suggestion that people choose who they want to be with and so by accepting that people aren't possessions and that they choose to be with us, then we are simply allowing that others have free will and if they come back, it means they are asserting their desire to be with us, by choice.