Hi Coly, I do not think it was cruel. You asked him to come over and he did. This weekend you can ask him to come over and pick up his stuff (and invite him for a visit if that is what you want to do).
To be cruel you need to have some preconceived or willful intention to want to cause harm, I really do not sense that from you.
I can understand you starting to doubt your actions and reasons. I think the really good chat and you missing him even more, and the people at work, stirred up some emotions, and that is the source of your doubt. Nothing wrong with that. Shows you care, IMO.
I do agree with OwnIt. You do not have to look at this as final or announce it as such. There is no reason for you to commit to that. The future is not written yet and very much unknown.
So, I see that you think he wants a social visit with you. However, do you want to have a social visit with him?
Feelings are fleeting. Be better, not bitter. Love the person, forgive the sin.