He has not "let" you go he took himself out. In a backward crazy world it's a way to not leave the relationship. It's his way to find something out there (nowhere, somewhere) your never going to know.
When he calls back decide if you want to answer, but don't bring up the same conversation if you do. Keep it light, to the point and say I'm off so - good by!
If he ask to do what - vaguely say I'm working on my hobbies!
OR don't answer - but be prepaid, you got this!
The past can't be ahead of you in the future. You don't have to figure it all out, just pick a direction. What's next...I don't know but I can't wait!