Kimmerz, Please stop beating yourself up over the comment he made. It is projection at its finest. He is the one that didn't come forward and talk to you about what he was thinking and feeling. This is the PA behavior and yes, they are going to be alone because they have chosen not to expose their true selves to people because of the "fear" rejection, etc.
You are not a mind reader! You did the best that you could over what you had control of. His projection and his warped perception are going to drive you insane.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.