You have to be able to live in the only real time that there ever is: NOW. The past and the future do not ever actually exist. The past is filtered memories, the future is hopeful or fearful fantasy, but the only time you can DO anything is NOW, and the trick to letting go is that you have to be aware of who you are when your ego is silent, when the demons of your unconscious mind have been put to rest, and when you are just living in the NOW, when you are just "being".
Over this past year as I've finally started really letting go, I have felt this very thing happening, more and more. I'm happy just hanging by myself sometimes, I don't fear the future, and I don't beat the dead horse of the past any more. It's a good thing.