You are very wise indeed to have 2nd thoughts.
over the past few years you have grown and matured and made the changes you needed to make.
just because you love someone and forgive them does not mean you have to allow them back into your life again.
take things very slowly and set your boundaries.
You have made a new life for yourself so don't be motivated by guilt or fear.
If it is meant to be it will happen in your own time and when you are ready.
blessings
YBR