Your feelings are totally NORMAL!!! Just ask anyone here who's gone through it!!!
Now my perspective is necessarily different as CJ never left the house physically.
What I CAN tell you is that you CAN indeed work through these issues of trust, safety, etc. WHILE living together. Perhaps a good idea is to discuss not having TOO high hopes for how the reunion will be.
For example you might fear that "letting him come home" might mean to HIM that you're "over it" and hence you migth feel constrained about bringing up these very issues that still need resolving.
I know that for the longest time, the trust thing was VERY iffy...and in my case rightly so. ...I think my biggest breakthrough was in really KNOWING that I could at least trust MYSELF to be okay not matter what CJ did.
The trust and forgiveness often come in bits and pieces, it's a process, not an event. IMO of course.
Your anxieties are totally understandable, even, I hate to say it, predictable (based on what I've read here).
But decisions based in fear.....hmmmm....what if your worst fears about the reunion NEVER come to pass...how would you feel in 40 years if you passed on this chance???