We all have a lot of fear, Alison. In your case you have a lot of power to work out what you will accept and what you won't accept if he asks to come back. For starters you don't have to accept him coming back at all. And if you do accept him coming back you can have certain boundaries. But it sounds like you need to move super slowly if he does come back. Like months and months and months. Is he in IC? That could be a condition. You could have certain milestones, like dating once a week for a couple of months, and if that goes well then move forward to the next stage. It's not all or nothing, it's both of you working hard on yourselves and on your relationship to make yourselves and your marriage the best it can be.
I've been thinking a lot about what me moving forward with my dh would look like (which sometimes I think is me being delusional given his signals, though they are mixed! 'Let's spend the weekend together! No you can't see my flat!'), and I've come to the conclusion that if we R things will have to CREEP forwards in tiny stages...Because it took decades for things to go wrong in our marriage, so expecting things to turn around in a short time will not end well. Behaviours take time and effort to change, to wear new healthy grooves in your brain instead of the old destructive ruts.