Don't worry Deejay your feelings and fears are completely normal. Many people here struggle with the same thing (myself included. I am in the same boat and have lately been grappling with that whole "second best" thing and thinking that he's only with me because OW dumped him... even this morning I confided in him that feeling of being foolish for staying!).
My point here is I think it's very normal. Fortunately there are some wonderful and amazing people here to help us work though all this, share what's helping them and help normalize our perspectives and really get to a point where we can build it back better than before (and build ourselves into stronger and better people).
By the way, a great book I've been going through a second time right now is "Not, Just Friends." I think the author's name is something like Shirley Glass??? One thing is says is healing and rebuilding after an affair is a process that generally takes anywhere from 1-2 years to work through. You still have quite a bit more time. (And for sensitive people like myself I'd guess it could take even a little longer ).
Oh well... one positive angle is that hardship and struggle help us grow and gain strength.
There is no arriving, ever. It is all a continual becoming.
). 