You know, I've started to write posts to you any number of times, but I stop not sure exactly what to say. It's difficult because, given the information you've provided (or even only the information that you know of), your sitch is a bit different than most others. Thus, it's hard to draw experience from others and apply it to yours... In any case...
I've often thought that your H had an EA with that woman way back when, but never "did" anything about it, and thus, has caused him to question things. With many of our Ses, we know this junk is process...a process they have to work through and come out the other side. How long does this process take? Well, seems to me, for the most part, it's about a year, give or take a month or two. You are approaching this "milestone."
You've been working at it a long time. Perhaps your H has become "content" to just remain in this stasis because, after a year, it's what he knows. Perhaps you are fearful of telling him how you feel because you want to give him space and don't want to push him away. It seems this fear has placed you in a sort of stasis as well. Maybe you guys are just stuck and one of you needs to kick the other in the butt...maybe???
There has to come a point in everyone's situation where we have to make some sort of stand or statement...tell them how we feel. We also know that to really work on an R requires spending quality time with each other rebuilding that R. Oddly enough, our Ses think this should come naturally...that their feelings should come "back, full-force, without having to work on it...with us... Many times our Ses need to be educated about this dynamic, but they will only listen when they are "ready."
Are you ready to make this stand, Sue? Or are you willing to give him more time to sort through things? Yes, this is a question only you can answer, but you also need to take care of yourself and your needs. Sue is no less important than H.
Sorry I couldn't be more help. Take care of yourself.