“I need to be vulnerable and I just can't with him. I don't know if I'll ever be able to let him see my vulnerabilities.”
I am going to go out on a limb here, I would guess after 20+ years being married to you, and if I am doing the math right, 3 years post BD your H has a pretty good idea of your vulnerabilities( what can physically or emotionally hurt you) if that's what you meant. I get it, you've put up a wall, drawn a line in the sand so H can’t hurt you, so unless you let H past you will never know if he has learned/changed, done work on himself, too not repeat the past….
That is totally up to you, if you don’t D is most likely in the future. If you do, and H truly has changed you just might get an amazing M.
“I would rather love him and remain a family.”
Love is a choice.. You would need to choose to Love him again.
"Why can't I stand to be around him?”
IMHO, because of past hurts, grudges, resentment, looking back instead of forward.