Annette, I think your H has a lot going on, underneath the surface, regarding the ED. Shame, disappointment, sadness, inadequacy, longing to be normal.
I would look into counseling to deal with it. Obviously he feels he can't talk to you about it, openly, (as many men can't) so maybe a trained sex counselor could help. Has it ever been checked to determine whether it's physical or psychological?
I think you should set up the appt and tell him he can come if he wants. Remind him of his statement of wanting a better M, from a year and a half ago. Then GO. Consider it part of your GAL.