No, you are not crazy, and yes, your H is confused. I agree with Puppy in that your H is cake-eating.
Here's a quote from Dr. Shirley Glass' book "Not Just Friends":
"The worst solution is a stable triangle. When involved partners stay on an eternal fence, ultimatums given by the spouse or lover move them from one side to the other. They cajole, seduce, and deceive both partners in order to have their cake and eat it too. The spouse and the lover help to maintain the stable triangle by making compromises and accepting whatever crumbs are thrown their way."
It is up to you, gsr, as to what you will and will not tolerate in your R. Everyone has boundaries. What are yours?
If you are ok with letting your H continue his A, then so be it. Do not focus on the A. Focus on you. Detach, GAL, and make the necessary changes in YOU that need to be made. Be the person your H fell in love with, if not better than that.
If you are not ok with the A, then you must establish your boundaries, make your H aware of what they are, and above all, as hard as it will be, you MUST stick to them. If you don't take your own limits seriously, then how can you expect your H to? It's a surefire way to becoming a doormat.
Explain that you are not trying to control him. You are not giving him ultimatums, although that is how he could possibly perceive them. Let H know that these are simply your own personal limits, boundaries, dealbreakers.
Take care of you.
Getting over a painful experience is much like crossing monkey bars. You have to let go at some point in order to move forward. ~ Joseph Campbell