Should I just confront him and say I am done or shall I say end the affair or I am going to let the cat out of the bag
No one can give you that answer. It is a personal decision that you alone have to make. Do what is right for you.
That said, whatever you do decide, no matter the outcome, no matter what H's reaction may be, YOU MUST ENFORCE YOUR BOUNDARIES BY FOLLOWING THROUGH. He WILL NOT take you seriously if you don't take serious ACTION.
If you tell H you are done, then be DONE.
If you tell him to end the A, and he doesn't, follow through. "Out" his lies and the A.
Prepare yourself, and realize that it may get worse before it gets any better.
Take care.
Getting over a painful experience is much like crossing monkey bars. You have to let go at some point in order to move forward. ~ Joseph Campbell