I do not have to deal with kid hand offs so please take this with a grain of salt. I do not think you did anything wrong - OK, the lying was not great but I do not see anything wrong with the boundary itself. He is going to have to learn that when the kids are his, they are his.
I also do not think that you should give up your DivorceCare classes especially if they helping you. You simply tell him that you have a commitment on those nights and he must bring the kids home at the agreed upon time. Again acting as a reality check for him.
And B, do not think for one moment that he thinks you are condoning his affair. You do not have to tell him. In fact, if you re-read the Rules, we are not supposed to discuss the OP at all. I just keep saying to myself that he is going to do what he is going to do. It does not matter to him how I feel about it.