My recommendation is for you to pray about this and ask the Lord to control your tongue the next time that he says something like this. Ask him what it is that he wants you to say. You maybe very surprised as to what is going to come out of your mouth the next time.
You won't have to say it in anger, but you will have to stand firm and let it out. When this happened to me, I swear that I felt someone was behind me and took their hands and pushed me right in front of my husband and made me confront him. Everyone kept telling me that I needed too, but I kept saying it wasn't the right time, and at that time it wasn't; but when it was, I was shoved and words came out of my mouth that I did not expect to come out of my mouth. They were not mean and they did not cut him down, but they were things that he needed to hear.
You could be getting to the point where you are being told that it is time to set some more boundaries. One of those boundaries could be about control and him holding this over your head all the time. This seems to be an ongoing thing with him and it needs to stop.
Ask the Lord to control your emotions so that anger is not involved in this, but that he takes control over your mouth and if confronting him is what he wants you to do, then to please show you. You will know exactly what you are supposed to do, just listen and wait for his answer.
