When I asked him about it, he said that he did it on purpose to piss me off so that I would leave him.
My H did a lot of this kind of stuff. I see it as "its not my fault" thinking. In fact pretty much all I hear is that. Lets face it, people having affairs (including OW) have to in some way excuse or justify their actions or they have to see themselves as the person responsible.
Since they are not taking responsibility for anything right now, the best they can do is try to make it your fault or your decision. My H did this to me, he pushed my buttons until I said get out. He knew my buttons and used them. I was in no shape then to see what was going on. Now I do. Best you can do is hide those buttons or don't let him push them...don't react.