I silently agree when H says "Oh you want this divorce too" but I always want to add "Yes, you got me to this place."
That's actually remarkable that your H left himself open like that.
My W would never concede such a point - she'd either try to turn it around the other way, saying I forced her into wanting a divorce, or just that I was always really in favor of divorce anyway, and any word I might have said to the contrary was just more of me "being difficult".