If you just stay quiet and don't communicate about anything to him he's going to be a lot more nervous... He doesn't know what you want to do or are about to do yet... Show him you won't be manipulated any longer.

You can talk to him in a few days after you have had a chance to put him a bit on the defensive... He's likley trying to diffuse things... He may be a bit scared..

Consequences are coming up over the hill and he may be seeing a few of them now... Don't let him control you... just go out.

If you told him you want a few days of peace and quiet then TAKE it... if he won't give it to you then go out.. stay at a hotel... hire a sitter and just dissappear for a couple days...