You and him are acting like you cheated, or lied, or betrayed or stole. So you wanted some time to yourself. You are allowed independence. And a real marriage and committment looks past that stuff. He is gaslighting right now.
He is just not a committed person.
What happened in his original marriage? In yours? How did you guys get together?