I also don't want to make him mad because he will seek revenge majorily. We have kids together, and he will divorce me and make it nasty.
is your mindset, you're going to continue to get walked all over. The fear is debilitating, and until you lose it, you're going to be stuck. Divorces are BUSINESS RELATIONSHIPS, sadly, and you have to treat it as such. Ours is an adversarial legal system, and there's a reason why they put that little "v" between the names of the two parties.