Well, good for you. You stood up for your family, your children, and they're your first responsibility.
Ofcourse you feel that he's taking advantage of you - HE IS!! I know you're willing to let that slide for now, but don't feel bad for feelings that are justified.
I applaud you - you're doing a great job hanging in there.