By give, I mean bend. To draw a hard line and have it work, you have to be perceived by the other person as having the upper hand in the relationship, and it doesn't appear that you do.
Your kids are all pretty young and that's a LOT of responsibility for someone who is feeling more like being free. And when kids aren't your own, it can be hard to have those affectionate feelings all the time for them, because not every child is pleasant, and especially all the time. You have to make this easier for him, and I'll bet you've tried.
But NO ONE ever tries EVERYTHING, even when they say they have. What have you tried? Who handles discipline? What specifically are his complaints there?
What are his complaints about you personally? If there are none, what is different about you now than the time when you fell in love? (If he is happier with YOU than the other things might pale in comparison).
The point is, you are probably not the problem, but you can be the solution. The situation isn't what it should be, it's just what you've got. He SHOULD be responsible, he SHOULD love your other kids as much as his own, etc.
The other point is--you have the absolute RIGHT to draw the hard line, but it isn't likely to save your marriage, and I'm just assuming that is your goal because you came to this site.
You have a very tough situation, but I would encourage you to fight for your marriage at least right now. That doesn't mean you can't have some boundaries, but to expect 'all in' is completely unrealistic, it doesn't happen that way. It's baby steps.