Neither would drop the other one, for the sake of the kids too. I think it gives you that much more to fight for when times get rough. It’s not easy to walk away now.
Not to freak you out, but this doesn't seem to stop people from breaking up at all.
As for M's comments about marriage - I wouldn't take too seriously what a guy says in the company of guy friends about marriage. Now granted, he has good reason to be gunshy about marriage, but that doesn't mean he's not going to want to make an honest woman out of you some day. I just suspect that would be on something like a 3-5 year timeline.
Sounds to me like what's bothering you most about the relationship is the lack of sex. Do you suspect his drive isn't as high as yours?