In some ways I don't feel like one to give advice, but you may want to rethink letting your kids go to his apartment. It's a reality check on his part. Your current situation allows him to "split" his life quite nicely.
Dropping off a noisy happy 2-year-old into a love cave was quite fun for me.
M: 16 years Bomb 4/07 OW 20s long gone Divorced 11/09 I remarried New Guy Cooperative r w/X regarding D