I think it is highly likely that once you find a place and begin to move that he will "break" and beg you to stay. That would be just empty words though unless he is ready to give up the OW compeltely and do the work needed to repair your relationship.
I fear that you are not going to find a place and then be trapped there, with him in limbo and instead of him realizing what he needs to do he will be able to continue to blame you for everything.
I think it is nuts that he wants you and your (both of your) children to move out of your home. What is he thinking with? How can you guys afford for all of you to live somewhere else any better than you can afford for him to live in his apt?
Me 54 DS19 and DS17 Married 06/1989 Divorced 01/2011