Hmm...not sure any of us can say what your feelings are, except for you. Keep listening to your little voice and see what it thinks.
And I agree, don't have him move in until you want him to move in, and he is ready. I know it was hard for my kids to see their dad move in and then back out...