FB, You are right in that the two of you have to start from scratch. 5 years is a long time and a lot has happened in that time. Build the friendship again and see where it may lead.
You mentioned that he will have to come clean about what and why he did what he did. However, he may not even understand it himself. What he said to you the day he walked out ILYBINILWY is pretty standard script for the WAS. Most of us on here hear those words.
My H and I are now Piecing and he realizes that he made many wrong decisions but he doesn't understand why he did what he did. He still carries a lot of guilt over what happened between us.
I would caution you to not let him move back in right away. Take things very slow.
Me47 H46 S13 M16 Piecing since May/09
"Life is 10% what happens to us and 90% how we react to it." Lou Holtz