I know how you feel, I too was getting I love yous until the day before the bomb. How the bomb just came out of the clear blue sky.
I would be careful allowing him to visit too much seems like it may help him relieve his guilt but is doing little for you. One thing I learned for the the book is you need to test the waters - maybe his coming over is bringing you guys closer.
Only you will know if this is helpful or hurtful. Trust your gut
---- M 39 H 35 D5,D4 M 4 T 9 ILYBNILWY 5/18/11 Left 7/11/11 Divorced 12/1/13