You have a game plan in your mind, you know what you want to do. I'd have to agree with Bond.
I'm not sure why you think you need to pull back, but that isn't good onesizefitsall advice. It will probably backfire on you, and it could be very detrimental to your parenting as well.
You ALWAYS have the option to back out and leave. Try building now while you've got him around, try what Bond is suggesting.