OK, so you didn't get the whole day. It is time to insist on the 20 minutes of dialogue. Without it you are lost. If there is no discipline you will get shut out of his day every day. 20 minutes isn't a lot, and it won't raise his blood pressure. From your vantage point, 20 minutes with no distractions puts the 2 of you in one place together....who knows, perhaps he won't jump up afterwards and go build any fences. You might end up spending more time with him without having to beg for it. Now for a dialogue question, "how do I feel about spending time together?"