Why do you even have to tell him you are changing offices? Why not just do it and wait until he asks you what's going on?
Make sure you are not setting yourself up to have a conversation with him that you don't need to have, just so you get a "fix" of him. You know what I mean?
You H has been making a lot of decisions about his life without consultation - why not give him a little of his own medicine? Keep him wondering?
V
Never make someone a priority, who makes you an option.