I think you are on the right track. The secret is to keep those expectations at bay when things start getting better (not that I can tell you how to do that, I just know it is important, lol)
I think what you said about "can we talk about this when I am not at work" would be a good way to handle it. He is always telling you to focus on your career anyways!
