Don't ASSume anything.

If he replies and says "no, really, it's work" then just answer "I'm sure then you won't mind if I bring some work with me and join you? I know it seems silly but that's what it will take to put my mind at ease."

Ellie