You certainly don't have to be there when he gets home. Heck, you aren't even obligated to bring him food. But I think the note is sweet..that way, you did tell him you would be gone.
Oh...and I agree with the idea of encouraging him to go for a walk/out/whatever. (This actually works well, I've done this with my H in the past...even though there were times I was littler choking on what I was saying. )