I don't think you have anything to worry about in that exchange. "Leaving him with bad feelings" might simply mean your visit to your grandfather, or it might be a general "I don't have bad feelings about this."
M60 H52 D20 M14 yrs OW-old gf from 1986 bomb-5/18/08 H filed for D-9/10/08 D final 4/24/09 xH remarried (not OW) 2012