Our expectations, especially the "wouldn't it be nice" kind, really can put us in a bind. So hard not to have them, but entertaining them seems to alway put a damper on us. Don't let it block out the recent times, where he does seem to be warming up when he is with you.
Try to be understanding that he is focusing all his energy now on this trip with the kids. Of all the possible choices on how to spend his time, this is a noble choice. I'm sure he will be full of stories to tell you when he gets back.