Cathy, Instead of confronting him, why don't you just "act as if" you are all going? I see a vicious circle here. Him saying something, him backing out, you confronting him. Do a 180.
You drive up too. You know you need to get away! And going on a trip with one's family can be so fun! Go, girl! And quit tucking your tail between your legs and running into the cave. If you want to go, go!