I wouldn't go, though I can totally understand your wanting to go. There has to be some consequence to what he is doing, yes?
Possible scenario's...
1. Whole family goes to dude ranch. WH thinks, great, I have family for these things, OW for whatever.
2. Mila does not go to dude ranch. WH thinks, hmmnnn, this will be my life now, fractured parts scattered around the country, is this what I meant to do?
I don't know. I cancelled trips with WH 2x in the early days when he wanted to be in separate living accomodations, because I did not want to see my future like that. Was very hard to do though.
