It's your (family) holiday home and he's unilaterally taken it over. I guess my comment would be let him and try not to let it bother you. Avoid going to visit if you have to because I know it's tough being there (and you feel emotionally attached to the house) and have him 'living' there. It will be all sorted as part of the financial settlement (if it gets to that).
Not the same thing, but every time I go to my H's flat (not been there in a long time, and probably less than 10 times since he moved out) I consciously make an effort NOT to look around. Eyes straight forward, back straight. Stand close to the front door and stay as short a time as possible. This is because my eyes will always fixate on something that I bought and he took when he left without asking - whether it be a picture frame (with the picture of us removed, replaced by a picture of him and the girls/just the girls), a cooking pot / set of glasses. It winds me up seeing things which I see as mine in his flat even though I probably hadn't even noticed he'd taken it.
