I agree with May. He's likely to be annoyed at you whatever you do because the status quo - having you and the kids in a place that his 'his' and also having the freedom and privacy to live his life as a single man - suits him. It no longer suits you, so you are taking appropriate and careful action to make a change. It doesn't matter how he interprets your actions - you could tie yourself in knots trying to get him to see things your way, or understand that you have the best of intentions - and still, the outcome would be the same or similar.
