I am struggling with something very similar regarding H coming to the house without set times. For the pat two years, H has been traveling almost non-stop. he spends more time away than he does in town. So I had been hesitant to put boundaries in place because he has such little time with them as it stands...and the kids are heartbroken enough as it is.
I would like to think that I will know when its time. At least I hope I do.
TPS Me: 44 H: 42 M14 T17 S10 D7 10/10 H moves out after death of his father-same month 21/04/12 H is 'DONE' 04/05/12 OW/PA confirmed (rumors from 2010) July '14 H ends affair May '15 H moves back home