Tough one. You do need to have your own time, which your extended family could help provide. But you dont want your daughter to miss too much time with her father and I am sure you can see how it would hurt him to miss seeing her.
How far away from Toronto is your home town?
If you did move, do you think that he might find work in your home town?
