I would stick to a schedule that you have. If he misses his weeknight visits then its his loss. That is a long time on the weekend for you to be away from her.

Get an attorney because he sounds like he is still running the show and it will only get worse.

Me: 46 FWS: 36
Married and Divorced 4/07, Pregnant 7/07,False R 7/07
Baby Girl born 3/08
Kicked him out because OW: 7/08
5/10 He realized what he had and lost.
Moved home! REMARRIED 3/14/11!!