Have you talked to a L yet? Find out what your parental rights are. Go for full custody and setup a visitation schedule. Do what YOU want to do. And NEED to do.
If he doesn't want a visitation schedule, that's his tough luck. You are the mother, stop having him control how you are going to live your life. He doesn't own you. Start living and thriving for yourself. He gave up his "right" to be a father when he started going after OW. You are not powerless. Being a mother makes you more powerful than you or he can ever imagine.
M-43 W-40 2D - 9 and 5
Emotion, yet peace. Ignorance, yet knowledge. Passion, yet serenity. Chaos, yet harmony. Death, yet a new life.