Stand your ground.

Tell him that you are concentrating on your needs and the baby's right now so he needs to back off and give you some space.

I would send this as a text message so you don't get sucked into a R talk where your emotions could get the better of you.

M-43 W-40
2D - 9 and 5

Emotion, yet peace.
Ignorance, yet knowledge.
Passion, yet serenity.
Chaos, yet harmony.
Death, yet a new life.

RECONCILED AND WISER