Part of me wants to say f*** him, run and just forget the whole damn thing. I don't want a penny if it means he disappears from my life, I'll make it work somehow. Another part of me doesn't want to let him off the hook. I'm torn which way to go.
I think that's just what he is hoping his intimidation will do. It sounds good, in the near term, but it isn't what is going to be best for you in the long run.
Don't meet him without a lawyer. He is not about to play fair, given any opportunity.