I vote that you not be around when he comes over. I think you need to see a lawyer first before you talk to him about kids or bills. And you can tell him that. I detect a nasty tone in his voice because he's now trying to threaten you into submission. It's clear he's nervous and the best defence is offence, right?
My husband does the same. It's a smokescreen. Don't waffle.