It's good you had an open conversation with your H, and maybe there's no EA--I know you think this is not true, but I guess there's a possibility there isn't. I still recommend more space. You can limit your conversations to the kids.
I get the feeling your bottom line wasn't him talking to the XW. So what is it? Think about it.