Meet with both.

I think I would tell him that you would be more comfortable with him on the couch. Not sure I would actually kick him out, at least not until you have protected yourself financially.