My wife, of course, gave me the same line. I asked her "Let's assume for a moment that's true, even though (and I looked her squarely in the eyes for emphasis) we both know that's not true. But let's assume he's just a friend. Are you telling me that you're placing your friendship ahead of your husband? Because if the situation were reversed, and I knew it was a dealbreaker for you, I would honor your request."
Methinks she doth protest too greatly. I'm glad you're protecting yourself. Just try not to say or do anything that's irrevocable -- leave the door open.