Just too many secrets. He'd have to go way out of his way to keep his kids from even accidentally tripping over another female. And what about his friends? If they are outwardly gay he couldn't have anyone see them either.
The fact that there was nothing really wrong with your marriage, he's not angry at you. There's no sexual tension going on.
Is he into Barbra Streisand, Liza Minelli, Ethel Merman, Bette Midler and/or Musicals?