That is a tough thing to figure out. Can you find ways to be out when he is home, or at least make sure you are not necessarily in the same room? I personally don't feel like you should keep him away. The more time he spends with the kids the better IMO.
