It's a big risk. If you've already done it by mistake, what's the chances you'd do it again? Not high, and that might make him suspicious to begin with. Then he would be suspicious and you would lose credibility.
You need to think of other ways to let him know you are out and busy. Have a night away. That will have him intrigued when you need to ask him to take care of your babies.
