It's an act of trust and choosing to love. Without trust and love, the house that is your marriage cannot stand. But that doesn't mean you have to do it silently. It's OK to let him know that you are doing it as such.