You mentioned that this happened before. What were the negative consequences for your husband when he did this the last time? Other than you voicing your anger and sadness to him, what happened?
I think you're right to let him know this is a boundary for you. But if he can cross it without anything other than angry words from you, without consequence, then it's really not a boundary at all.
