I think that the longer you don't give him the reaction he is after, the less H will show that irrational anger. Though he might well still try the occasional ambush. Whether that means things actually improve, I don't know, but at least you will feel safer while you figure that out.
