The anger really is perfectly natural and also at the same time destructive and a hindrance. So try not to dwell on feeling bad that you're angry; it's natural that you're angry. At the same time, it's good advice to work on letting go of anger early, if only because it will probably take a long time to do it and the longer you put that process off the longer it will take.
If you and your wife make positive changes, it will get easier and easier to let the anger go because you won't be getting it reinforced every day as you are now.