Sound like you are doing great then! I have the luxury of hindsight, so as crappy as it feels right now, just keep sticking with it. I promise it works!
Also, instead of worrying about how the mom acts (I know it, it [censored]), just pour yourself into being the best dad ever. Honestly, some of the most amazing times of my life happened in XW fog - the kids and I went EVERYWHERE and had a blast.