Document document document. Keep track of every time he doesn’t take his visitation. It may come in handy in the future. (And if you want him to take it, simply tell him you have other plans and are bringing the boy to his house, but I suspect you would rather have more custody in the long run).
Enjoy your son, enjoy the present. Don’t let H’s issues rob you of joy today.
And whatever the reason for OW leaving her job, be glad, because now he’s less likely to lose his job and file for spousal support.