Bullcrap.

You need to document the times he misses his appointed times. You have absolutely no requirement beyond his appointed visitation times, period.

Next time you get a nasty letter, send one right back with a documented list of his missed visitations.

I guarantee you won't hear another peep.