All I can say is document,document,document. Keep track of anything and everything. That way if in the future you do decide to go for a change in custody you will have it all there and it won't end up in a he said - she said sit. If you truly feel that X is being neglectful - you can get social services involved.
BTW - we aren't that far apart - I'm in the Yorktown area. I can give you direct contact info if interested - let me know.
