Did you journal when he had visits with the kids? That would probably help.
Generally custody agreements are established between parties and then there's a term that basically says "and any additional visitation the parties agree to." I think of the DD as our minimum, or our guidelines if we can't get along or agree.
You're other questions likely need to be addressed by an attorney. Each state (and country) is a little different.
