Is it really a mistake? That may be the legal term for this situation even though you have joint custody, it may still be called visitation in this context - don't jump to conclusions.
If you pick up the kids and take them with you, your fear is that the kids will call mom to pick them up? Do you really believe the kids don't want to be with you? I would pick them up, explain the terms of custody in the divorce papers, explain that their mom doesn't have the right to unilaterally decide for all of you, and have some honest discussion with them about their fears/concerns/desires. And tell them it's your turn. Simple.
