Well.... He is their father and the boys seem to relish their time with him, no matter how confusing it may be to them. I wouldn't block his access to them unless he was doing something illegal or unsuitable (in your standards) for the children.
Some kind of agreed upon schedule would be a healthy thing for everybody. Your 2 "protected" days is a good start. As you go along, if the schedule doesn't work out, simply change it.
