You plan your own schedule. You don't have to consult with him. If that bothers him then tell him that he's the one that doesn't want to be part of the team. You plan your comings and goings with your team. He's welcome to be part of that team but there are rules/standards to live by. Being loving, honoring, caring, respectful, AND faithful are a part of those rules. Your team has value. Not everyone can join. You do what you want to do. It's your time now.