I agree with LR. Stop freaking out. Arrive back home with S at the time he's showing up. Then it's not "freaking out over nothing." You actually were busy and actually made plans, which is fine and GAL.