Magic, Why offer? Just do it. That would be a change in your normal routine.
Why wait so long in advising him that you have to leave work early? If you worked for a company whereby you were an employee, you would have to give ample notice that you planned to leave early. Why not change the way that you deal w/him when it comes to notification? It's better for all concerned when people are given ample notice if someone is leaving the work site. Also, you do not need to tell him why you are leaving early...have you considered saying, "I have to leave a bit early this evening because I have something to do". You don't own him an explanation other than to advise him you are leaving.
Consider how you interact w/him. You and your xbf are mirroring each other's actions. If one does something and the other doesn't like it, someone turns around and does the same to the other. This is not going to work. You have to be the one to change the dynamics in your interactions w/him or both of you will remain in the sandbox and not come out on solid ground. Why not try something different?
I'm glad to see you are getting out and about and this is very good because it gives you something else to think about. Why worry about whether he's happy w/you or not? You aren't together except for work and you are entitled to down time, just as he is. If he wants to work today, so be it. Put whatever time in that you need to and go home. Be sure to advise him early in the morning that you have plans for the evening and will not be working a full day.
Break the mirror and cease mirroring his behavior. Be yourself and try something different. Give it a whirl...you just might be pleasantly surprised to see how he reacts to your changes.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.