Try not to take it personally, it is his confusion although I know it is really hurtful.
You mentioned in your post that he thinks you are not working on the marriage. I wonder what 'working' on the marriage looks like to him. Have you any ideas on that?
