You handled it perfectly. Nothing wrong with telling him it doesn't work for you. He's noticing you are getting fed with his BS and he doesn't like it.