Definatly, spending time with your son is a priority over book sorting!
Just remember, if you end up dealing with H's books, you don't have to do a good job. Stack them in the corner, put them in plastic bags, or the worst moth-eaten box you can find.
Did you already threaten him with donating his stuff if he doesn't take care of it, or was that TTD?