He is wanting to do some "cake eating", pure and simple. At present, I'm sure there is nothing else there. Just do as you said and nicely, gently, but firmly reject his offers. It is that simple. If you are worried that you might be missing some sort of "reach out" from him, don't worry. IF you want that, IF it is for real, and IF it is truly some sort of change in heart, then it won't be fleeting. Just throw your arms around your kids and develop a good relationship & life for the three of you.
As for the trip to Hawaii, I agree with MB. It sure seems that your S is way too young to be away from his Mother for that long. That is aside from how you feel about his family.
As for that about his family... wow! Ought to serve as a good reminder for you and help steel your resolve a bit. Refuse his requests. Start moving on without him...