I say this in a gentle fashion but IMO you make yourself far to available at the last minute with your H. It seems when he has a spare moment that is when he wants to make plans and because you mostly agree, you enable this and allow him to continue to think you will be there when he feels so inclined to see you.
I would text back (and this is JMO) and let him know that he is welcome to come get the kids for lunch but you have plans so you will just have to take a rain check on the lunch. I might even say... "hey H, I may have a few available days next week for lunch so let me know your schedule, I will let you know mine and we can see what day would be good for both of us".
This serves several purposes (A) you are allowing him to see his kids (B) you are not dropping everything to be with him and meet his last minute requests for plans as they suit his schedule (C) you are requesting some consideration for your schedule (D) you are making yourself less available as your time is of no less value than his and (E) you are letting him know that getting together for lunch is cool but your schedule is busy too so some advance planning is the route you would prefer.
