You need to plan your life as if H is not a part of it. Don't ask your H what his weekend plans are any more. Start planning your weekends. If H comes into town, you can tell him that you have plans and/or he is free to join if YOU want to extend the invitation. Continue with your plans regardless of how H's responds and don't let his response both you.