YR, Reality only sinks in w/them when it's money out of pocket to you and your children. They do not like to "share" their funds w/us, but that's the consequences of his actions.
You have done absolutely nothing wrong and believe me, from what I know about you from the board, you've been more than honest and willing to work w/him everything.
I would go ahead and advise him when he comes to see your daughter, that he needs to either take her out for a bit or back to his place. Your home should not be a place for him to make as his permanent visitation site.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.