I understand exactly why you did what you did. I don't necessarily think it was the right thing, but I understand it.
My XH adopted my two youngest at 2 & 4 years of age. They are now 23 and 25. He has no contact with them or their children. He raised my two oldest, and thought my oldest daughter's son, the first grand, hung the moon. Again, no contact. I don't remember your entire story exactly, but drinking was a big factor in mine as well.
If I may make a suggestion. Make a pledge to yourself and to God, that from this day forward, you will not initiate contact with his family in any way. They are obviously hostile, for what ever reason, and blood will always be thicker. I had an excellent relationship with my X's family. Now, I hardly hear a word. They are afraid it will put a strain on the relationship with their son. I understand that. I don't like it, but I understand it.
Regardless of what your BIL or SIL said,let it drop. If your H resumes a R with his D and she is open to it, just swallow it and let it be. If he does not with your son, do whatever you have to to help him see it is not his fault or problem. Guilt has the major part in this play. ((Hugs to you and your kids))