He may be a nice guy, and ex-addicts certainly deserve love too. If it was two years from ages 20-22 I'd probably not be too concerned. But nonetheless - YOU don't have to take this on. YOUR family history makes this especially problematic for YOU and you don't have to nice yourself into dating someone if it's going to be a problem for you with your history. And you can present it to him that way too. It's not about him, it's about the way his history interacts with your childhood trauma. It's okay to say no on that basis alone. And if he can't be understanding of that, you wouldn't have wanted to date him anyway.