Politely & calmly, go thru how you came about the letter like you did here and sum it up by saying you are just wondering how that address came to be on the letter. Its his choice to make the step whether it will be closer to you or further away. If he backs off, drop it, otherwise any more pursuit in the matter will just drive him futher away.
... and like Ellie mentioned, you may have just pick up on a residual aftermath of a past action that is no longer relevant today.
