I think you SHOULD have answered with those remarks - maybe it would have shut this silliness of his down. Either that or ignore him from the very beginning when he does this stuff. All he is doing is baiting you and trying to reach out and frustrate you.