You have nothing to be sorry about - totally scummy of him to take advantage of his anonymity.
I have seen a few times where a client taunts an escort, saying "I know you and you don't know who I am". It should be an instant ban imo. Pity we can't ban them from calling escorts as well.