Although I'd love to have told the guy in this situation where to go and what to do with himself once he got there, you probably did the right thing by being cautious rather than confrontational or defensive. Having no idea who this guy is or what other friends of his might've been there with him in 30 seconds if he'd called them, which seemed likely, as I take it that he approached you with a rather aggressive and confrontational tone, I'd say you were best to just leave. You're also just as well exercising caution with it being your home area.
Fighting the Good Fight...