I used to worry about this a little until I coped on that the ladies aren't expecting me to impress them. Manners, cleanliness, timekeeping and paying up front are probably far more important to the ladies than what a bloke is wearing. Of course many of us guys feel like we're on a date - hence the wardrobe dilemna.
I dress up for work and dress down for all other occasions. So while I think I look best in a suit, I always dress casually (but smartly, I hope) when visiting a lady for an appointment. Others have made a point about blending in - and this is part of it. I've walked by work colleagues on a the way to a punt on at least two occasions, and they haven't noticed/recognized me in my casual gear. By casual gear - I mean jeans, t-shirt or pull over, summer/winter coat etc.. the sort of thing I'd wear on a night out. Track suits I think should only be allowed if you're a family member of the Royal Tenenbaums!