Maybe just tell him that from a work point of view, you dont thik its a good idea if you see each other as its not really working out for you. But re-assure him that you think he's a great guy, but sometimes these things just happen. I'm sure he'll appreciate the fact that you're being honest with him rather than going through the motions.