I used to have the same fear OP. Not about size, but about performance. I could never go out with/ sleep with a girl if she had any links with anyone I knew.
I actually passed up loads of great girls for this reason, and I'm kicking myself for it now.
I eventually talked to a friend about it, and he basically just pointed out that if a girl DID blab about it, it would be more of a poor reflection on her than me.
Honestly, just imagine the worst possible scenario realistically. Even if she did blab, it's not the end of the world. It's not like you're going to be ostracized or anything.