I once read a line in a Glen Duncan book about a young lad in college having sex for the first time and as she is putting a condom on him his penis becomes "....almost defiantly limp."
It made me laugh that line but bloody hell folks ain't it the truth??
The whole act/experience of sex is such a complex interplay of physical sensation, hormones, conscious/subconscious and a good deal of your lifes conditioning it would be very difficult to say exactly what causes the whole scenario to "collapse" so to speak.
But whatever the reason(s) I don't think its anything to be feeling bad about yourself for (on the contrary I think that would be most counter-productive)