Having no sexual partners over a lengthy duration may be more damaging , for some ,psychologically speaking.
Certain religious people may have found this out in the past.
Like anything else , it is hugely dependant on the individual and their own mentality.