Because they wake with that well known salute to all things sacred -
The Morning Hardon (TM)

get all excited, time for action, ring you, make the appointment probably with good intentions, but then:

a/ the wife wakes up - which either kills all thoughts of passion, or if they are lucky - its time for the weekly quickie, so she can get back to sleep!

or

b/ The thought of you, one on one with his masssive tool, is just tooo much, so he knocks out a quickie and then falls asleep

or

c/ His mam rings inviting him to sunday lunch - a home cooked meal vs nookie
You know the winner here!