I get the same symptoms if my battery is a bit discharged due to not using the van for a time , if i put it on a charger for a while it starts straight away, I dont think the temperature is to blame as i live in a hottish climate and for me its always been a partially discharged battery.
Next time its warmer stick the battery on a charger for 4 hours or so and see if it then starts straight away. Or if you have been driving for enough time to put some charge back in your battery, pull over, turn it off and see if it restarts straight away.
This may prove its only a partially discharged battery.