It's this huge misconception that space is super cold. A better mind model is that space doesn't have much of a temperature at all just cause there's nothing there. Vacuum doesn't have a temperature.

You won't immediately freeze if you enter space without a suit. You'll die almost immediately, but it will be because your blood will begin to boil because of the lack of pressure kinda like an extreme form of divers sickness - it's just not the cold that'll get you first.