Not all armor is created the same, some is made from sterner stuff, others, not so much.
When was the last time a character caught a cold or flu? Real people get them every now and again even with solid heating and dry interiors. Your characters are out in the cold and wet, and even inside is not always dry and warm. Imagine having to stop your dungeon crawl, sitting in a tight defensive position, while you all are feverish.