A draft, even a weak one, destroys the thin protective layer of warm and humid air directly adjacent to your skin. As a result, you start losing heat much faster, up to the point of hypothermia. And this can harm your health, sometimes quite significantly
Hypothermia itself does not lead to illness. BUT! Hypothermia, even local, quickly reduces immunity, and with weakened immunity, it is much easier to catch an infection. And if it were just a cold..