A draft, even a weak one, destroys the thin protective layer of warm and humid air directly adjacent to your skin. As a result, you start losing heat much faster, up to the point of hypothermia. And this can harm your health, sometimes quite significantly