The vulnerability of the magic user lies in the fact that he is a single individual, as compared to phalanxes of soldiers or other military professions that rely on discipline and numbers. A single assassin isn't going to do to much damage against a large number infantry, but if that same assassin is able to slit the throat of the head magus then the attacking army is in serious trouble. There would become an ongoing arms race between assassins guilds and wizarding organizations between penetrating mage defenses and detecting assassins, defeating protective magics and protections against knives and poisons. This answer is fun because it gives rise to large potentially powerful assassins guilds, assassin mages, mage assassins, and the like.
2. Magic Scarcity
Magic is rare, or there is a limiting factor to using magic. Magic is rare is a common enough answer, low magic settings are their own genre. Another aspect is that using magic to massively affect warfare requires a certain amount of magical firepower. Magic can remain common but doing large works, like breaching a wall, or blasting towers into dust, but to do so requires a certain magical reagent,like powdered dragon bone. Which is easier, simply maintaining a siege or hunting down a dragon, deboning it, and then alchemically treating and powdering the bones to make the dust to blast the castle to dust. All the while, the dragons have issue with being used as magic fuel, and rival nations and countries can be hunting their own dragons or allying with the dragons to protect them.
3. The Fabric of Reality
The fabric of the universe doesn't like being messed with. It overlooks small transgressions like curing diseases and injuries and raising small numbers of undead, small fireballs, etc, but when the dead are fully returned to life, towers are turned to fire in an instant, and other such things, well, the universe gets mad. Everything exists for a reason and magic goes against this. Elementals can arrive to dispense extreme justice, or the servants of the gods can show up, and start asking why impertinent mages dont see the wisdom of leaving their perfectly good middle realm alone.
4. The Race of Magic
Personal scale magic is fine, and not really heavy enough ordnance to take out the sort of things that a siege engine would. Handling large scale workings require the assistance of an astral race. This is not so difficult, and becoming the master or ally to one of these astral beings isn't hard. The hard part? The Astral Beings are pacifists and loathe war. Convincing pacifist beings to blast thousands of living beings into ash is going to be really difficult.
5. The Oath of Mages
Magic is considered the highest art, and a cadre of mages going into something as base as war is as unlikely as a cadre of college professors acting as shock troops. Such things are beneath the magi, and they abstain from warfare. Mages function as an untouchables class or an elite faction that holds itself aloof from secular issues. Mages who stoop to involving themselves directly in war and other material issues can expect to have themselves drawn into magical duels against their peers, censuring, magical imprisonment and the like.
6. Magic is Divine
In the Tolkien universe, magic was the echoes of the music of the gods. Performing the magical arts was something that followed the flow of reality rather than destroying it. Gandalf was a powerful wizard, but he barely ever used magic because it wasn't part of the greater plan. Trying to use magic in such a vulgar and blatant way is only possible if the magus channels the discordant music of Morgoth, and therefore, can only be done by evil magi who are the primary targets of the good magi. Typically good gods outnumber the evil gods.
7. Magical Cold War
During the Cold War, the USSR and the USA stood toe to toe armed to the teeth with nuclear weapons. Mages function in the same balance of power. The only reason that they do not unleash the arcane firepower is that there are pacts and deals made that would open dimensional gates and flood the world with demons, fire elementals, or some other civilization ending event. The mages know the power that will be unleashed if they start slinging spells haphazardly and thus push for peace, or strive to restrict wars to conventional means only.
Additional Ideas (1)
This is the simple idea that the glyphs, wards, runes, area enchantments, magical emblems, circles of protection, and so forth are just more effective than the opposing counterpart. Every fortresses with any longevity is sure to be loaded with such permanent magical 'armor'. As a Siege or a Battle mage, are you entirely comfortable loosing your 'wall crushing' spell on an unknown fortress wall when the spell might be blocked, changed, and spit back at you magnified ten fold? This of course means that to properly engage in a magical siege one needs to send in spy/magical types in to try figure out how the defences work and then a way needs to be created to 'hack' those defences. I wager this is not an easy task, and therefore possibly a great quest for PCs.
9. Anti-Magic Stone
Athough possibly uncommon today, at one point loads and loads of anti-magic stone was found and used to create a large number of strongholds. Simply put, a magical siege simply doesn't work on any major location. As an added twist, wizards are useless when touching said structures or even when passing by.
10. Divine Protection
In the tradition of Greek mythology, each city or major location has a patron deity. Provided that the patron deity is properly appeased, major magical assaults will fail. Here's a few examples of how a magical siege would be thwarted:
A Luck God's City – All magic seems to miss or deal minimum damage.
A Death God's City – Offensive mages tend to have fear issues which most of the time trips up their spells and on occassion they drop dead for no reason whatsoever
A War God's City – Strangely arrows and other missiles fired from the city walls tend to kill the siege mages very quickly. The offence’s defences are somehow nullified.
A Love Goddess's City – the siege mage has a change of heart and abandons the siege. Reasons possibly include, a new love interest or a rekindled romance. Alternatively, the mage just wakes up and finds himself a total pacifist and unwilling to let the spells go no matter how angry the generals get.
A Nature Goddess's City – this one has many possibilities. A few examples include plain anti-magic, the walls that rebuild themselves, a perpetual storm outside the city walls that keeps line of sight to a minimum and spoils magical ingredients.
A Sun God's City – most offensive magi suffer from sun stroke to the point where even a night assault is unlikely. If a mage has the constitution and the stubbornness, at the moment a spell is to be loosed the caster becomes blinded by a bright light.
Mathom's City – Are you kidding? Just walk away, if you can. (I threw this one in just for fun)