1-Magic Users Don't Like Competition
One reason why magic-users are so rare outside schools and universities of magic is that they don't like other magic users in their area. If they are just passing through it's fine, but if they set up shop in the area, they are taking away customers that would otherwise go to the other user of magic. And that costs him or her money. Users of white magic will have a friendly talk with the other magic-user and try and persuade him or her to move on. If that does not work, the resident mage or witch will use his or her influence with the movers and shakers in the area to harass the other one into moving on. If that does not work,and it normally does work, things can get nasty and gray or even black magic can be used as a last resort against the rival.
Black mages and witches, as they work outside the law anyway, are much more direct and the gloves quickly come off. An interloper gets one stern warning that the area they are in is already taken and they have twenty-four hours to move on. If that does not work, then the resident mage will try to kill or at least heavily curse the intruder. Users of White and Black magic can often co-exist perfectly well, as long as the black magic user does not practice his or her dark art too openly and bring down the wrath of law enforcement, vigilantes or bands of wandering adventurers upon all users of magic in the area. After all, they are not practicing the same arts so will not steal each others customers.
2-Magic is Slow
Magic works, but the fast moving magic people think of is fantasy. Healing magic is rather like chemotherapy; it takes a long time, can often hurt the patient, but is sometimes the only cure for certain ailments. Love spells take as long too.By the time it works, you may have fallen out of love with the one you cast the spell on. Fire magic has to be cast on something that will burn, and can take up to a year, slowly heating the object to the point of ignition, before it flashes into flame. It can be used for arson; if cast as a house is being built for example, a year later said house will catch alight, by which time if the caster has any sense he or she will have long since left the area. If there is no system of detecting the magic, it can be impossible to tell if an arson attack was caused by magic or other means or was just a genuine accident.
Curses work, but slowly. If a starvation curse is cast for example, rather then make somebody wither away to flesh and bone within days, or even weeks, the effect is more like first a tapeworm and then a cancer. Only slowly will the person's hunger grow and at first regular meals are enough to solve the problem. Then he or she will need more and more food to stay nourished and it will finally reach a stage where no amount of eating will solve the problem. Most people do not have the time to make magic work for them, even if they have the knowledge to use it.
3-Magic Messes with your Mind
Any *bad* magic or combat magic causes what we would call nowadays Post Traumatic Stress Disorder, making the caster have nightmares for months or even years on end. People who cast a large enough spell of evil or battle risk ending up with serious mental problems perhaps for the rest of their lives. Good spells also have an unwelcome result but in a far different way, causing the caster to fall in love with the target. So very few spells are cast, when they are cast, it is as a last resort.
Magic bounces back horribly. Any curse will inflict itself on the curser as much as on the cursed one, regardless of why the curse was cast in the first place. As for healing magic, you can only use it to heal somebody if you are willing to accept the person's wound or illness yourself. And bringing back the newly dead is only possible if you are willing to give up your own life for that person.
Learning magic costs a lot of money, to the point that many magic users make most of their money teaching magic to others. Even cantrips cost the gaming equvilent of hundreds of dollars. A middle ranking set of spells costs the equvilent of several years at colledge, and really powerful spells are more expensive then buying a ship, a large house, or an entire platoon of mercs. This suits the users of magic well as they can make lots of money teaching to others, and prevents the poor and indeed the middle class from using magic to overthrow the aristocracy.
6-Only the Gods Can Use It
7-Only Dragons Can Use It
8-It Really Really Hurts
Anybody who is clever enough and has enough money can learn magic, but to cast it requires a magical focus. The focus itself is a kind of crystal that is quite common and easy to buy. But instead of fitting it into an amulet or wand or staff, the would be wizard or witch must pull out one of his or her eyes and put the crystal into the eye socket. Only then will he or she be able to cast spells that really work. As you can imagine, even with the vast power and influence that a caster can wield, most would-be spellcasters are not willing to go that far. The few that do often become true masters of the magical arts as they put so much time and determination into learning magic.
9-It risks bringing the Netherworld to Earth
11-It damages body parts
12-It causes people to sing
13-You need to implant something into your body
Additional Ideas (5)
There's a brilliant short story by Robert Sheckley. In a world of black magic and darkest sorcery, one man's son... wants to be an accountant.
Magic drains away your soul.
Magic eats away your body.
Magic prevents you from wearing armor, wielding weapons or using any iron utensils, maybe even going so far as riding a shod horse.
Magic makes you go mad.
Magic is only inherent in a certain species/race.
Magic requires someone to die.
Magic can only be used in certain places with inherent laylines.
In order to use it, you can't have ANY other talents.
There is only so much magic to go around.
Magic turns you into a horrendous monster.
Magic is evil and only evil people can use it.
Magic is a disease and most people are immune.
Plus only few of those who have the time actually study it since, even if you have the time, there's always "something more important to do", like planning taxation, siring enought male heir so that at least one will reach adulthood ...
Not to mention that without a proper method of styding magic one can only learn as an autodidact or from some rare master that maybe don't even fully understand what they are doing, sure when i burn salt and say these word i can cast a fire ball but why? (basically no scientific method)
That said some rare genius born in the right family or at the right time and at the right moment can still achieve considerable results.
Magic brings age. The reason why wizards always have long, flowing beards? Using magic takes years off of one's life. Powerful users rarely make it past 30.
Magic is socially unacceptable. Sorcerers are more than just feared, they're hated, even hunted. Why risk using magic when you'll wind up cut off from society?