The mind of a wizard is more open to the more esoteric elements of his surroundings Ã¢â?¬â?? this is why he is able to perceive the weave of the Great Tapestry that is the multiverse, and pull at the strings. This increased sensitivity can manifest in several ways.
Historicly, we have had plenty of rl groups like the Taliban and the Puritans who supressed anything which was in any way fun. What if there was some group that was the opposite? (which in time would cause problems of it's own for the civilians under their control.)