I gained such knowledge by always being the DM. Sure I had control of all the NPCs, but I never experienced the true complexities of figuring out the puzzles and strategies set before the characters (sob, sob).
I am thinking of posting an item I used to buy time developing a next scenario (I always had a small outline with characters and maps for a game, but prefered to create their encounters on the fly). I would always try to throw little quirky experiences to keep them guessing.
This continued drive to develop fresh ideas led me to create a system of my own. So, much of what I write will be what I used in the past, or desire to flesh out now. This seems to give me a good forum to toss out my ideas. Kinda like the character Ur'jund, I wrote in. I've had the idea for him for ten years (roughly), but never was able to sit down and give him life. I changed who his enemies are (races and characters from my system) to known opponents- drow - so as to keep the focus on him. I also found myself developing a magician class based on quick bursts of power to gain an advantage in melee combat (must keep on back burner. . . need to resist being side tracked again).