Maybe it has been destroyed, repeatedly. Or not quite destroyed, per se, but the surrounding city-states have sent armies to destroy it, but every time they reach it the city is gone. Seeing as they're gypsy people it's not impossible for them to pack up everything and move, and come back a couple months later to resettle. Going with that idea, basically the entire 'population' of the city might actually still be transient, it's just an extremely prosperous caravan city. Or option b, put the city in the middle of swamps that swallow invading armies whole.