If life hands you lemons, make lemonade.
If life puts you in charge of defending a gold mining operation and you are only a mage....well
Elves are not actually aloof. They are actually extremely family-based and insular, and often intermarry inside their families, making them uncomfortable around those not of their family. Therefore, they shield themselves behind pretended aloofness and coldness until they are comfortable in others' company.