llamaenterhear
Username: Password:

Author Topic: The Day the Horde Stood Still  (Read 1433 times)

0 Members and 1 Lonely Barbarian are spying on this topic.

Offline Strolen

  • Ignorance Incarnate
  • Guild Leader
  • Emperor
  • *
  • Posts: 7991
  • All your base are belong to us.
  • Awards Locations Guild Questor Locations Guild Elite Item Guild Hall of Heroes 10 Plot Guild
    • Strolen's Citadel
    • Awards
The Day the Horde Stood Still
« on: June 14, 2013, 01:04:34 AM »
First things first. We MUST find a name for this horrendous event so that we can appropriately refer to it in our rage.

Second, I must take full blame for it.

In a nutshell. I have been working the udpated submission page, as most of you know, and spent a lot of days on it. It uses an active record style database query creation. So basically, to make a query on the database, it uses a lot of stand alone lines of codes that are collected by the script and it creates the query from them. So, depending on if statements and the state of the sub, the query slowly gets built as you progress through the code. Unfortunately, through one piece of it, the main update query was interrupted by another active record query. You can't, necessarily, do that. So the first query that had all the sub details was hijacked and executed by another queries execution call at which time the database was updated per that queries final logic...the wrong logic.

So, bottom line, I wrote a rogue script on a live database.

Prior to that I made two local backups, the host backups daily and the server backups locally every two days and on the weekends.  HOWEVER, the control panel that runs all this has some sort of s**tty bug so every backup that was saved in every location was corrupted. I opened trouble tickets and the server's files were refreshed with all corrupted data and all my local backups for the past 2 months were all useless. Until I got to the April 14th backup which was the first in the string that was actually usable.

Dossta and and some other folks (Muro did an awesome job as a cheerleader) acted as the quick reaction force and started going to google cache and salvaged most of the visible submissions out of the cache. Valadaar put on his paladin armor and in its glowing arrival, he was able to manipulate multiple backups and salvage what we were able to keep. He is responsible for the saving of the forum and all comments/votes/anything else associated with the missing submissions. We collected those in a shared google doc here https://docs.google.com/document/d/1cgT_Ql7PHZraATtb8J0_ZtH3_ipAa1u2eQ8c58yh2y0/edit?usp=sharing

Unfortunately, we can only save the visible. There are drafts that I have in the database, I will sort through those this morning as a first step and get them back up ASAP.

What do we now?

I ask for your patience for a little bit today. I have a lot of stuff I pretty much need to do local before we can start the process. I want to jump on that now while I have some time. Then we have to come up with some procedures on how to get us back to where we are. I will have a better idea about that once I get some more clean-up done.

Give me a couple hours this morning to get organized and I will make another announcement.

My heart hit my throat when I saw what happened. There is no way I can apologize enough for making such a stupid mistake. I know better than to work on a live site, but as you know, that is what I tend to do most of the time.  This was a slap in the face of reality and, with the help of some very smart people on the site, we will see about making sure this doesn't happen again.

Thank you for the support and assistance in getting us back up. If anybody needs a physical address so you can come shoot an arrow or if you want to invite me to marry one of your daughters and have a royal wedding in your keep, I will attend.

Updates to come!!
 
« Last Edit: June 14, 2013, 01:41:32 AM by Strolen »

Flying Squirrel – Strolenati Guild
Grothar Rockfury - Dwarvish Guild
Minor Minion - Cartographer's Guild
Level 3
STR: 5 | END: 2 | CON: 3 | DEX: 2 | CHA: 2 | INT: 6
Authentic Strolenite™©® | Llama is as Llama does.


Offline Strolen

  • Ignorance Incarnate
  • Guild Leader
  • Emperor
  • *
  • Posts: 7991
  • All your base are belong to us.
  • Awards Locations Guild Questor Locations Guild Elite Item Guild Hall of Heroes 10 Plot Guild
    • Strolen's Citadel
    • Awards
Re: The Day the Horde Stood Still
« Reply #1 on: June 14, 2013, 01:44:28 AM »
Just as a quick follow-up and to reinterate an important fact.

We have saved EVERY comment, vote, idea, upvote, suggested sub, author XP (sub count will change obviously but that is more dynamic) and any other trackable piece of information EXCEPT the missing submissions.

Our goal over the next days is to resubmit the missing submissions, update their ID to the OLD ID and at that point, all the other stuff will magically reconnect.

So we will be coming up with a plan for the resubmission of old subs so that those with database access can update those to the old ID and get them back to normal as quick as possible.

Flying Squirrel – Strolenati Guild
Grothar Rockfury - Dwarvish Guild
Minor Minion - Cartographer's Guild
Level 3
STR: 5 | END: 2 | CON: 3 | DEX: 2 | CHA: 2 | INT: 6
Authentic Strolenite™©® | Llama is as Llama does.


Offline Strolen

  • Ignorance Incarnate
  • Guild Leader
  • Emperor
  • *
  • Posts: 7991
  • All your base are belong to us.
  • Awards Locations Guild Questor Locations Guild Elite Item Guild Hall of Heroes 10 Plot Guild
    • Strolen's Citadel
    • Awards
Re: The Day the Horde Stood Still
« Reply #2 on: June 14, 2013, 01:46:35 AM »
Follow up to the follow up.

You CAN comment and vote now without issue. Please DO NOT submit any submissions until I am ready though please. If you do, you might have to lose it.

Flying Squirrel – Strolenati Guild
Grothar Rockfury - Dwarvish Guild
Minor Minion - Cartographer's Guild
Level 3
STR: 5 | END: 2 | CON: 3 | DEX: 2 | CHA: 2 | INT: 6
Authentic Strolenite™©® | Llama is as Llama does.


Offline valadaar

  • Lord Ascendant of Typos
  • Emperor
  • ****
  • Posts: 3138
  • Awards Lifeforms Guild Elite Master Questor Golden Creator Locations Guild Elite Hall of Heroes 10 Elite NPC Guild
    • Awards
   
         Human Strolenati
         Strolenati Guild
         DwarvenGuild - Dorak Stonehammer
         Weavers Guild
         Level 3
         STR: 4| END: 4| CON: 4| DEX: 4 | CHA: 3 | INT: 7
         Authentic Strolenite™©®