First things first. We MUST find a name for this horrendous event so that we can appropriately refer to it in our rage.
Second, I must take full blame for it.
In a nutshell. I have been working the udpated submission page, as most of you know, and spent a lot of days on it. It uses an active record style database query creation. So basically, to make a query on the database, it uses a lot of stand alone lines of codes that are collected by the script and it creates the query from them. So, depending on if statements and the state of the sub, the query slowly gets built as you progress through the code. Unfortunately, through one piece of it, the main update query was interrupted by another active record query. You can't, necessarily, do that. So the first query that had all the sub details was hijacked and executed by another queries execution call at which time the database was updated per that queries final logic...the wrong logic.
So, bottom line, I wrote a rogue script on a live database.
Prior to that I made two local backups, the host backups daily and the server backups locally every two days and on the weekends. HOWEVER, the control panel that runs all this has some sort of s**tty bug so every backup that was saved in every location was corrupted. I opened trouble tickets and the server's files were refreshed with all corrupted data and all my local backups for the past 2 months were all useless. Until I got to the April 14th backup which was the first in the string that was actually usable.
Dossta and and some other folks (Muro did an awesome job as a cheerleader) acted as the quick reaction force and started going to google cache and salvaged most of the visible submissions out of the cache. Valadaar put on his paladin armor and in its glowing arrival, he was able to manipulate multiple backups and salvage what we were able to keep. He is responsible for the saving of the forum and all comments/votes/anything else associated with the missing submissions. We collected those in a shared google doc here https://docs.google.com/document/d/1cgT_Ql7PHZraATtb8J0_ZtH3_ipAa1u2eQ8c58yh2y0/edit?usp=sharing
Unfortunately, we can only save the visible. There are drafts that I have in the database, I will sort through those this morning as a first step and get them back up ASAP.What do we now?
I ask for your patience for a little bit today. I have a lot of stuff I pretty much need to do local before we can start the process. I want to jump on that now while I have some time. Then we have to come up with some procedures on how to get us back to where we are. I will have a better idea about that once I get some more clean-up done.
Give me a couple hours this morning to get organized and I will make another announcement.
My heart hit my throat when I saw what happened. There is no way I can apologize enough for making such a stupid mistake. I know better than to work on a live site, but as you know, that is what I tend to do most of the time. This was a slap in the face of reality and, with the help of some very smart people on the site, we will see about making sure this doesn't happen again.
Thank you for the support and assistance in getting us back up. If anybody needs a physical address so you can come shoot an arrow or if you want to invite me to marry one of your daughters and have a royal wedding in your keep, I will attend.
Updates to come!!