Author Topic: UPDATE to the UPDATE to server outage (Read 12294 times)

matt · « **on:** November 21, 2011, 10:17:37 am »

What Happened?

Our long time hosting company decided late last month that they were closing down shop. Notices were sent to all customers including me, explaining the situation and giving a time table for gathering data and working through trasnfers. This was great as I was looking to update our hardware anyways and was already planning a move.

My false understanding was that we had until the end of the month to work through our migration. Well that leads us up to yesterday, Nov. 21st. Apparently this is the day the switches were flipped off, 30 days from the day the notice was sent out.

But Matt, don't you backup?

The server is setup to run nightly database and disc backup on the server as well as monthly off-site backups.

Excellent...Right?

Well here is where I break the bad news. We have lost all data from Nov. 2nd through Nov. 21st. I created a manual backup on Thursday of all files but did not include the database in this process. I attempted to grab a dump from the server but I no longer have any access to it. The offsite backup is from last month and the only manual backup I have was from Nov. 2nd, which is where we are right now.

The Future

We are now safe and sound at our new host on a new managed virtual server. Our backup process will be updated later today to manual do an offsite database dump nightly.

Summary

I screwed up and lost some user data. I feel horrible about this and will make sure to fix the mistakes that led to this. The site has grown from a small little project that I started earlier this year, to a small, awesome community and something you all rely on. I know some of you lost some data and work but right now all I can offer is my apology and a guarentee that the issues have been addressed.

If there are any other questions or comments, please drop them here or in a pm. Thanks.

Matt

matt · « **Reply #1 on:** November 21, 2011, 10:32:34 am »

Also you may notice a good amount of box art still missing. I'll be pushing these out over the afternoon.

desocietas · « **Reply #2 on:** November 21, 2011, 11:24:15 am »

I'm glad I'm not @foxhack right now.

On the bright side, this is a good way for us to jump on our individual submission/edit numbers by pushing a whole bunch of stuff out after this little setback. The holiday weekend - a perfect time to avoid the family by obsessively cataloging our toys.

matt · « **Reply #3 on:** November 21, 2011, 11:39:17 am »

That sounds like a fantastic idea!

As for those who have lots of recent submissions, @scott and I may have a plan here. We have all of the box art for the items submitted. I'm thinking we can spend some time and restore those items this week. I'll post an update on this once I figure out the plan for sure.

tpugmire · « **Reply #4 on:** November 21, 2011, 12:44:10 pm »

Do you happen to have a cached copy of the duplicate listing thread? I see that you have been able to restore a few threads. I'd just like to see what was reported and fixed so we can fix them again.

desocietas · « **Reply #5 on:** November 21, 2011, 12:45:20 pm »

That's a good question. I'll see if I find it on my hunt. (Honestly, I'm just going through google's cache feature.)

scott · « **Reply #6 on:** November 21, 2011, 12:52:48 pm »

This is depressing. It'll be a lot of manual labor, and may take a while, but, I'm thinking I should be able to restore the games that were added between Nov 3rd and Yesterday. Until then, I might suggest holding off on resubmitting new games to the database, as when I start updating I'd be overwriting any newly submitted games with "old" games.

foxhack · « **Reply #7 on:** November 21, 2011, 01:03:29 pm »

I'm glad I'm not @foxhack right now.

*ahem*

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGH

Well, I still have -all- of the artwork I've uploaded since then. I always keep a local copy. I'm just going to hate having to do everything again... T_T I had 650-something items on my collection, boohoo.

ryan · « **Reply #8 on:** November 21, 2011, 01:45:45 pm »

.... We have lost all data from Nov. 2nd through Nov. 21st....

Yup. I signed up on November 7th. Now to readd my games and reupload some box art.

foxhack · « **Reply #9 on:** November 21, 2011, 01:49:40 pm »

So it's safe to start submitting stuff again, right?

Edit: Speaking of, editing entries gives me a 500 Internal Server Error. Guess not.

madmax · « **Reply #10 on:** November 21, 2011, 02:55:12 pm »

I lost 100+ edits. not too bad. ill resubmit my front-page picture when I get home.

This is what happened to my other website but the offsite backup was over a year old and the service provider didn't give us a notice.

I hate to say I told you so...but I mentioned regular offsite backups last time we had server problems :p

also does this mean the buy and sell features need to be remade?

scott · « **Reply #11 on:** November 21, 2011, 02:56:35 pm »

I'll post an update on this once I figure out the plan for sure.

I think the plan is that Matt and I are going to look at the box art images that we have. Each image is numbered with the same number as the item it belonged to, which will tell use exactly where in the database they went and which game it was. Then we are going to manually insert them back into place in the database and rebuild it to the point where it was before we lost the data. I was planning on starting this tonight after I get out of work at 5p EST. If all goes well, we just need to run a batch upload that will automate matching the data with it's proper item number. If it's not so simple, we input each item by hand.

Until then, I suggest waiting on submitting new items.

scott · « **Reply #12 on:** November 21, 2011, 02:57:30 pm »

@madmax I have the frontpage picture saved on my Laptop at home, so I didn't lose that. Almost all of the images for the site are backed up twice.

foxhack · « **Reply #13 on:** November 21, 2011, 03:03:25 pm »

I think the plan is that Matt and I are going to look at the box art images that we have. Each image is numbered with the same number as the item it belonged to, which will tell use exactly where in the database they went and which game it was. Then we are going to manually insert them back into place in the database and rebuild it to the point where it was before we lost the data. I was planning on starting this tonight after I get out of work at 5p EST. If all goes well, we just need to run a batch upload that will automate matching the data with it's proper item number. If it's not so simple, we input each item by hand.

Until then, I suggest waiting on submitting new items.

Then you should probably pull the plug on the "submit new item" button for the moment. Otherwise you'll get people like me messing with stuff without meaning to.

scott · « **Reply #14 on:** November 21, 2011, 03:04:45 pm »

@foxhack Nevermind, Matt just told me to "Let them go for it" ha. So Start resubmitting items to the database asap. ha