ATS Downtime on 9/27/2006

As you all know, we had some downtime today.

Some critical server maintenance and upgrades were begun, and the firm hired to do the work started on all our servers at once, rather than one at a time. The result was a hard-crash of all ATS servers as one of the updates ended up conflicting with existing customizations.

After some creative server acrobatics involving myself, our host, and our contract service firm, we're back online without any loss of data. At one point, it looked like we'd have to "re-roll" all servers which would take days (nearly 40 hours to rebuild our massive database alone).

In the end, we had about 4 total hours of downtime.

I know return you to your regular hourly does of paranoia. (I'm headed for the scotch)

Originally posted by SkepticOverlord
At one point, it looked like we'd have to "re-roll" all servers which would take days (nearly 40 hours to rebuild our massive database alone).

Yikes!
Glad things didn't turn out that way. Enjoy your drink. I'm sure your nerves need it.

Nice job!

Enjoy the Scotch.

and if performing acrobatics, especially creative ones..It pays to be dressed in tights!

*cheers!*

Great to be back. I would be pulling my hair out if I lost one of my little websites and had to start over. I could not imagine the feeling if something this big were to have to be re-rolled.

Grats to the team, and thanks as always for working hard to get it done fast!

SkepticOverlord called me this afternoon and after the typical "What's up?" from me he said "Not us". My gut tightened and he went on to tell me about the nightmare he was in the middle of with not one, but TWO "outside providers".

Our own S.O. had to teach the technicians exactly how to restore our beloved ATS.

That, after THEY CRASHED the server to the point they couldn't even get back in to correct the conflicting software versions.

At any rate, we've had one HELL OF A DAY here at ATS HQ,and every Member needs to thankful we have our SkepticOverlord!

Springer...

[edit on 9-27-2006 by Springer]

Originally posted by SkepticOverlord
As you all know, we had some downtime today.
I know return you to your regular hourly does of paranoia. (I'm headed for the scotch)

I've been belting back the beer about to go to the whiskey with a threat,
I hope you gave the smack down to these guys, send one over so we can explain

Cheers S.Overlord

[edit on 27/9/2006 by Sauron]

Thanks for bringing us back S.O.!!

That 4 hours made the Internet lonely without ATS.


Toogood

Originally posted by Sauron

I've been belting back the beer about to go to the whiskey with a threat,
I hope you gave the smack down to these guys, send one over so we can explain

Cheers S.Overlord

*cracks knuckles* I'm in.

What scotch do you drink, SO?

Originally posted by spacedoubt

Enjoy the Scotch.

Oh he WILL! It's 25 YEARS OLD! The Overlord is cracking open one of the finest available on a retail basis, that should give some indication of the severity of the day's possible outcomes.

Hell's bells, I'm working up Makers Mark right now my damself!

Springer...

[edit on 9-27-2006 by Springer]

SO!

sounds like the biggest tech nightmare,

Thanks for getting us back up!

Originally posted by SkepticOverlord
(I'm headed for the scotch)

Glenfidich or Glenlivit?

Originally posted by niteboy82
What scotch do you drink, SO?

I have that waiting for me at home.

Well done SO....you really are a superhero. Sorry you had to go through all that....

so glad we're back up again, I thought I was going to have to have an early night!
many thanks SO, hope you enjoy that tipple you have waiting for you, you deserve it

Good man!

Yeah, I wouldn't be drinking a bottle of that except for really really bad days. Tis good though. I usually go a bit less pricey lol. I like McCallan for an good drink. Only takes me the one though.

[edit on 9/27/06 by niteboy82]

Here's what struggling writers can afford to drink to sate their scotch cravings....

yummy

Major kudos and thanks to our resident superhero for saving the day yet again.

You get the 21 thumb salute.

Originally posted by behindthescenes
MY! ATS must pay you well!

ATS?!?!?!?

ATS has never "paid us" ANYTHING, it rarely covers its own expenses and one day like this will set us back months on that score. This man, (SkepticOverlord), Simon's and my partner, handled all of this while running his division at his DAY JOB.

IF ATS ever gets to the point it can simply "pay us back" for investments already made I'd be impressed!

Springer...

[edit on 9-27-2006 by Springer]

Think of the Mastercard commericial theme music...

Having an online community - good.

Having an online community like ATS - better.

Having SkepitcOverlord/Springer/SimonGray protect it - Priceless.

Well-done Amigos 3 - Cheers!

Victor K.

yay!!!

that was a tough wait

and i only waited for an hour