HERE'S WHAT I KNOW, AND WHAT I'VE DONE

I'm sure everyone is wonder what happened over the past two days.

What I know for sure: We were hit with overwhelming traffic.

What I suspect: A possible distributed denial of service attack.

Both of these can look like the other... because a well coordinated DDoS attack typically looks like normal traffic, just heaping hoards of it.

The activity began yesterday at about 8:00 AM when server performance went down, and traffic went up. I immediately became suspicious because in all my time online (a long time), I've never seen traffic spike early monday morning EST!!

Immediately, there were about 100 unique IPs hitting us with typical DOS traffic patterns (kill Apache, wait for their typical connection timeout, start Apache, see who immediately connects). GPan, a Linux expert who upgraded our server was keeping track of new attackers, and also upgraded to the most recent version of our firewall which has some built-in DDoS counter measures.

Yesterday evening, the server was going up and down, requiring a hard reboot each time it went down, since no services were running. Once I got the server working, I kept all services off during the night... the Skeptic Overlord needs his sleep.

Today, an additional problem that contributed to the up-down-up-down seesaw all day... nearly every mySQL database table was seriously corrupt. XMB uses far too many database queries on every page load, and open connections when a server crashes, will very often corrupt the table structure. This added to the server's instability.


I've turned over several log files and suspect IPs to an Agent with the FBI CyberCrimes division here in NYC. While there's very little they can do from a statute point of view (no real financial loss), he assured me that if we can identify the attacker, they will receive a visit from agents and will be expect to provide good answers to hard questions... because he just happened to know of ATS and likes the site... we have a fan in law enforcement.


For now, incoming users must have an existing cookie to be allowed access to the forum (prevents DDoS attacks hitting the more code-complex pages and preserving the databases). I'm going to keep it this way overnight, and try backing off tomorrow.


I'm sure we all can think of who may be behind these attacks. I urge everyone not to jump to conclusions until solid data is discovered. After all, it is still very likely that somewhere, there is a link on a very popular site that has sent us a god-awfull amount of traffic.... it happens. If this is the case, I've optimized Apache to handle a higher level of traffic without failure.


If there are any Linux experts in the house, please contact me by U2U... I'm not a Linux guru and I don't play one on TV. I have some questions about optimizing Apache, mySQL, and Sendmail.


I now return you to your regularly scheduled paranoia and lunacy.

Feew, sure am glad things can get back to normal...oh yeah. Things are normal.

Thank you, William. Excellent work, and excellent detective work!

William, if I havent already said it,

You are kicking a$$ against the a$$holes who are doing this. Kudos! *applause*

Now, if the FBI have thier hands tied on this, I imagine there are a few pissed off ATSers who are willing to pay this guy a visit as well...

Thanks for working so hard to right things...I hope that if any "idiot" proves to be the cause of the problems, may he be punished to the full extent.

Thanks again...
Michael

thats what i thought he was goin to say at first i think the debate competiters should go after him.

I'll pay to see that!

Yeah, I can imagine it now...*dreams off*

Be happy to introduce the offender to the "South Texas Deathride"

Before anyone goes ape$h!t about being violent or anything, its a song!

*cough* AV *cough*

I know South Texas DR.
We can stop in Acuna on the way back for some ice cold Mexican brews, mas tequila shots and have a fiesta with some pretty senoritas!

Good to hear will I thought I might have had somthing to do with this.

www.wired.com...

This is one hell of a nasty virus that has been infecting everything from p2p messages to servers. If I find the actual File name on this I will post it here for people so they can aviod getting this thing.

Falcon


<----puts on Bonnie Tylers song - I Need A Hero for william

Great find Falcon, was a pretty good read. Even if not connected, it might prove wise to develope defences against known threats.

Well maybe one.

How about, "I told you so."

Oh yeah Dragonrider, thanks for mentioning my humane ideals. I really appreciate it.

I really have a lot to say, but I will once again sacrifice my senses and unreasonable crusadism for the insanity of the board. Wouldn't want to create any more conspiracy then we already have.

William has done a great job, and I think that he deserves to be respected. Therefore I say we take no prisoners. Yeah you heard me. This means war. We can't be stopped. ATS will prevail. Watch out corruptors, we are now ready to mobilize the the shock troops. No one ever said that my actions were stopping them. It is funny that I am considered a pacifist on this matter, when in fact I was the most hardened on the issue. Oh well, I guess that proves that no one listens to me or that I just didn't make myself clear enough.

That is okay, the stability of the board was at risk due to my so-called over-reacting rationalisms.

That is understandable.

Like Nans told me, I am the boy who cried wolf.

So as long as the ball is rolling so will the heads.

William...................

I don't know if you received my e-mail but I would like to know if you had or not, and I want to ask if I can post my reply to the debate later when I have a better connection.

I don't know if it is just my system or what, but I seem to be having much trouble getting a good connection speed here. It took around 20 minutes for that last post to load, and I would expect that this one will take about the same time. It would be great if I could wait until I have better luck with the board before I continue the debate. That is just a request not a demand. If you cannot do so, then so be it.

Also, what is a cookie???

Once again, thank you William.

To all:


So is anyone else experiencing extremely slow connections here, or is it just me???


Abraham

I was also going through some very slow times trying to connect here. I noticed it only had effect while I was attempting to access ATS. Don't worry, things should get back to normal soon.

And a cookie is a small piece of info, like a ticket stub. Best I can do to explain it.

Thanks William for keeping us imformed.

I dont have any cookies stored on my computer, i ate them all. lol

Pizza if you don't have any cookies in your computer it won't work properly.

So open that coffee cup holder in the front of your computer and put a cookie in the round hole and close it again, that way your computer will be equipped to run properly..

WOHOOO!!!!

Finally got on today after almost 48 hours!!!!

William, i have the exact same problem as Abraham on the debate...due to me at the hospital for nearly 14 hours a day i can't get access to the computer very often?

I also have another problem...will u2u you about it as i need to know what can be done about it?

Going to be a pain in the ass but i'll send a e-card from the middle east for y'all!

Thanks for the e-mail yesturday William, nice to find out whats going on, when we cant get in!

Well done for all the hard work you are putting in to keeping the site running, give yourself a pat on the back bab!

I have been having alot of problems lately with log-in and then loading posts, very slow and shakey screen sometimes (i think that maybe me though?)

Once again William, keep the good work up!!!!



blackwidow

Hey Blackwidow

It's not you...had that problem for awhile..if the screen gets shakey what i do is hit F5 (reload) and it sorts it out!

I never got on yesterday at all...didn't help that i found out i'v been pulled for medic service in egypt!