I had some unusual web traffic on my website yesterday, and when I investigated, I found that my entire site had been crawled by a spider that doesn't honor the robots.txt. The name given in my server logs was kintiskton llc and the visits came from the following IP addresses:

65.208.151.112
65.208.151.113
65.208.151.114
65.208.151.115
65.208.151.116
65.208.151.117
65.208.151.118
65.208.151.119

At first I just assumed that this spider was either searching for vulnerabilities for hacking, or was seeing if there were any email forms to exploit. After I researched this spider, though, I noticed that the vast majority of the people that were hit were blogs. This got me thinking, why would someone need a comprehensive database of all the pages in a website, even those that the site owner wishes not to be displayed to the general public. Do you think this could be the NSA or homeland security searching for information on possible terrorist attacks? Is it another government?

You would think that if this were a government probe that they would make sure to keep the traffic invisible. Maybe the fact that it isn't is a sign of the times. Am I just being overly paranoid?

I'm not real technical with IP stuff. Did you notice a common catch phrase in the blogs or topics covered ie."ufo" or are these random pings?

The way I found to block these guys are to deny Access through the htaccess. I just blocked them and if it was a lawless spider, I won't allow it access!!

Originally posted by Anonymous ATS
The way I found to block these guys are to deny Access through the htaccess. I just blocked them and if it was a lawless spider, I won't allow it access!!

I have done that, and so far it has worked like a charm (at least until they change IP addresses )

My biggest question isn't how to stop them, but why are they doing what they are doing? Why would anyone who is not a search engine want a comprehensive database of every page on the internet. There are much more efficient methods for hackers to discover exploits, and there are easier ways for a spammer to get a hold of email addresses and email forms.

Maybe this isn't the government searching for terrorist keywords as much as a new form of vandalism, one that wrecks any sort of analytics or testing metrics that you might be running.

[edit on 23-2-2009 by Bullfrog]