posted on Feb, 21 2009 @ 04:23 PM
I had some unusual web traffic on my website yesterday, and when I investigated, I found that my entire site had been crawled by a spider that
doesn't honor the robots.txt. The name given in my server logs was kintiskton llc and the visits came from the following IP addresses:
At first I just assumed that this spider was either searching for vulnerabilities for hacking, or was seeing if there were any email forms to exploit.
After I researched this spider, though, I noticed that the vast majority of the people that were hit were blogs. This got me thinking, why would
someone need a comprehensive database of all the pages in a website, even those that the site owner wishes not to be displayed to the general public.
Do you think this could be the NSA or homeland security searching for information on possible terrorist attacks? Is it another government?
You would think that if this were a government probe that they would make sure to keep the traffic invisible. Maybe the fact that it isn't is a sign
of the times. Am I just being overly paranoid?