White House website directing spiders NOT to catalog Iraq statements, page 1

White House website directing spiders NOT to catalog Iraq statements, page

Pages:
ATS Members have flagged this thread 0 times

Topic started on 27-10-2003 @ 04:56 PM by kukla

Just saw this on another board. Not that this should be a suprise to anyone, but the WH has disallowed most mentionings of Iraq in the robots.txt file. This file is used like a directory for search engine spiders when they come to catalog a site. Also note when looking at the txt file that the State of the Union from 2002 & 2003 is also disallowed.

Perhaps the White House doesn't want to make it easy for people to compare its older statements about Iraq with current realities -- though that doesn't explain why the pages are searchable on the White House site itself. Maybe, then, the White House wants to know who's looking for these things (e.g. by tracking IP addresses of people who query the government site).

Either way, the blocking of search engines is a bad idea, and fundamentally an abuse of the public trust.

weblog.siliconvalley.com...

www.whitehouse.gov...

reply posted on 27-10-2003 @ 06:23 PM by Cyrus

they've taken away so many rights, and you still see a majority supporting the bush administration..
thank god for re-elections...

but then what gurantees there wont be another lout instead of the current oaf?

freedom of speech for freedom of search ai?

reply posted on 27-10-2003 @ 06:27 PM by ktprktpr

This is really shocking. I think a lot of people here don't quite understand what a move this is. First, it acknowledges the power of the internet as a social network. Second, it shows this adminstration is grasping straws. Third, it also redefines their secrecy policy to absurd levels (in a way).

I wish I were debating PNAC right now cause those guys are going down, and really hard too. Rumsfeld, Wolfwitz, all the NeoCons are going outta style.

reply posted on 27-10-2003 @ 06:27 PM by herm

the current administration is in full on cover up mode...

reply posted on 27-10-2003 @ 06:34 PM by smirkley

That is blocking the freedom of speech!

I guess their thoughts are,..

they can make it hard to find.

I definetly do not like the idea that the spider's can be 'directed' by any other than the spider source (engine), for catalog redundancy elimination only, when it comes to public information.

That is adjusting public opinion, it seems, since the public opinion is usually biased by what they read!

Fortunatly for us, many engine spiders have been known to ignore these type of instructions.

(edit:actually the robots.txt file only applies to the domain directories under whitehouse.gov to prevent the spiders from cataloging info on the wh website only) Not that I feel any better about limiting what google spiders when it visits there.

[Edited on 27-10-2003 by smirkley]

reply posted on 27-10-2003 @ 08:35 PM by ktprktpr

www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...

Are among the files excluded by robots. Some argue taht these files don't really matter, but if you actually look at any of the above links you'll find that they very much do matter. Also, put "Iraq site:whitehouse.gov" into google and you get 14,000 hits.

The first hit is the root directory of /infocus/iraq
So they didn't block access very well.

read yro.slashdot.org...
for more info. Specifically post yro.slashdot.org... points out just how revisionary this adminstration is...

check out the file for yourself: www.whitehouse.gov... Notice how many press releases are cut out ... and what those releases point to. I do believe a poker hand has been slightly revealed...

EDIT- this "investigative" work was compiled from slashdot, from the link given above. The above dirs are in the robot.txt file for all to see.

[Edited on 27-10-2003 by ktprktpr]

[Edited on 27-10-2003 by ktprktpr]

reply posted on 27-10-2003 @ 08:51 PM by kukla

I'm hoping that someone (not me), will start to spider the whole WH site on a daily basis or something close to it and start to look for patterns of deletion or modification of existing files.

Frankly kt..I'm a little suprised by your investigative reaction to this news. You've expressed certain paranoias about this type of work.

reply posted on 27-10-2003 @ 08:57 PM by smirkley

For someone with the resources and know how, a spider can spider a site, ignoring the robots.txt or any other robot meta tags as such,...but of course all visits...including spiders,...are generally ID'd and logged...and being it is the WH, I imagine a little bit of an elaborate ID would occur.

reply posted on 27-10-2003 @ 08:57 PM by ktprktpr

That's an asute observation Kukla. You made me realize that my source wasn't clear enough so i re-edited my post. The links above can be found at slashdot.org. I just added several ideas.

As for spidering WH it's already done adn has been done a long time. Skim the slashdot page for a link to WH archives or just type in white house web site archives into google.

reply posted on 27-10-2003 @ 09:06 PM by kukla

[Edited on 30-10-2003 by kukla]

reply posted on 27-10-2003 @ 09:15 PM by ktprktpr

hd on kukla.... here you go: (from slashdot.org again)

web.archive.org...://www.whitehouse.gov/
web.archive.org...
Jul 13

(that's from web.archive.org, so you can refer to it for other info)

and...

web.archive.org...://www.whitehouse.gov/
web.archive.org...://www.whitehouse.gov/robots.txt
Sept 13

Compare and contrast, like from 7th grade

reply posted on 27-10-2003 @ 09:23 PM by kukla

I just saw that and was preparing to post it...

I'd like to know when the WH kicked of that new website? 9/13??

reply posted on 27-10-2003 @ 10:15 PM by ktprktpr

www.differentstrings.info...

www.washingtonpost.com...

Bush and crew has been dabbling in historical revisionism for a while.

reply posted on 28-10-2003 @ 10:10 AM by John bull 1

Good post.

I was thinking about asking to start a research project on the lead up to the war in Iraq.
Basically collecting links to important stories under various headings like Genesis of the Coalition and Resolution 1441.In an effort to establish a timeline of facts which might give a better indication of motives.As time goes on people forget and issues become blurred.Ask people today what it was all about and it would not reflect accurately the true history.
When was the last time you heard a government official voluntarily mention the subject of WMD's yet only one year ago few pronouncements on Iraq did not contain references.Perceptions are being subtley changed.

Pages: ^^TOP^^

Now a witness to Breitbart\'s death vanishes
Posted 17 days ago with 6 member flags

Cuba: Judge Rejects Efforts to Open CIA Volume on Bay of Pigs
Posted 13 days ago with 4 member flags

Meet Soros-Funded Domestic Terrorist Brett Kimberlin aka "The speedway bomber"
Posted 1 days ago with 4 member flags

US Official Detained for Donation to Al Qaeda - Anonymous Involved
Posted 13 days ago with 2 member flags

Left wing political thugs use criminal tactics to intimidate and bankrupt law abiding conservatives
Posted 0 days ago with 2 member flags

Newest topics, updated in real-time:

I was a Mason and here's what I learned...
Secret Societies: 9 minutes ago

DEFCON - "That's not a glitch sir!"
Video Games: 20 minutes ago

The founding fathers of the US, and other great minds, speak on bankers and paper currency
General Conspiracies: 20 minutes ago

Ooparts and Ancient aliens
Aliens and UFOs: 33 minutes ago

Androgyny as a Non Gender Spiritual Concept
Philosophy and Metaphysics: 46 minutes ago

Asthma Sufferers, What will we do when the SHTF?
Health & Wellness: 1 hours ago

Pentagon Armies of Paid Trolls
General Conspiracies: 1 hours ago

The Most Gangster Description of Those Who Control The World, Illuminati.... Raekwon and ODB.
General Conspiracies: 1 hours ago

Newest topics getting flags, in real-time:

Undercover US agents brought down our new Superjet: Russia’s extraordinary claim about crash which
General Conspiracies: 14 hours ago, 28 flags

Before There Was Welfare There Was Charity
US Political Madness: 12 hours ago, 21 flags

AFTER MIDNIGHT: 70: So We're Different Colours...
ATS Live: 8 hours ago, 21 flags

Accept it!
Rant: 17 hours ago, 18 flags

7-year-old's suicide shocks Detroit community
Social Issues and Civil Unrest: 17 hours ago, 17 flags

One of these politicians is not like the others
Politicians & People: 15 hours ago, 16 flags

Reddit Founder And Activists Aim To Build A 'Bat-Signal For The Internet'
Breaking Alternative News: 13 hours ago, 15 flags

Honors student arrested for truancy...it gets worse.
Social Issues and Civil Unrest: 9 hours ago, 12 flags

Newest topics getting replies, in real-time:

Before There Was Welfare There Was Charity
US Political Madness: 12 hours ago, 160 replies

The Afghanistan Exit Will Humiliate The United States Worse Than Vietnam
World War Three: 15 hours ago, 126 replies

Updates - Answers from an Alien from Andromeda
Aliens and UFOs: 16 hours ago, 93 replies

7-year-old's suicide shocks Detroit community
Social Issues and Civil Unrest: 17 hours ago, 91 replies

Japanese man cooks, serves own genitals
Global News Discussion: 16 hours ago, 83 replies

Obama Twice Mistakenly Mentions ‘My Sons,' While Defending Contraception Mandate
US Political Madness: 7 hours ago, 83 replies

say BYE BYE to ice cream
General Conspiracies: 9 hours ago, 65 replies

DIY lets cure cancer simple as that
Medical Issues & Conspiracies: 13 hours ago, 62 replies