White House website directing spiders NOT to catalog Iraq statements

Just saw this on another board. Not that this should be a suprise to anyone, but the WH has disallowed most mentionings of Iraq in the robots.txt file. This file is used like a directory for search engine spiders when they come to catalog a site. Also note when looking at the txt file that the State of the Union from 2002 & 2003 is also disallowed.

Perhaps the White House doesn't want to make it easy for people to compare its older statements about Iraq with current realities -- though that doesn't explain why the pages are searchable on the White House site itself. Maybe, then, the White House wants to know who's looking for these things (e.g. by tracking IP addresses of people who query the government site).

Either way, the blocking of search engines is a bad idea, and fundamentally an abuse of the public trust.

weblog.siliconvalley.com...

www.whitehouse.gov...

they've taken away so many rights, and you still see a majority supporting the bush administration..
thank god for re-elections...
but then what gurantees there wont be another lout instead of the current oaf?
freedom of speech for freedom of search ai?

This is really shocking. I think a lot of people here don't quite understand what a move this is. First, it acknowledges the power of the internet as a social network. Second, it shows this adminstration is grasping straws. Third, it also redefines their secrecy policy to absurd levels (in a way).

I wish I were debating PNAC right now cause those guys are going down, and really hard too. Rumsfeld, Wolfwitz, all the NeoCons are going outta style.

the current administration is in full on cover up mode...

That is blocking the freedom of speech!

I guess their thoughts are,..

they can make it hard to find.

I definetly do not like the idea that the spider's can be 'directed' by any other than the spider source (engine), for catalog redundancy elimination only, when it comes to public information.

That is adjusting public opinion, it seems, since the public opinion is usually biased by what they read!

Fortunatly for us, many engine spiders have been known to ignore these type of instructions.

(edit:actually the robots.txt file only applies to the domain directories under whitehouse.gov to prevent the spiders from cataloging info on the wh website only) Not that I feel any better about limiting what google spiders when it visits there.




[Edited on 27-10-2003 by smirkley]

www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...
www.whitehouse.gov...

Are among the files excluded by robots. Some argue taht these files don't really matter, but if you actually look at any of the above links you'll find that they very much do matter. Also, put "Iraq site:whitehouse.gov" into google and you get 14,000 hits.

The first hit is the root directory of /infocus/iraq
So they didn't block access very well.

read yro.slashdot.org...
for more info. Specifically post yro.slashdot.org... points out just how revisionary this adminstration is...

check out the file for yourself: www.whitehouse.gov... Notice how many press releases are cut out ... and what those releases point to. I do believe a poker hand has been slightly revealed...

EDIT- this "investigative" work was compiled from slashdot, from the link given above. The above dirs are in the robot.txt file for all to see.

[Edited on 27-10-2003 by ktprktpr]

[Edited on 27-10-2003 by ktprktpr]

I'm hoping that someone (not me), will start to spider the whole WH site on a daily basis or something close to it and start to look for patterns of deletion or modification of existing files.

Frankly kt..I'm a little suprised by your investigative reaction to this news. You've expressed certain paranoias about this type of work.

For someone with the resources and know how, a spider can spider a site, ignoring the robots.txt or any other robot meta tags as such,...but of course all visits...including spiders,...are generally ID'd and logged...and being it is the WH, I imagine a little bit of an elaborate ID would occur.

That's an asute observation Kukla. You made me realize that my source wasn't clear enough so i re-edited my post. The links above can be found at slashdot.org. I just added several ideas.

As for spidering WH it's already done adn has been done a long time. Skim the slashdot page for a link to WH archives or just type in white house web site archives into google.

[Edited on 30-10-2003 by kukla]

hd on kukla.... here you go: (from slashdot.org again)

web.archive.org...://www.whitehouse.gov/
web.archive.org...
Jul 13

(that's from web.archive.org, so you can refer to it for other info)

and...

web.archive.org...://www.whitehouse.gov/
web.archive.org...://www.whitehouse.gov/robots.txt
Sept 13

Compare and contrast, like from 7th grade

I just saw that and was preparing to post it...

I'd like to know when the WH kicked of that new website? 9/13??

www.differentstrings.info...

www.washingtonpost.com...

Bush and crew has been dabbling in historical revisionism for a while.

Good post.

I was thinking about asking to start a research project on the lead up to the war in Iraq.
Basically collecting links to important stories under various headings like Genesis of the Coalition and Resolution 1441.In an effort to establish a timeline of facts which might give a better indication of motives.As time goes on people forget and issues become blurred.Ask people today what it was all about and it would not reflect accurately the true history.
When was the last time you heard a government official voluntarily mention the subject of WMD's yet only one year ago few pronouncements on Iraq did not contain references.Perceptions are being subtley changed.