For the Researchers: Searching the Invisible Web

We all think of Google as the largest search engine -- and that's true, up to a point. But it brings back only web pages and doesn't actually search the much larger (up to 5 billion) pages of information in various databases (library catalogs, for instance.)

This huge set of internet resources (not available to most search engines) is called "The Invisible Web."

One of the big search hubs is called "RDN." The RDN is a collaboration (metasearch engine) of over seventy educational and research organisations, including the Natural History Museum and the British Library and is found at: www.rdn.ac.uk...

It even searches the amusingly named "eevl" www.eevl.ac.uk... (which is an engineering search engine)

Do you know how spiders find new pages? Is just chance by following links? Or do they some how grok the server directory structure and/or get a listing in some manner? Or both?

Byrd, thanks for this link. Its appreciated.

Thanks great links.

I always wanted to access real information that I know is on the web but unaccessable through most known search engines.

Have to agree that EEVL with the egyptian eye is pretty freaky. Horus worshippers.

Originally posted by ktprktpr
Do you know how spiders find new pages? Is just chance by following links? Or do they some how grok the server directory structure and/or get a listing in some manner? Or both?

A bit of both depending on who's doing it. Search engine bots are generally "well-behaved" and if they encounter a page with "nofollow" in the metatags, they'll leave it alone (and ditto "noindex.")

So you won't find these pages on the big directories.

...now, a "rude spider" (generally run by humans and not an automatic spidering) will run in and finger everything it finds.

Ummm...Byrd...

How long have you been keeping this info secret for yourself?...

www.archive.org...


This is a site where you can search over 30 BILLION web pages dating back to 1996.


Have fun


Oh and all you have to do to search is sign up with a simple user name and password to get your virtual library card.


Peace, Omega

[Edited on 10-30-2003 by CPYKOmega]

Not so interesting... I've done some search on the topics related by the engine and I had no really good results...

Are you sure this not only search on the database of the website hosting the engine ?

Originally posted by MidnightDStroyer
Ummm...Byrd...
How long have you been keeping this info secret for yourself?...

I could tell ya, but somebody would come by and thwack you.

...actually, I do a bit of reading on search engines and was idly contemplating doing a poster presentation for an anthropology conference on the hidden web and research techniques.

Technically, to do a good search on the web, you should start by the root...

Try this site :

www.com...

(Why there are some search engine which bring back always only porn sites results ?)

[Edited on 30-10-2003 by Nans DESMICHELS]

Thanks Bryd and CPYKOmega.....
BOOKMARKED!


regards
seekerof

Strange, neither www.com or the first search engine posted find ANYTHING for Anaemic (my band).
I'd have expected a fair few medical sites at least.

Very odd...