Do you know how spiders find new pages? Is just chance by following links? Or do they some how grok the server directory structure and/or get a listing in some manner? Or both?

Byrd, thanks for this link. Its appreciated.

Thanks great links.

I always wanted to access real information that I know is on the web but unaccessable through most known search engines.

Have to agree that EEVL with the egyptian eye is pretty freaky. Horus worshippers.

Originally posted by ktprktpr
Do you know how spiders find new pages? Is just chance by following links? Or do they some how grok the server directory structure and/or get a listing in some manner? Or both?

A bit of both depending on who's doing it. Search engine bots are generally "well-behaved" and if they encounter a page with "nofollow" in the metatags, they'll leave it alone (and ditto "noindex.")

So you won't find these pages on the big directories.

...now, a "rude spider" (generally run by humans and not an automatic spidering) will run in and finger everything it finds.

Ummm...Byrd...

How long have you been keeping this info secret for yourself?...

Not so interesting... I've done some search on the topics related by the engine and I had no really good results...

Are you sure this not only search on the database of the website hosting the engine ?

Originally posted by MidnightDStroyer
Ummm...Byrd...
How long have you been keeping this info secret for yourself?...

I could tell ya, but somebody would come by and thwack you.

...actually, I do a bit of reading on search engines and was idly contemplating doing a poster presentation for an anthropology conference on the hidden web and research techniques.

Thanks Bryd and CPYKOmega.....
BOOKMARKED!


regards
seekerof

Strange, neither www.com or the first search engine posted find ANYTHING for Anaemic (my band).
I'd have expected a fair few medical sites at least.

Very odd...