Originally posted by Skyfloating
Whats always frustrated me is if I look for, lets say "Atlantis" related items I first get tons of stuff about Atlantis - The cartoon, Atlantis -
the Holiday Resort, and have to minus all kinds of things out before even getting close to starting research. Your Library Index seems to help with
There's daily spiders or robots that search pages and index them, then there are the less frequent deeper robots that take the entire site into
account, relate it to others, and rank them.
If you are searching "Atlantis" and say you want the mystery's about it, then add that word mystery, or add key words that relate. Be specifc, as
if asking it a question. The search engine isn't psychic! Google will more than likely be displaying the results of whoever called their site
"atlantis" or had the word "atlantis" more times on the page result than any others.
New sites will take longer to be recognized and indexed and should be submitted to every search engine and directory to speed things up, but not every
new webmaster is aware of that. They also may have a good site, but without any SEO expertise, their pages will be missed. Not knowing how to write
error-free search engine friendly html and overloading of pages which may cause the page to be missed when the spider visits is a prime reason.
Also some sites get banned and removed for a variety of reasons, and can even just plain vanish off the searches for months. Then you have sites or
pages who don't want to be indexed, like RATS forum.
Google usually searches first for the page title, then content and popularity (has most sites linking to it). Yahoo searches differently, as does msn
and others. Google changes the algorythm so smart webmasters can't figure out the secret formula to getting their site on the first pages to come up.
With the competitiveness of optimizing a site to get the best ranking, it buries some good pages so deep, you'll never find them.
I would say the dis-satisfaction at finding pages you want is a combination of the spidering algorythms and the webmaster not learning how to optimize
their site for google to index and rank - and ones that do know how and bump other sites down the list.