Deep Web, Secret Web, Second Web

I post this to clear up some confusion for myself and maybe for others relating to the internet. I trust there are people here more expert than me who can shed some light on these issues.

A few weeks ago I was sitting on an airplane flipping through a computer magazine which had an article about google. In this article they said that google only finds 2% of all the information in the internet.

"What????" I thought. I was under the impression that it covered at least 50%, if not more. In the last 10 years our society has been taught that ALL information can be found on the internet or at google, and people have bought into it.

The same article then went on to explain that "deep web" engines such as this one - Completeplanet - could find up to 20% of all internet content.

This was the very first time I ever heard of a "deep web".

I always realized that not all information can be found on the internet (at least not yet), but I was shocked to hear that not even most of internets information can be found by search engines.

Why is this important? Because here we are, at ATS, believing we have access to any and all information at the push of a button.

Two related items:

What ever became of "Internet 2" and what exactly was that supposed to be?

And on a speculative note: Do you think it is possible or viable to establish a sort of "secret internet", or where do you think internet will be heading in the next 50+ years?

I am actually a web developer and haven't ever heard of it myself! I am definitlely goingto check out that deep web search engine though that sounds very elite. Thanks for sharing! I have much more exploration to do!

May be deep web, but that search engine you posted SUCKS!!!!!!!!! I am SO dissapointed

Just something I noticed that is kinda funny. The deepweb site has Google adsense going down the side. Kinda ironic huh?

I never used it before. Dont even know what to use it for. I wish I had kept that article, damn it.

I remember the article saying that "deep web" engines apparently have acess to databases, and that normal search engines, such as google only have access to the "surface web".

As I said...its all new to me too. I´m looking for answers.

reply to post by Skyfloating

---

not sure about the deep web stuff. but, I use a meta-search engine. dogpile.com is my favorite. it searches multiple search engines at one time, and displays what engine it was found on.

a clip form their site:

Hi

The deep web or invisible web as it is called is actually all the information held on databases which search engines cannot access directly. So to access the info you need gues what, software. I use Copernic excellent and useful, as to your other points the military used the internet then we got access why? The military have newer technology which will be handed down in due course like older sibling clothes. Google this and you may be suprised. SIPRNET is a start.

50 years well heres hoping we are still here, if so then the current net will be like black and white tv compared to HD and blue ray. Holographic interfaces and Virtual world similar to matrix, johny mnemonic type special effects will be common place.

Anyway hope this helps.

On that page I found a link detailling what the "deep web" compared to the "surface web" is:


Deep Web

internet2 is a new network that links (for the most part) universities. it's extremely fast. we have a net2 connection where i work and i can pull stuff from other universities at several mb/s.

reply to post by Munro_DreadGod

---

yeah, thats helpful. so you can actually confirm that without certain software, we are only able to scratch the surface?

Why doesn't someone contact Al Gore about internet 2 and the deep web, surely he's in on the development of it....


I googled Internet2, the article I was reading stated it was bigger and faster, well if we're using the same computers and the information is accessed by the same connetions, how would it be faster ?

Well I will explain a little.

The informatiion held on a library catalogue can be accessed online.
That information (the records) however cannot be googled, google can find the catalogue search engine but not the records held on the catalogue. This is the same for all database websites - so if certain information is stored in a database online you can search for it using the search interface for the database.

I cannot for one minute think that anything secret would be accessible from the net, that would be very negligent of the organisation. However if you could crack or get access to other networks you may find something bearing in mind in would be a crime and one guy (UK citizen) is facing 70 years in jail for doing just this, so it is a risk.

reply to post by elevatedone

---

Better infrastructure and less takeup of bandwidth by millions of users - private network basically (Universities, government, etc)

Alright. Questions answered. Thread completed.

Thanks Munro_DreadGod.

Here's a site you might find interesting-
The invisible Web Revealed.

One of the things I find most frustrating isn't that too little information is returned, but a search request will return too much drek to wade through.

I like to use the Librarians' Internet Index to cut down on some of that.

reply to post by yeahright

---

Thats a pretty good reference page there.

Whats always frustrated me is if I look for, lets say "Atlantis" related items I first get tons of stuff about Atlantis - The cartoon, Atlantis - the Holiday Resort, and have to minus all kinds of things out before even getting close to starting research. Your Library Index seems to help with that.

reply to post by Skyfloating

---

Yes, I think it's a great place to start a search. It eliminates all sorts of unrelated foofery and less-than-ideal sources. Eventually, it's a good idea to broaden out, but up front, I like the Librarians' Index very much.

Glad you did, too.

content deleted as subject was clearly defined and answered.


[edit on 12/19/07 by surfinguru]

[edit on 12/19/07 by surfinguru]

Good thread.

Some of the stuff that isn't initially available via ordinary Google searches can be found if you use better search terms. Most people know how to look in unprotected directories using "/index of" in the search box.

Google also isn't very good at searching ftp sites, but there are other engines that are specifically designed to do that. One of the better ones no longer exists, that I'm aware.

Back in the early 90s there was a competition held periodically in which a very obscure fact was sought and the 'Net wizards would search for it and try to find it with the least possible branches. (forget the name).

Anyway, yeah, I'm surprised if it's listed as 2%.

i just tried an "/index of" in a google search and compared it to a search on the same topic without putting in the "/index of" and i was amazed, i had no idea! thank you VERY much for sharing this with me. if you have any other tips along the lines of that one i would love to see a post on them. again thanks for teaching! salutations.