The Glory of BIG DATA

This just blew my mind and I felt compelled to share it here at ATS.

The amount of data available to us is increasingly vast. In 2010 we played, swam, wallowed, and drowned in 1.2 zettabytes of the stuff, and in 2011 the volume is predicted to continue along its exponential growth curve to 1.8 zettabytes. (A zettabyte is a trillion gigabytes; that’s a 1 with 21 zeros trailing behind it.) The IDC Digital Universe study from which I’ve plucked these numbers helpfully notes that if you were inclined to store all that data on the hard drives of 32-gigabyte iPads, doing so would require 57.5 billion devices—enough to erect a 61-foot-high wall 4,005 miles long, from Miami all the way to Anchorage.

www.popsci.com...

Processor or Virtual Storage
1 Bit = Binary Digit
· 8 Bits = 1 Byte
· 1024 Bytes = 1 Kilobyte
· 1024 Kilobytes = 1 Megabyte
· 1024 Megabytes = 1 Gigabyte
· 1024 Gigabytes = 1 Terabyte
· 1024 Terabytes = 1 Petabyte
· 1024 Petabytes = 1 Exabyte
· 1024 Exabytes = 1 Zettabyte
· 1024 Zettabytes = 1 Yottabyte
· 1024 Yottabytes = 1 Brontobyte
· 1024 Brontobytes = 1 Geopbyte

For perspective, 200 terabytes is 20 times the size of the Library Of Congress.
I know advances in tech are going to blow our mind and it is my belief that technology may be our salvation to some degree. Naturally there will be some problematic issues, but like the advent of the interwebz, good and bad will come, but a paradigm change will certainly manifest.
This article is specifically about data storage and transfer, which are the foundation of advancing tech, and it seems Moores Law has become antiquated by recent advances. Quantum computing and shrinking hardware are taking us into new realms of potential and existence.

In other words, in this new era—the transition from digital code to digital-plus-life code—the capacity to generate data exceeds our capacity to store and process it. In fact, life code is accumulating at a rate 50 percent faster than Moore’s Law; it at least doubles every 12 months. Without extraordinary advances in data storage, transmission and analysis, within the next five years we may simply be unable to keep up.

The first part of the article mentions a Google site called NGram Viewer Here, and has collected data on books, subjects and reading uptake. The example they use is the word sex, but putting in different subject words reveals interesting evolutions of topics read by people, and other data gathering results.

One tiny part of that vast wall would house Google’s effort to create as complete a census as possible of the published word since 1500. The company has already gathered enough data—some 500 billion words from more than five million books—to plausibly claim the emergence of a new science, culturomics. Eventually the coinage, evolution and decline of every word and phrase could be traced across centuries. Using Google’s handy Ngram Viewer, we can already observe the explosion of the word “sex” after 1960. Or watch Rembrandt citations gradually grow, exceeding those of Cezanne in 1940, only to witness Picasso blow past both of them less than a decade later. These are not scholarly samples and inferences drawn painstakingly from a few great books; this is the exacting examination of how a word or phrase’s spelling and use actually mutated year by year.

"Culturomics?" Wow, this is a unique and fresh perspective to view things. I have often wondered if the day will come when the entire span of human knowledge could be put on a pinhead, and it sounds like we are headed that way. What does this mean?
The 2nd part of the article describes some of the details surrounding 'Big Data.'

So this is the paradigm shift whose fruits I witnessed in presentation after presentation at TED: a shift from a world of data sampling and extrapolation to one in which all data in a given realm can be collected and analyzed. That is Big Data.
AND BIG DATA is about to get much, much bigger, as we enter an era in which digital data merges with biology. This synthesis of codes takes the abstract world of digits and brings it back into the physical world. We of course know quite a bit about how life is expressed—in the four letters of DNA, in more than 20 amino acids, in thousands of proteins. We can copy life through cloning. Now we are beginning to be able to rewrite life, not just gene by gene, but entire genomes at a time. This is the difference between inserting a single word or paragraph into a Tolstoy novel (which is what biotechnology does) and writing the entire book from scratch (which is what synthetic biology does). It is far easier to fundamentally change the meaning and outcome of a novel, seed, animal or human organ if you write the entire thing.

Man are we ready for this stuff? It seems to make us appear infantile in our development, with a long ways to go. I mean this stuff is the source of our sci fi future, and it may take us beyond our imagination. I just thought about how it will affect us not only here on the planet, but in our space exploration endeavors, as well as exploring 'inner space.'

In 2008, three scientists—Venter, Hamilton Smith and John Glass—and their colleagues took a basic gene sequence from a computer, programmed robots to pick the four chemicals that make up DNA from jars, and assembled the world’s largest organic molecule. They then developed techniques to insert this new molecule into a cell. Bottom line, they programmed a cell to become a different species. Some called it the world’s first synthetic life-form. It is really the first fully programmable life-form. And it reproduces.
Programmable cell platforms are like computer chips. They could eventually be designed to help create or do anything, if you figure out the right code for what you wish to make.

More links on data progression:
www.popsci.com...

Wow, wonderful and scary too! This stuff makes my head and imagination spin. Will technology enable us to meet all of our needs so that we no longer have to play the part in our current economic servitude?
Maybe tptb realize this and are desperately gathering power, ownership and assets to prepare for a great equality? Am I fantasizing here, will this tech help us all and not just those who can afford to use it? Are we at the doorstep of outgrowing multiple paradigms? Can we even imagine the end result? Interesting times, that is for sure.

Peace and imagination,
spec

I think things will really take off once Personal Identification and the majority of household appliances go digital and begin to transmit information.

Absolute statistics are something every company in the entire world is crazy for. When your toaster is constantly telling it's manufacturer how often you use it, how you use it, and how good of a job it is doing - it takes R&D to a whole new level.

reply to post by maus80

That will be the day I rid myself of a toaster!
It does seem inevitable that these advances will become a vehicle for marketing, message and even surveillance by outside entities, because it can easily be done. Maybe these advances could enable live, full time, widespread surveillance, with visual and audio, thermal imagery, etc. The processing and storing capacity may be able to record everybody, all the time, everywhere.

I hope though, that these advances may lift social, psychological and physical health, as well as enable us to gain independence of self. Space exploration, physics, music and art, may all accelerate and find new ground, hopefully. I can't seem to think of anything else that will ever change our direction like technology at this point.
Yea, my glass is half full...today

Peace,
spec

ETA: Man I despise marketing that is force fed to our senses, and it will just push me more towards a simpler life, with less distractions, even if that means an empty room in the hills with just my dog, some books, a guitar and fishing rod. And maybe a solar powered charger for my ATS

Give a man a backyard shed, and he will fill it up.
Give him a bigger one and he will fill it too.

I wonder how much of that 1.2 zettabytes is actually useful?
Look at the snippets that get copy/pasted to forums all over the world.

If there was 1 central repository, which held 1 compressed copy,
that was automatically (and correctly) translatable...
It would probably be a 1/3 or less of that 1.2 zbytes.

reply to post by CitizenNum287119327


But what will his shed be filled with? We have a choice, but yea I here ya, our nature to collect and consume is deeply ingrained and you make a good point.
This is so much data that who knows where it will lead, but it is an inevitable part of our growth and still leaves me in awe and with hope.

Peace,
spec

Thought I would add this related info about super computer the Hopper.

Vital Stats: System: Cray XE6 supercomputer. Theoretical peak performance: 1.3 petaflops (1.3 quarillion calculations per second). Inside: 153,216 compute cores connected by a very fast interconnect. Supported by a High Performance Storage System where users archive their data. The HPSS system has a peak capacity of 59 petabytes. On Oct. 26, some 98,750 files were transferred into or out of the HPSS system.

Sounds fast huh?

Fusion energy is a technological holy grail, and today Hopper is on the hunt for it.

www.popsci.com...
Glad to hear this, let's just hope the results are shared and utilized and not hidden from public awareness. This sounds like the tool that will bring us closer to the next paradigm in energy, computing and who knows what else. Makes one wonder where we will be at in another 10, or even 5 years!

Peace,
spec

Wanted to add this report on Quantum Computing and Quoogling:

Seth Lloyd on Quantum Computers

Popular Science: How are quantum computers different from ordinary ones?

Seth Lloyd: Quantum computers operate at the smallest, most fundamental levels allowed by physics. On a regular computer, a single bit of information is represented by a whole bunch of electrons. In a quantum computer, you store bits of information on the most elementary particles. So a "qubit" might be represented by a single electron.

Why is a smaller bit better?

At the quantum-mechanical level, an electron can be here and there at the same time. And if you're here and there, you can do this and that simultaneously.

PS:How is it different from regular computers with multiple processors?

On a regular computer, a bit can be 0 or 1. Electrons over there means 0; electrons over here means 1. On a quantum computer, a bit can be 0, 1 or both. So in your quantum computer, one qubit means two things at once, or with two qubits, four things at once, or three qubits, eight things at once.
How many qubits are there in today's quantum computers?
We're up to around a dozen -- so we can solve complicated equations really fast. And if you have 300 qubits, you can do 2300 things at once, which happens to be the number of elementary particles in the universe. So you can do a lot.

How does the machine actually work?

If you control an electron, you can control the qubit. You flip the qubits by zapping them with microwaves or lasers. Those are things that we know how to do pretty well. That's all that an ordinary computer does -- move bits from place to place.

Do quantum computers look like regular computers?

No. Your typical quantum computer is more like a digital computer of the 1950s. The qubits can be stored in molecules, which sit inside a tiny test tube. But to zap the qubits, you need to pop the test tube between superconducting magnets. That's inside a cryostat of liquid helium. It looks like a beer keg with a bunch of cables snaking out.

How do you do the zapping?

You give instructions on your ordinary computer. These get translated into a series of zaps by the microwave generator. Then you look at the weak microwaves given off by the molecules. Those are the results of your computation.

What if you don't use your computer for solving linear equations? How about quantum Googling?

You mean Quoogle? We've played with this idea. You could search databases faster with complete security and anonymity. After Quoogle gives you the answer, you're absolutely guaranteed that Quoogle cannot copy what the question is, because when you make a measurement on an unknown quantum state, you inevitably mess it up. The no-cloning theorem says that if you try to copy an unknown quantum state, (a) you can't, and (b) you inevitably mess up the quantum state by trying. So you can't copy the question.
This is private browsing on a new level.
Yeah, and actually I took the idea to Sergey Brin and Larry Page [of Google], and I said, "Hey dudes, we came up with this awesome idea for quantum Internet. How would you like to fund this or buy the company?" And they came back and said, "We're really sorry, but our whole business plan is based on knowing everything about everybody. So this goes against our business plan."

What are the biggest questions quantum computers could tackle?

Where the universe came from and where it's going in the far distant future. We can try to answer these questions because the universe is a quantum computer. Think about it in terms of information instead of energy. It's made of bits -- elementary particles -- and how they interact are operations. You can calculate how many bits are in the universe, how much energy it takes to flip them, how much energy exists, and use that to rule out lots of things about the universe's history. Anything that takes more bit flips couldn't have happened. That also means with enough bits you could make a quantum computer that would effectively be indistinguishable from the universe.

www.popsci.com...

Peace,
spec

reply to post by speculativeoptimist


Back in the late 80's there was AI software company that had a great tagline:

"We're drowning in data but starving for knowledge".

I think it's more true today than ever...

reply to post by Riffrafter

Touche' friend, what good is the vehicle if we don't learn how to drive correctly?
Perhaps one day the sum of all knowledge will be conveniently accessible by all, then maybe things can change.