WOW! M.I.T. Researchers can recreate sound from objects in the room

originally posted by: leolady

I had to go look up frame rates to educate myself on them and I found the following information:

30 fps is usually what a typical camcorder has
24 fps for motion picture recording like for movies, some camcorders have this too
60 fps are on some camcorders for faster frame rate (to capture fast movements, like sports / action)
120 fps or higher can record video in slow motion

***Their camera had 5602 frames per second (they show this in the video at the 3:04 mark). Probably a very expensive camera.***

You might be interested to know that research using fighter-pilots showed a capability to register images at a speed approaching something ridiculous, the equivalent of (from memory) 300 fps!

Most birds and insects have a much higher critical flicker frequency (the bit that is superceded when a film exceeds 24 fps, when we stop being able to distinguish individual frames and see the whole thing blur together as motion). A bird watching a film would be able to see each individual frame as a slideshow, rather than motion.

It's all fascinating stuff.

a reply to: EvillerBob

I have come to the conclusion that is almost certainly either a troll post to get people worked up, or he/she simply isn't interested in (or perhaps capable of) understanding.

I am familiar with the pattern of posting. It doesn't work me up but I guarantee nobody will get anywhere so there really is no point in trying. I think its more ego based.

No matter, we have made a t attempt at denying ignorance, sometimes you need to know when to cut your losses and move on.

yeah, personally I don't care with this since its work related chat. I don't care to talk shop here. Not at this level anyway.

a reply to: EvillerBob

What??

You're the one that doesn't understand or you realize you were wrong and you don't want to admit it.

You didn't make any attempt. You made a claim about waving a bag of potato chips and it's just nonsense. This is why I kept asking you to make your case based on the technology and the algorithm. You never did. You just obfuscated about magic waves that nobody talked about but you.

So when you make a claim, at least know what you're talking about before you make the claim or nothing that you say will make any sense.

I ask for the umpteenth time. How can waving a bag a chips around disrupt this technology when they're talking about vibrations less than 100th of a pixel? You would have to shake the bag so fast that it couldn't be captured in frame to run through the algorithm. It's obvious why your claim is simply absurd based on this technology.

a reply to: neoholographic

The video demonstrates the full capability of the software. No more. I cant find any source code or demo software so its more like vaporware. However the project that this is based off is freely available. people.csail.mit.edu...
The limitations are the same.

a reply to: ZetaRediculian

Source Code?

It's just obvious that you supported the silly notion that waving a bag of chips will bother this technology and that's a statement that doesn't make any sense so you're talking about source code because you think it means something.

Tell me exactly what source code you're talking about that would allow the waving of a bag of chips to avoid detection by this technology.

They spelled it out for you and they're presenting their finding at a top computer graphics conference.

Researchers at MIT, Microsoft, and Adobe have developed an algorithm that can reconstruct an audio signal by analyzing minute vibrations of objects depicted in video. In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass.

In other experiments, they extracted useful audio signals from videos of aluminum foil, the surface of a glass of water, and even the leaves of a potted plant. The researchers will present their findings in a paper at this year’s Siggraph, the premier computer graphics conference.

Like I said, this is just basic common sense. They're recording video and then running the frames through their technology. In order to to say that waving a bag of chips will disrupt this technology you have to show how waving a bag of chips can't be captured in frame when you're using cameras that capture between 2,000 to 6,000 frames per second.

You don't need source code to know that statement is devoid of any common sense.

You then mentioned the earlier work from M.I.T. and again, you need to read the actual article on the technology. Here's more:

Some boundaries in an image are fuzzier than a single pixel in width, however. So the researchers borrowed a technique from earlier work on algorithms that amplify minuscule variations in video, making visible previously undetectable motions: the breathing of an infant in the neonatal ward of a hospital, or the pulse in a subject’s wrist.

That technique passes successive frames of video through a battery of image filters, which are used to measure fluctuations, such as the changing color values at boundaries, at several different orientations — say, horizontal, vertical, and diagonal — and several different scales.

The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it’s struck by sound waves. Different edges of the object may be moving in different directions, so the algorithm first aligns all the measurements so that they won’t cancel each other out. And it gives greater weight to measurements made at very distinct edges — clear boundaries between different color values.

I remember in another debate you had a problem with taking things out of context. Like I said, you have to read the actual article and they lay it out for you.

They just borrowed the technique and then developed an algorithm that allows them to recreate sound from these frames. It's right there in black and white and simple for anyone to understand.

So again, waving a bag of chips is just nonsense because you have to wave the chips at Superman speeds in order for it not to be captured in frame.

a reply to: neoholographic

Source Code?

yes. Its a common thing amongst people that write software. It works like this. They share the code, I get to play with it and use it in my projects. They provided that with their other project. I downloaded it and played with it. Just like that. Pretty cool. I happen to get paid to write code. I have the same tools they do. Its not magic. They don't seem to have this one shared probably because its not ready and really buggy. Did you come across it?

So if you want to demo your superior knowledge, go for it. There is lots of freely available code out there.

a reply to: ZetaRediculian

I know what source code is, I was asking why do you need it. You haven't explained what you would be looking for in the source code that would show that you can wave a bag of chips around and it couldn't be captured in frame.

My point is, it has nothing to do with the assertion that waving a bag of chips can disrupt this technology. Unless you're moving in speeds that can't be captured in frame, the source code has nothing to do with waving a bag of chips.

This is about the camera and frames per second.

The source code comes into play when you talk about using these frames to recreate sound. You can't just say source code in a vacuum and think it means something. You need to explain exactly what would be in the source code to stop a camera from shooting a video at 5 or 6,000 frames per second.

It's a disconnect to what you're saying because you have no argument.

Also, why do you keep talking about magic? You heard the other guy mention magic and now you sound like a parrot. Nobody is talking about magic it's laid out for you in black and white.

The talk about waving around a bag of chips is just nonsense.

a reply to: neoholographic

I know what source code is, I was asking why do you need it. You haven't explained what you would be looking for in the source code that would show that you can wave a bag of chips around and it couldn't be captured in frame.

I wouldn't be looking for anything. I would just add it to my collection. I was suggesting that you could set up a demo and show how this magic would occur.

I could easily set it up and demo that I can't get it to work. Same as a lot of other things I have done. Obviously your knowledge is much better and nobody else has a clue. So go for it.

point is the code is somewhere and demonstrations can be made. No need to go back and fourth over who is right. Contact the guys at MIT. Have them comment. Do a demo. Provide the code. Anything. The explanations you are providing are way over everyone's head. I am sure MIT could use you. If you do contact them ask them why they needed to set up the camera behind a glass door. That was genius.

a reply to: ZetaRediculian

This is just gobbledy gook and you're not looking for anything because you have no point.

They tell you in the video why they set up a camera behind a glass door. Like I said, you need to actually watch and read about these things before commenting.

That was soundproof glass and they were showing that the technology isn't dependent on the audio from the room and they can recreate the sound from the room just through video by taking frames from the video and running them through their algorithm.

a reply to: neoholographic

This your time to shine. This isn't some obscure thing that we can only talk about. This is something that could be easily shown. You need a computer and a camera and a glass door.

Its vapor ware. Totally useless. The only reason you don't demo your software's capabilities is because it isn't capable. So they impressed you. There is no source code and no demo beyond what they showed. Vapor. Magic. Same thing. Come on now. Show me the magic!

originally posted by: ZetaRediculian
a reply to: neoholographic

This your time to shine...

While I applaud your commitment and have given you stars accordingly, I wouldn't suggest that you get your hopes up

The thread is becoming a very entertaining read, though probably not for the reasons that the original poster intended!

a reply to: EvillerBob
I hoping those knuckleheads at MIT are noticing that I'm calling their software worthless vaporware. My posts do come up on google sometimes. A glass door? Really. I guess we can rule out microphones. I might email them myself!

a reply to: ZetaRediculian

What??

Are you still in High School??

You're not making any sense.

Like I said, it's basic common sense. You're the one that hopped on the bandwagon with the other guy and talked about waving around a bag of chips yet you didn't provide any evidence that shows waving around a bag of chips can't be captured in frame when using a camera that's around 6,000 frames per second.

You then talk about source code but then again you provide zero evidence as to what could be in the source code that would prevent this technology from recreating sound.

So at this point we have a video and an article with the researchers explaining their technology and the algorithm they used. They about to be published and present this at a highly respected conference.

On the other side we have you and some guy called evil bob talking about waving a bag of chips and source code when you can't put into words what you're talking about.

I'll take my chances with the researchers from M.I.T. versus guys who have no idea as to what they're talking about.

It's your claim that waving a bag of chips will disrupt this technology. I simply asked to show how waving a bag of chips can't be captured in frame. Again, you and evil bob made the silly claim.

originally posted by: neoholographic
It's your claim that waving a bag of chips will disrupt this technology. I simply asked to show how waving a bag of chips can't be captured in frame. Again, you and evil bob made the silly claim.

Waving the bag will change the frequency that the bag is 'vibrating' at. It will now have the micro vibrations of the speech, the micro vibrations of the noise of the bag moving, and also the vibrations (micro and macro) that will occur as the structure of the bag moves.

That will change the sound that is derived by the software, based on the video of the now waving bag.

Same reason you can defeat this (from a clandestine perspective) with a tactile transducer. It will change the frequency of everything it is in contact with, thereby changing the output recorded.

a reply to: peck420

Nope,

This is exactly why they talked about frames per second and the cameras that they're using.

All of the noise will be captured in frame an then recreated through the algorithm. Did you guys even read what they said? How is waving a bag of chips around and measuring vibrations less than 100th of a pixel going to be disrupted when you're recreating the sounds as a whole in frame?

That makes zero sense. Again, they explain this in simple terms.

“When sound hits an object, it causes the object to vibrate,” says Abe Davis, a graduate student in electrical engineering and computer science at MIT and first author on the new paper. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”

A VISUAL SIGNAL!

This is why they did this through soundproof glass. To show you that it's the visual signal that's recreating the sound at vibrations of less than 100th of a pixel. To suggest waving a bag a chips around will disrupt this technology is asinine. Here's more:

Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.

Why did they mention this?

IT'S BECAUSE THE ALGORITHM RECREATES THE SOUND THROUGH FRAMES.

Why is this so hard to grasp. Unless you can wave the bag of chips so fast that these small vibrations can't be captured in frame, then what's being talked about is senseless. There's more:

“We’re recovering sounds from objects,” he says. “That gives us a lot of information about the sound that’s going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways.” In ongoing work, the researchers have begun trying to determine material and structural properties of objects from their visible response to short bursts of sound.

Again, waving around a bag of chips is just silly talk. Here's more:

The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it’s struck by sound waves. Different edges of the object may be moving in different directions, so the algorithm first aligns all the measurements so that they won’t cancel each other out. And it gives greater weight to measurements made at very distinct edges — clear boundaries between different color values.

The algorithm is recreating sound by inferring the motions of an object in frame. So the whole notion is senseless unless you can wave the bag around fast enough where it can't be captured in frame.

All you have to do is read what they're saying.

Think of how silly this sounds.

Criminals are going to be in a room waving around a bag of chips and that's supposed to stop this technology.

What about the walls, the columns in the room, the table, a picture on the wall, a fixture on the wall or anything else in the room.

That's just nonsense.

The best way to get around this technology is to write out what you're doing if you think you're being watched then burn whatever you wrote down after the meeting is over.

It's just stupid to sit there and wave around a bag of chips when you will not avoid being captured in frame when you're dealing with a camera that may have a 6,000 fps.

a reply to: neoholographic

What stops this technology is noise, just like with any other bugging technology,

a reply to: GetHyped

Nope, because the sound as a whole is being recreated through visual data. This is exactly why they showed you the technology working through a sound proof window.

It's spelled out for you when they explain the technology.

a reply to: neoholographic

There is nothing magical about the spectral band the human voice occupies. Any background noise of a sufficient level occupying this band (i.e. lots of things) will mask this.

There is no way of getting around this fact.

a reply to: GetHyped

Again, that makes no sense. Criminals have been caught throughout the years from getting bugged even when they suspected they were being bugged.

This technology recreates sound through visual data and it silly to think that criminals haven't been able to escape being bugged but they are masterminds that can disrupt technology that's recreating sound through visual data that's less than 100th of a pixel.

You can't be serious.