WOW! M.I.T. Researchers can recreate sound from objects in the room

originally posted by: leolady
Interesting technology, I can think of several beneficial uses

At this stage it's still relying on very high quality footage to work, even with their "reduced quality" version. The camera would need to have a sufficient resolution to allow an incredibly tiny movement to register as at least one pixel's worth of change, and with a frames-per-second rate capable of detecting that change. Without going back to check, I seem to recall that the "reduced quality" version was still recording at 60fps, roughly twice the normal fps.

I'm too lazy to try any calculations at the moment but you could probably work out a rule of thumb for resolution needed at a given distance. I suspect it's far beyond the majority of CCTVs - and even a good camera with high fps is still going to rely on sufficient lighting or a ridiculously large dynamic range. Not to mention the storage space those massive files would need! For CCTV recording, 60fps represents a colossal waste of resources just on the off-chance that it might detect some audio. If a dedicated microphone (designed and built with years of practical technological development behind it) isn't able to pick up the audio in the room, the chances are that the vibrations aren't strong enough to be picked up effectively by a packet of crisps either.

Interesting and potentially practical uses, yes, but I don't think they will come to fruition in the near future - not so much an issue with their technology, more the failure of existing technology to give them the raw material they need.

It's a bit like the old joke about the first guy to invent an electric desklamp, sitting around for hundred years waiting for someone to invent an electric socket so he could plug it in. Hey, I never said it was a funny joke

Edited to add:

You'll also notice that they are focussing on things that aren't very heavy. For this to work, you are relying on the object being sufficiently light to be able to react - and react fast enough - to mimic the frequency. This is why heavy objects block high frequencies such as voice but not lower frequencies such as heavy basslines in music. The mass of the object simply does not allow it to respond quickly enough to be an effective medium for transmission of high frequencies.

Any audio technicians on the board are now hopping up and down in anger over my wildly summarised explanation, but I'm aiming for simplicity of concept more than technical accuracy!

a reply to: EvillerBob

Sorry but you have it all wrong. You need to read the article and the link to M.I.T. news where they explain what is happening.

Like I said they're recreating these sounds through vibrations that can't even be seen. These vibrations can even be picked up through soundproof glass. You still haven't explained your position based on the algorithm that's being discussed.

First, these were done with very good cameras but you can get even better quality with cameras that have better frame per second. Here's more:

Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.

What you're saying has nothing to do with the technology and the algorithm they uses to recreate the sound. This is why it's talked about being used by law enforcement.

It's next to impossible to manipulate vibrations that you can't even see.

The researchers’ technique has obvious applications in law enforcement and forensics, but Davis is more enthusiastic about the possibility of what he describes as a “new kind of imaging.”

“We’re recovering sounds from objects,” he says. “That gives us a lot of information about the sound that’s going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways.” In ongoing work, the researchers have begun trying to determine material and structural properties of objects from their visible response to short bursts of sound.

The researchers are talking about using it in law enforcement, forensics and other places because like I said it's next to impossible to manipulate vibrations that you can't even see. So what you're saying has nothing to do with the article or the technology. It ended with this:

“This is new and refreshing. It’s the kind of stuff that no other group would do right now,” says Alexei Efros, an associate professor of electrical engineering and computer science at the University of California at Berkeley. “We’re scientists, and sometimes we watch these movies, like James Bond, and we think, ‘This is Hollywood theatrics. It’s not possible to do that. This is ridiculous.’ And suddenly, there you have it. This is totally out of some Hollywood thriller. You know that the killer has admitted his guilt because there’s surveillance footage of his potato chip bag vibrating.”

So like I said, based on what the researchers are saying, the algorithm and the vibrations being used to recreate the sound, what exactly are you talking about?

It's a neat technology, I wonder if they'll be able to get any audio off of some old historical footage where there isn't any sound?

As for eavedropping? Yeah I could see that it'll work. And it's a passive method (compared to lasers or planted bugs), so you can't detect it. However that doesn't mean that it can't be defeated. I've heard one of the best and easiest ways for a private conversation when worried about eavedroppers is to go for a car ride with the person you want to talk to. You turn the stereo up enough so the laser on the window method doesn't work so well. (And it makes getting clear audio on any bugs difficult if not impossible.) And it also makes it obvious if somebody is even trying to listen in, as in most cases they'll have to tail you.

originally posted by: neoholographic
a reply to: EvillerBob

Sorry but you have it all wrong. You need to read the article and the link to M.I.T. news where they explain what is happening.

Like I said they're recreating these sounds through vibrations that can't even be seen. These vibrations can even be picked up through soundproof glass. You still haven't explained your position based on the algorithm that's being discussed.

First, these were done with very good cameras but you can get even better quality with cameras that have better frame per second. Here's more:

...

What you're saying has nothing to do with the technology and the algorithm they uses to recreate the sound. This is why it's talked about being used by law enforcement.

It's next to impossible to manipulate vibrations that you can't even see.

...

The researchers are talking about using it in law enforcement, forensics and other places because like I said it's next to impossible to manipulate vibrations that you can't even see. So what you're saying has nothing to do with the article or the technology. It ended with this:

...

So like I said, based on what the researchers are saying, the algorithm and the vibrations being used to recreate the sound, what exactly are you talking about?

I think that you have misunderstood what they are claiming, because you are quoting elements to contradict me that are basically saying exactly the same thing as I said in my previous posts.

I've gone back to re-read the article again and my position hasn't changed in the slightest. The algorithm is simply the process that takes the output of various filters applied to the image and combines them to identify the vibration. The "special" version they use for "normal" video relies on the fact that a single frame actually consists of lines of data read over time (albeit an incredibly small amount of time) so that single frame has managed to record the object in several different states caused by rapid vibration. In other words, while that frame might represent 1/60th of a second of time, each individual line in the frame represents a different slice of that 1/60th of a second, as the camera reads each row separately.

I'm not entirely sure what you want me to say about the algorithm? I have no doubt that the process works and is viable, I don't question the effectiveness of the algorithm. In fact it's all rather clever. I'm simply pointing out the natural and obvious limitations of the process.

The vibrations can be seen (otherwise they could not be recorded with a video camera) but they are too subtle for the human eye to detect them. The whole point of the experiment is to show that, with sufficient quality recording equipment, you can slow down the footage enough to observe movement that is too fast for your eye to process, or make use of a "bug" in the recording process to actually observe much smaller slices of time than the stated fps.

This is all purely visual data. The fact that the glass is soundproof has absolutely no bearing on that whatsoever (if we leave to one side the issue of distortion arising from light travelling through different mediums before it reaches the camera, which doesn't appear to be an issue here).

If it helps, think of it this way - if you cannot see an object in the sky without a telescope, does that mean it's invisible? Or just that your eyes aren't sensitive enough to see it? That's exactly the situation with this technology.

Let's try to address some of your points more directly.

What you're saying has nothing to do with the technology and the algorithm they uses to recreate the sound. This is why it's talked about being used by law enforcement.

I am talking about sound, vibration, and being able to detect the vibration caused by sound to a sufficient degree of resolution to allow it to be analysed. That is the entire basis of the technology in the article. I can't talk about the technology without addressing these areas.

There are no magic waves being emitted by the bag, no special form of radiation that is being detected, the bag is not somehow storing this information to be released later, it's just plain old visual observation of the movements induced by variation in air pressure - in other words,the bag wobbling a bit in response to sound while someone records it with a fancy video camera.

It's next to impossible to manipulate vibrations that you can't even see.

I'm actually at a bit of loss with this statement, but I'm suspecting that it indicates a troll post. I will address it anyway.

A vibration is just a series of movement, albeit within a range of conditions that would classify that particular sequence of movements as a vibration. Whether or not you can see it makes no difference, it's just a series of movements.

Something is moving. If you want to change how it moves, you could hold it tight to stop it moving. You could push it harder to make it move more. You could wave that bag of crisps around in the air while you talk, and suddenly the algorithm would be faced with the massive (and probably insurmountable at this stage) challenge of working out which movements were the result of sound and which were the result of you moving the bag.

Perhaps a better question would be... what exactly do you think sound is?

a reply to: EvillerBob

Your whole post is just a bunch of nonsense that has nothing to do with what's being discussed. I can tell you probably have long winded conversations because you're trying to sound like you know what you're talking about when you don't. You're just obfuscating and it has nothing to do with what's being discussed.

First, nobody said anything magical was going on. This is the first sign of someone who can't debate the issue so they try to debate against things that nobody said.

Nobody has talked about magic waves or invisible waves. Of course we're talking about vibrations that can't be seen by the naked eye. This is what the article talks about so your mentioning of magic waves is just a red herring that you mentioned to try to mask the point that you don't know what you're talking about. Who said anything about magic waves???

Your entire post doesn't refute anything I have said.

You're debating against magical waves or these vibrations being invisible when nobody has made these claims.

You then ended with this.

Something is moving. If you want to change how it moves, you could hold it tight to stop it moving. You could push it harder to make it move more. You could wave that bag of crisps around in the air while you talk, and suddenly the algorithm would be faced with the massive (and probably insurmountable at this stage) challenge of working out which movements were the result of sound and which were the result of you moving the bag.

After an entire post filled with gobbledy gook you finally tried to address the issue and it's clear you don't understand the algorithm or the technology.

First, this isn't just about a bag of chips. Anyone should be able to see this if you watch the video or read the article. Criminals are not going to be in a room without any walls waving around one bag of chips. That's not the point. What they have shown is that you can zero in on just about anything in a room and recreate the sounds coming from the room. So the fact that you think waving around a bag of chips will stop the sounds from being recreated makes zero sense.

Secondly, the algorithm recreates sounds from the room. So the algorithm will only pick up any waving of a bag of chips if it's making sound. This why I keep asking you about the basis of what you're saying based on the actual algorithm.

You said something about the algorithm working out which movements were the result of the moving bag. What is this nonsense based on? You think someone waving a bag will effect an algorithm when we're talking about movement that's less than 100th of a pixel? Here's more from the article:

The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it’s struck by sound waves. Different edges of the object may be moving in different directions, so the algorithm first aligns all the measurements so that they won’t cancel each other out. And it gives greater weight to measurements made at very distinct edges — clear boundaries between different color values.

The researchers also produced a variation on the algorithm for analyzing conventional video. The sensor of a digital camera consists of an array of photodetectors — millions of them, even in commodity devices. As it turns out, it’s less expensive to design the sensor hardware so that it reads off the measurements of one row of photodetectors at a time. Ordinarily, that’s not a problem, but with fast-moving objects, it can lead to odd visual artifacts. An object — say, the rotor of a helicopter — may actually move detectably between the reading of one row and the reading of the next.

The researchers already addressed this issue and what you're saying makes zero sense because you're dealing with very small vibrations that can't be seen by the naked eye. These vibrations aren't distorted by someone waving around a bag of chips.

You have to have fast moving objects like the rotor of a helicopter to begin to see strange visual effects that can effect the recreation of sound but they go on to say these vibrations can still be detected.

a reply to: neoholographic

Your entire post doesn't refute anything I have said.

I have read your posts and am quite confused about what you are trying to say and what you think has been accomplished. They recreated sound from video of vibrations. I think we all agree that its impressive. I don't see any practical application for this yet in its present form. Some of your quotes mention that they are able to tell what an object might be made of based on the vibration.

I'm really sorry to say this but there is no politer way for me to put it: your understanding of this article, the science behind it, and the technology involved is very, very flawed. Or, you are trolling, which is increasingly likely.

The difficulty I am having is in pinpointing what the specific issue is that seems to be confusing matters for you, because I'm more than happy to try and work through it with you if I can.

To consider the most relevant points from your post:

originally posted by: neoholographic
Nobody has talked about magic waves or invisible waves. Of course we're talking about vibrations that can't be seen by the naked eye. This is what the article talks about so your mentioning of magic waves is just a red herring that you mentioned to try to mask the point that you don't know what you're talking about. Who said anything about magic waves???

Based on your posts, I was trying to understand how you came to form your opinion of how the technology worked. You did not use those exact phrases, but the only way some of your comments make sense is if you considered the vibration from the song/voice to somehow be fundamentally different to any other vibration or movement.

originally posted by: neoholographic
You have to have fast moving objects like the rotor of a helicopter to begin to see strange visual effects that can effect the recreation of sound but they go on to say these vibrations can still be detected.

They say that the movement of the blade can be detected, as an example of how a single frame can be used to demonstrate movement that is even faster than the frame rate. They do not say you have to move something that fast to be able to prevent sound from being recreated, nor do they say you would have enough information to still recreate the sound made by the blades - though it is possible that you would, it wasn't the point that they were making. It's just an example of how their software can detect incredibly fast movement that is usually a blur even for high-fps cameras.

originally posted by: neoholographic
...the algorithm recreates sounds from the room. So the algorithm will only pick up any waving of a bag of chips if it's making sound. This why I keep asking you about the basis of what you're saying based on the actual algorithm.

OK, answer this. How is the algorithm "recreating sounds from the room"? Don't just quote the article because I've read it several times now. Explain in your own words. What is it in the video that the algorithm uses to recreate the sound?

a reply to: neoholographic

Signal

To

Noise

Ratio

What they have shown is that you can zero in on just about anything in a room and recreate the sounds coming from the room.

Not if there is sufficient noise masking the signal.

These vibrations aren't distorted by someone waving around a bag of chips.

They are distorted by noise (i.e. all other background sources). I'm not sure why you keep going on about "very small vibrations that can't be seen by the naked eye". ALL sources will contribute to these vibrations. A sufficiently high level of background noise will mask the signal you wish to record. No amount of fancy algorithmic processing will get around this. None. Nothing. Nada. Zip.

a reply to: EvillerBob

You said:

Based on your posts, I was trying to understand how you came to form your opinion of how the technology worked. You did not use those exact phrases, but the only way some of your comments make sense is if you considered the vibration from the song/voice to somehow be fundamentally different to any other vibration or movement.

Of course I didn't use any exact phrases about magic waves or invisible waves you can't even show where this was implied. This was just nonsense that you conjured up because you can't debate the issue. So you spent an entire post debating against magical waves. We're debating the article that goes with the video and we're talking about the technology not magic or invisible waves. That's just a flat out lie that you made up out of thin air because you couldn't debate against what I actually said so you made up this nonsense about magic waves and spent an entire post debating against something that was never said nor implied.

You then totally misunderstand what they're doing and this why they talked about the rotor of a helicopter. It says:

The researchers also produced a variation on the algorithm for analyzing conventional video. The sensor of a digital camera consists of an array of photodetectors — millions of them, even in commodity devices. As it turns out, it’s less expensive to design the sensor hardware so that it reads off the measurements of one row of photodetectors at a time. Ordinarily, that’s not a problem, but with fast-moving objects, it can lead to odd visual artifacts. An object — say, the rotor of a helicopter — may actually move detectably between the reading of one row and the reading of the next.

For Davis and his colleagues, this bug is a feature. Slight distortions of the edges of objects in conventional video, though invisible to the naked eye, contain information about the objects’ high-frequency vibration. And that information is enough to yield a murky but potentially useful audio signal.

Yes, you would have to have very fast movement to disrupt these small vibrations. Try making a You Tube video waving a bag a chips where your movement can't be captured by the cameras they're using let alone where your movement disturbs vibrations that are less than 100th of a pixel. That's just simple minded and makes no sense.

Here's more

That technique passes successive frames of video through a battery of image filters, which are used to measure fluctuations, such as the changing color values at boundaries, at several different orientations — say, horizontal, vertical, and diagonal — and several different scales.

The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it’s struck by sound waves. Different edges of the object may be moving in different directions, so the algorithm first aligns all the measurements so that they won’t cancel each other out. And it gives greater weight to measurements made at very distinct edges — clear boundaries between different color values.

Again, your claim is just PURE NONSENSE.

You're the one who talked about waving around a bag of chips. This is why I asked you to show me where waving around a bag of chips would stop the algorithm from recreating the sound in the room.

It's obvious that this was just nonsense and this why you blathered on and on about magic and invisible waves. It's because your assertion doesn't have anything to do with the algorithm used to recreate the sounds from the room.

Again:

The researchers developed an algorithm that combines the output of the filters to infer the motions of an object as a whole when it’s struck by sound waves.

The fact is, you would have to have Superman like speed to move a bag of chips fast enough to where sound couldn't be recreated by this algorithm lol.

I ask you again, based on this algorithm how could someone move a bag a chips fast enough to avoid the recreation of sound through vibrations that are less than 100th of a pixel?

It's a simple question that you have failed to even try to answer. So please answer the question and no more long winded posts about magic waves. Stop debating against nonsense that you're making up and debate the issue.

a reply to: neoholographic

I ask you again, based on this algorithm how could someone move a bag a chips fast enough to avoid the recreation of sound through vibrations that are less than 100th of a pixel?

If they could do what you think, there would be some video of that. The thing with this is that there is no magic to it. You are inferring something that is not there. If there is such a claim from these folks, it still would have to be demonstrated. There is no such claim.

a reply to: EvillerBob

Yes I see what your saying. The video cameras frame rate has to be high for it to work most effectively and yes they did show that a slower frame rate still works but quality goes down.

It is an awesome discovery and maybe we will see something come from it. I imagine researchers at MIT are always making discoveries such as this one and it doesn't mean it is the finished product. It may need to be fine tuned and made practical for us in the everyday world.

I had to go look up frame rates to educate myself on them and I found the following information:

30 fps is usually what a typical camcorder has
24 fps for motion picture recording like for movies, some camcorders have this too
60 fps are on some camcorders for faster frame rate (to capture fast movements, like sports / action)
120 fps or higher can record video in slow motion

***Their camera had 5602 frames per second (they show this in the video at the 3:04 mark). Probably a very expensive camera.***

leolady




leolady

a reply to: ZetaRediculian

What in the world are you talking about? I'm not surprised that you showed up in a thread spouting things that has nothing to do with what's being said.

What do I think? You said if they could do what I think. Explain to me what I think. That's just silly and a blanket statement that has nothing to do with the debate. I suggest that instead of being a Johnny come lately and just saying things that make no sense, you should actually read what's being discussed before making blanket statements like this.

Who said anything about magic? Show me where I said anything about magic.

People on this thread have been talking about the technology and nobody has mentioned anything about magic. This is the height of ignorance.

What claims are you talking about?

This is just the oldest and silliest debating tactic that's always used by you and others on this board. When you can't debate what's actually being said you make stuff up and then debate against the things you made up.

So please to come onto this thread with your usual nonsense. If you want to debate a specific issue about the technology then that's fine. Don't come in making these vague and senseless statements about magic when nobody has talked about magic.

a reply to: leolady

You highlighted why this technology will be good for law enforcement. You said:

The video cameras frame rate has to be high for it to work most effectively and yes they did show that a slower frame rate still works but quality goes down.

This is exactly the point that was made in the article. They said this:

Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.

Again, you have commercial cameras that can go up to 100,000 frames per second. There's even a camera that captures 1 million frames per second but at a lower resolution.

www.wired.com...

So as the technology gets better and they use cameras that have a faster fps, this will be an invaluable tool for law enforcement and they said in the article the Researchers also produced a variation of the algorithm for conventional videos so eventually the algorithm might be used on older videos as well.

a reply to: neoholographic

So, tactile transducer outputting at 10,000hz+?

For about $50 (+shipping), you can make your room unreadable by this technology.

a reply to: neoholographic

eventually the algorithm might be used on older videos as well.

Yes and that is interesting to me.

I was having a discussion this morning with someone and we were saying what if we could go back and look at the JFK shooting video to find more answers that were never resolved.

leolady

a reply to: neoholographic

I ask you again, based on this algorithm how could someone move a bag a chips fast enough to avoid the recreation of sound through vibrations that are less than 100th of a pixel?

If their software were this good, they would be showing someone shaking a bag of chips while their software did its thing. They don't and they don't claim that they can do this. Showing that would be very magical. At that point I would need to see more technical documentation than what you are providing. The basic gist of this is not that hard to follow. Are they providing the source code or an executable? If they are, it should be easy enough for you to set it up and demo.

a reply to: ZetaRediculian

You can't be serious.

Shaking a bag of chips will not effect this technology unless you shake it fast enough not to be captured in frame. That's just a nonsensical statement. The algorithm is capturing motion at very small vibrations.

Showing this has nothing to do with magic it just takes a little common sense. This is why they talk about frames per second.

Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.

Again, unless you can shake a bag of chips so fast that it can't be captured in frame then what you're saying makes zero sense unless you're Superman lol.

The algorithm is based on capturing FRAMES. How are you going to move the bag of chips fast enough not to be captured in frame??

If you would have just read this:

That technique passes successive frames of video through a battery of image filters, which are used to measure fluctuations, such as the changing color values at boundaries, at several different orientations — say, horizontal, vertical, and diagonal — and several different scales.

You wouldn't ask such silly questions or make blanket statements about magic.

So when you say if there software was that good, it's a senseless statement. They're capturing these vibrations in frame and then using the algorithm to recreate the sound. So talking about magic is just pure ignorance and you need to actually read about the technology and the algorithm before you make blanket statements that make no sense.

a reply to: neoholographic

You can't be serious.

um, yes. I am very familiar with vision software and it is painfully obvious that you are not. Thanks for the link. Enjoy your fantasy.

originally posted by: ZetaRediculian
a reply to: neoholographic

You can't be serious.

um, yes. I am very familiar with vision software and it is painfully obvious that you are not. Thanks for the link. Enjoy your fantasy.

I have come to the conclusion that either neoholographic's posts are troll posts to get people worked up, or he/she simply isn't interested in (or perhaps capable of) understanding.

No matter, we have made a solid attempt at denying ignorance, sometimes you need to know when to cut your losses and move on.

originally posted by: leolady

Yes and that is interesting to me.

I was having a discussion this morning with someone and we were saying what if we could go back and look at the JFK shooting video to find more answers that were never resolved.

leolady

It would be fascinating - theoretically, you might even be able to extrapolate directional information with this technique, which would be very intriguing for the Zapruder footage.

Unfortunately this technique could not be applied to that footage, which is a shame.