Social Networking and Data Mining: What It Reveals About You

Pete Warden, a Colorado-based, British-born ex-Apple engineer who has spent the last six months gathering and analyzing data from more than 215 million public Facebook profile pages. What he’s discovered just might shed more light on the culture of connected America than the 2010 census.

"If you actually look at [Facebook user data] in the aggregate, it's like a painting," Warden told TechNewsDaily. "Each individual data point isn't interesting, but when you step back and look at the trends in millions of profiles, you start to see some pretty interesting pictures emerging."

Facebook Mining

Now, after gathering the data from Facebook’s site using software he designed and honed in the process, and making a first round of enticing observations, he wants to turn the raw data he’s culled over to academia for further analysis. But he also hopes to steer investors and customers to his own software and services for further data gathering and aggregation.

"I'm much better at building the pipeline for processing the data than I am at doing really rigorous stuff with the results that come out at the end," Warden said in a telephone interview. "The patterns that I've blogged about in the U.S. data are very qualitative."

Serious about privacy

But Warden is serious when it comes to people’s privacy concerns, even though all the data being gathered is publicly available on Facebook’s site, and can be found via Google. He says he wants to make the data useful for large-scale data analysis, but not for tracking down individuals.

To that end, Warden has delayed releasing the data for the time being (he initially intended to release it yesterday, Feb. 9), after someone from Facebook contacted him, asking for some time to check the privacy implications.

Once Facebook clears the data for release to the academic world, Warden says he’s ready to pass the task of interpreting all this data on to others and feature their conclusions on his blog more often than his own. Link

I think that the majority of ATS users are well aware of the data mining via Facebook and other social networking sites, how that can be used for marketing and possible other purposes and how to best protect one's privacy; whether one chooses to abstain from social networking altogether, or whether one confines social networking to close family and friends, etc...

I do have to wonder how much thought the typical person puts into how much information is available to the masses and whether the typical person considers this availability of information a concern?



How to split up the US - Warden's Blog

Warden gathered data on 210 million public Facebook profiles and discovered emerging patterns and detail. His observations and visualization represented in the map below shows the information by location, with connections drawn between places that share friends.

The descriptions below are just part of the analysis Warden was able to make based on public profile data. Bear in mind, this analysis is amateur and speculative. However, Warden does plan to turn his data mining results over to the Pro's for further study and use.

[atsimg]http://files.abovetopsecret.com/images/member/1e8617900857.jpeg[/atsimg]

Quoted for interest: there is greater detail in the actual Blog descriptions, this is just bare bones...

Stayathomia
Stretching from New York to Minnesota, this belt's defining feature is how near most people are to their friends, implying they don't move far.

Dixie
Probably the least surprising of the groupings, the Old South is known for its strong and shared culture, and the pattern of ties I see backs that up. Like Stayathomia, Dixie towns tend to have links mostly to other nearby cities rather than spanning the country.

Greater Texas
Orbiting around Dallas, the ties of the Gulf Coast towns and Oklahoma and Arkansas make them look more Texan than Southern. Unlike Stayathomia, there's a definite central city to this cluster, otherwise most towns just connect to their immediate neighbors.
God shows up, but always comes in below the Dallas Cowboys for Texas proper, and other local sports teams outside the state.

Mormonia
The only region that's completely surrounded by another cluster, Mormonia mostly consists of Utah towns that are highly connected to each other, with an offshoot in Eastern Idaho. It's worth separating from the rest of the West because of how interwoven the communities are, and how relatively unlikely they are to have friends outside the region.

Nomadic West
The defining feature of this area is how likely even small towns are to be strongly connected to distant cities, it looks like the inhabitants have done a lot of moving around the county. For example, Boise, ID, Bend, OR and Phoenix, AZ all have much wider connections than you'd expect for towns their size.

Socalistan
LA is definitely the center of gravity for this cluster. Almost everywhere in California and Nevada has links to both LA and SF, but LA is usually first. Part of that may be due to the way the cities are split up...Californians outside the super-cities tend to be most connected to other Californians, making almost as tight a cluster as Greater Texas.

Pacifica
The most boring of the clusters, the area around Seattle is disappointingly average. Tightly connected to each other, it doesn't look like Washingtonians are big travelers compared to the rest of the West... Link

As interesting as this amateur study is, I think that it drives home a significant point. Regardless of whether one personally uses social networking and media - or doesn't - it's affect upon you are still relevant and will probably still be used for target marketing, at the very least.



ed: Shorten Title


[edit on 14-2-2010 by LadySkadi]

I read somewhere that Facebook was initially funded by the CIA, or at least partially funded by them, and I know our local law enforcement agencies use it all the time to locate people.

I agree it's much more than just social networking anymore, and people should be very cautious about what they put on it.

Particularly I would caution those in the job market, as potential employee's lives are examined by those who can get them. It's a good way to lose a job if you're not careful. This I know from friends who are in hiring positions.

It does paint an interesting picture of some demographics around the country.....provided you are on facebook. One of my friends insisted that I sign up years ago....but I haven't been back since. So, I'm thinking that to get a clear picture of what goes on, it would have to be more inclusive.

But no doubt, we are getting there! Good article.

I hope your thread does better then mine did...
www.abovetopsecret.com...
same topic pretty much

I do have to wonder how much thought the typical person puts into how much information is available to the masses and whether the typical person considers this availability of information a concern?

I am afraid that many users of social networks do not realize just what kind of private information they are putting out on the web.

Even a simple statement as "My six year old is just now learning his basic math" can say many things that you might have not intended to say.

"My six year old is just now learning basic math, means your age is somewhere between 25 and 35, that you are most likely married, and you live close to a school of some sort, more likely a high end school. It would also hint at your spouses age, and stating "my six year old" might actually hint that you have more kids referring to one by his or her age.

The point is one sentence of information can allow one to come to some logical and accurate educated predictions about ones life.

Now one must consider how much information is in their posts, your profile and even your scree name/avatar times how many sites you are on.

If someone wanted to they could come close to any information needed.

That is unless one is taking steps to fix this problem

Additionally, there are considerations that are out of one's control. Reference the article in the opening post. Programs are being developed to mine information from public profiles on a larger scale and this information is being used to "profile" areas of the country, gender, education level, etc. based (not necessarily on your profile alone) but all of one's friends and friend's of friend's extended throughout the parameters of the "mining search."

Not to mention that these inferences will be extended to people who are not even using these sites, simply by being a member of the "profile" category or location. Social sites and the information gleaned from them, now have the capacity to impact every single person, not just those who actively use it.

ed: for sp.



[edit on 14-2-2010 by LadySkadi]

FACEBOOK: Federal Human Data Mining Program




*I think this video has been shown on ATS before, but never hurts to revisit it*

Google + the NSA

What would be the point of a Google-NSA partnership?

In theory, the NSA could help Google defend its systems (and thus your information) from future attacks. The newspapers' reports describe the arrangement as providing a kind of "technical assistance" that'd allow Google to better understand who breached its network and how they managed to pull it off.

Google- NSA Alliance

ed: fix link



[edit on 15-2-2010 by LadySkadi]

Great Thread...

But I wanted to briefly suggest that this isn't just related to social networking sites. Microsoft is hashing out a program to glean relevant marketing directives based on all websites visited by a web user...

Detecting Online Commercial Intention

Note the tool input in the upper right. Type in any web address and the likelyhood of consumer motivated intention will be displayed (ATS came back as a 0.84 probability for Non-Commercial Intention").

And this is just a basic assessment based on website hits and subsequent activity regarding each website.

Who here thinks that they aren't already attempting to piece together personality profiles based on the entirety of one's internet habits?

And would it be irrational to suppose, even for a minute, that social networking sites may perhaps be sharing their 'generalized' data for use within much more complex internet social/consumer models?

Edit To Add -

People Name Detection
Many online queries are either personal names or they contain personal names. For example, "John Smith" is a personal name, and "I am pleased to tell you that Andy Beal will be working as an internet marketing consultant" contains a personal name. This tool detects names of people in a query, which can improve ad relevancy for online marketing. This also helps in understanding a user's Web search intent and could be used to provide more relevant search results to meet the user's needs, such as providing the biography and other attributes of the targeted person to the users.

Same site as above link

[edit on Mon, 15 Feb 2010 00:50:00 -0600 by MemoryShock]

I read a few weeks back that the owner of Facebook has been contributing money towards the socialist american party.

My .02

reply to post by MemoryShock
Excellent point MemoryShock and very relevant.

This is not just about social networking sites, as you pointed out and IMO is only going to continue to grow tentacles that will reach into every aspect of online activity.

Let us not forget Google's recent announcement that it would start to track all searches, regardless of whether one was signed into their account plus their recent partnership with the NSA (though at present, does seem to be with the intent to protect privacy and information) but nonetheless, all signs point to continued advancements in harnessing the publics information.

The money is there and large profits to be made. Among other things.

The link between Facebook and U.S government agencies presented in that Youtube video is pretty ridiculous. I’m sure if you looked hard enough you could find similar links in plenty of other big businesses.

This kind of data collection sounds like an awesome idea as long as it’s fairly generic.

reply to post by maxcobalt

This kind of data collection sounds like an awesome idea as long as it’s fairly generic.

I suppose the question is, will it stay that way?

Of course, marketing campaigns are big business, consumer information is its backbone and there are numerous ways to go about ensuring that companies get the information they need to supply the public with its wants (or some say, decide for the public what they want.) Social networking and online tracking are added components that will be incredibly useful going forward. No doubt.

However, always consider both the positives and negatives... there will be many of both...


ed: sp (doh)

[edit on 15-2-2010 by LadySkadi]

reply to post by MemoryShock


Great point as I see a thread about FB every so often and evreyone puts FB down but are expressing away on ATS.

S+F - This is my favourite subject.

I personally have no problem volunteering my information out to certain sites. It allows me to have opportunities and establish a cultural identity with which I can make friends and enemies.

I do have a problem with who is in control of my data though. God and angels I'm fine with - but sadly they're fictional (IMHO).

When I was back in the UK data blunders (which still haven't been cleared up) cost millions of people their personal private information. Addresses, next of kin, bank account details, date of birth...pretty much the whole shebang have been 'lost' on trains and trashcans and 'disappeared' from where they should be. So if this is the data we know about (because it's physical) then what of the data that just just copied and passed along without any one knowing?

I gave a lecture once on Facebook, it's benefits and potential problems. It was about a year ago. During the lecture I had the class pick a city that wasn't in our country and asked them to select a male or female candidate. I then hunted down through the publicly available information and found a single male who had pictures of his house and its equipment, his pet fish, his 'friends', his phone number, favourite football team, date of birth, email and fact that he was going to a 3 day rock festival 300 miles north of where he was located the following weekend. All of this was projected onto large display for the group to see. It took about 10 mins.

I put forward the idea that if we were of a criminal nature we could break into his house whilst he was away and steal all of his stuff. I asked the class how viable that option would be and they agreed it was incredibly realistic. I then put forward the idea that since we know who his mother's mother is from his status updates we could get some credit cards planned as we know his date of birth, mother's maiden name and address. We could pick up the cards when we visit his house to break in. Again, it was seen as a viable option should we have a criminal nature. Finally I said, we know his email address. Who thinks that the password will be his pet fish, football team or some other piece of information that he's volunteered?

I didn't go so far as to break into someone's email (As I'd already shown the class prior that most of their passwords were either their kids or loved ones names, pets or sports teams) but it left the group with the feeling I was aiming for. To break into a house you have to be there - but he'd given us all we needed to know on how to do it. To break into someone's entire online life you only need an email and password and you can be anywhere.

With social media being integrated more and more with third party applications one fault in our security could have all our online identity compromised and we wouldn't even be aware of it until it was too late.

URL shorteners also cause problems. With Twitter constricting us to 140 characters people use Bit.ly or some equivalent to point people to sites they think are worthy of note. These mask the target URL and are a ripe market for being exploited in the phishing market. One click from a Twitter posted link could lead you to a Facebook like page which only has the purpose of stealing your email and password. Chances are that your Facebook password is the same as your email password which further compromises security.

I think 2010 will be the year of identity theft and compromised security. No longer do scammers have to write complicated coding, they can just use the tools already available to them to help fabricate their ruse. Check YouTube for a bunch of videos telling you how to do this already - that's how easy it is that 13 year olds are propagating it.

I agree the long term problems potentially brought by Facebook should be voiced loud and focussed on by government and human rights activists. However I think a basic literacy on how to use the internet is much more important for the short term as common criminals are much more dangerous in an immediate way.

I do think Facebook (and Twitter) are government funded operations that have the potential to go awry, but as long as we hold the leash and not the other way around it can be a useful tool for a great number of things.

Cheers for letting me vent!

-m0r

The only reason I joined Facebook was to access a relative's page. The next day, I logged in to FB and I was horrified to see thumbnail photos of the people from my e-mail contacts listed on the page under, "People you might know"... YA THINK?

I don't remember some of the exact verbiage used, but FB did make reference about obtaining the names from my contact list in my e-mail. I seriously don't recall seeing any request for my permission to go into my e-mail for these names. If there was, it must have been disguised in some obscure way because I would have never agreed to it.

Doesn't this just amount to them hacking into your e-mail to get the names of the people you correspond with? How can they do this? If they can do that, can't they read your e-mails if they want as well? I'm no tech expert, so I'm just stating what seems to be the logical conclusion for me with my minimal knowledge.

Account settings

Privacy settings

Who can view my photos

only friends

disaster averted.

Absolute Technologies and Absolute Statistics are buzzwords that people will be hearing much, much more of in this new decade.

Absolute technologies are the golden goose for most companies, they take the gray area out of market research.

You plug in your new coffee pot, it automatically connects to the manufacturer and registers itself. Of course this validates your warranty along with a few other crumbs they'll throw in. It now reports back, right through the power jack - who uses the coffee pot (it links up with your rfid), how often they use it, when they use it, etc. These statistics are absolute - they require no research and they leave nothing to question.

This is coming people, get ready. We are all about to be completely reduced to a large database of assets and liabilities. The only things standing in the way were advancements in information technologies, and they've caught up now. There is plenty of storage for massive information, and viable methods for collecting and moving it.

Originally posted by MemoryShock
And this is just a basic assessment based on website hits and subsequent activity regarding each website.

Who here thinks that they aren't already attempting to piece together personality profiles based on the entirety of one's internet habits?

Don't forget about In-Q-te (CIA!!) buying into Visible Technologies

Visible Technologies, which is based in the Seattle area, provides services that allow companies to monitor social-media activity. Companies tend to be interested in consumer opinions. With Visible Technologies' service, companies can view content from mainstream media, cultivate information from blogs, check out open Web 2.0 sites, read tweets, and more. Visible Technologies said its goal is to provide clients "with actionable insight into social-media conversations."

Visible Technologies, which is based in the Seattle area, provides services that allow companies to monitor social-media activity. Companies tend to be interested in consumer opinions. With Visible Technologies' service, companies can view content from mainstream media, cultivate information from blogs, check out open Web 2.0 sites, read tweets, and more. Visible Technologies said its goal is to provide clients "with actionable insight into social-media conversations."

link

I wonder how many Predator drones and snatch squads the CIA has reserved for 'action' after evaluating the 'insight'.

reply to post by LadySkadi


Caution! This Thread is already examined!

www.google.ch...

add:
Try search in google.com this without ": "site:www.abovetopsecret.com intext:google"

sorry, can't post a link, it's to complex..try by yourself with copy and paste. Be careful, any user can use google for data mining!



[edit on 15-2-2010 by cushycrux]

Originally posted by cushycrux

Try search in google.com this without ": "site:www.abovetopsecret.com intext:google"

I did. My recent thread on google isn't listed. Wonder why it's filtered out.

Google's Big Fiber Play: What Gives?


?????





[edit on 15-2-2010 by soficrow]