Secret Facebook Search Engine

Facebook is definitely getting pretty big these days. A Harvard University science student goes from no fame to seemly instant success. I guess if you have a really cool project, paying that Ivy League tuition may actually be worth the investment.

Well I've heard of a few conspiracies that Facebook may be sharing some of their data with HS. I pretty much shrugged that off as with 80% of the rest of the nonsense I see floating about the Internet. That was until I ran into something interesting in a server log of a website I put together less than two days ago. I saw the following entry under the User Agent.

Unknown browser: facebookexternalhit/1.1 (+www.facebook.com...)

Hardly anyone knows about the new website - its not even finished yet! When I pull up the link provided here, this is the explanation I get.

Why does Facebook appear in my server logs?

Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the temporary display of certain images or details related to the web content, such as the title of the webpage or the embed tag of a video. Our system retrieves this information only after a user provides us with a link. You may have found this page because a Facebook user sent a link from your website to other Facebook users. If you have any questions or concerns about any links or content sent by one of our users, please contact us at [email protected].

There is no way someone has linked this site to their Facebook account already. It's an impossibility. So I decided to log onto my Facebook account, which I hardly ever use. I typed into their search bar some of the key words of the website I'm working on. Zero results showed up.

Honestly, I am Pro-Cloud. I think it's great to have an on-line backup of data that I personally give access too. But I can see Facebook is retrieving information for a system that isn't publicly available. That's just my thought. I wanted to post something here for your opinions.

I would think anything that is being hosted would have a path from the main framework. It may not be public but still on the system.

reply to post by CodeRed3D


Do you have a static or dynamic IP from your ISP?

You give so little info in regards to YOUR setup, there's a bazillion ways this can occur.

What information do you think it is retrieving? Its nothing more than a bot that collects data from a website when a user provides a link to that website in order to direct a user to the originating source. It does not appear to be a crawler in the sense that it just randomly traverses the web indexing data like say Google does.

Thanks! I appreciate your input. I'm not linking to the site or explaining the setup in detail for the sake of my anonymousness. It's a stock Wordpress installation with no cron jobs installed in a directory of an existing website. The site has only been up since this past Sunday. I half expected to see a Google bot as a first visitor.

Instead, I see what appears to be a Facebook bot.

I don't have a facebook account , but I have a question that some of you may be able to answer .

My son and daughter-in-law came to the house one night and she pulled her cell-phone from her purse and said "That's odd" . We asked her what she meant , and she showed us a detailed , running list , of everywhere she had been for the last hour or so . What I mean by this , is the list showed her location , from minute to minute , giving locations all the way to my house .

For instance , "time : x:xx , member at 1234 West XYZ street", and it showed a running list , meaning that she was being tracked as she drove , and the location updated every minute . And , it showed to be somehow connected to facebook .

Has anyone else seen this ? Is it normal for facebook members , or what the heck is going on ?

reply to post by okbmd


That sounds like foursquare, or something similar. I have no idea how foursquare works, so I don't know if it is an automatic update of locations, or if you must approve before the posts show. I do know several of my friends use foursquare, and I think it is ridiculous. I don't care if someone "Just checked in at Red Robin/Toys'R'Us/Target/amillionotherplaces".

Originally posted by okbmd
I don't have a facebook account , but I have a question that some of you may be able to answer .

My son and daughter-in-law came to the house one night and she pulled her cell-phone from her purse and said "That's odd" . We asked her what she meant , and she showed us a detailed , running list , of everywhere she had been for the last hour or so . What I mean by this , is the list showed her location , from minute to minute , giving locations all the way to my house .

For instance , "time : x:xx , member at 1234 West XYZ street", and it showed a running list , meaning that she was being tracked as she drove , and the location updated every minute . And , it showed to be somehow connected to facebook .

Has anyone else seen this ? Is it normal for facebook members , or what the heck is going on ?

Facebook mobile has a thing where it you can tell your facebook friends where you are. Its called 'places'. Its supposed to ask you if you want it turned on or off. I used it once but Im sure you have to manually say where you are, at home, at the bar, at the shops etc. I never use it. I dont want people to know when Im not at home

LOL! That's hilarious. I think it's pretty ironic that most people are so enthralled with the new technology that they don't mind exposing their intimacy. I think you can come up with a similar situation growing up.

Instead of Facebook Mobile or Foursquare, why not take a trip down to the local Sheriff's station and ask for an ankle bracelet?

I would say most people rather their shackles be self imposed, versus imposed by authorities (sheriffs).

Originally posted by CodeRed3D
LOL! That's hilarious. I think it's pretty ironic that most people are so enthralled with the new technology, that they don't mind exposing their intimacy. I think you can come up with a similar situation while growing up.

Instead of Facebook Mobile or Foursquare, why not take a trip down to the local Sheriff's station and ask for an ankle bracelet?

edit on 26-1-2011 by CodeRed3D because: (no reason given)

reply to post by CodeRed3D


Add the user agent to robots.txt on the server, that should take care of the problem. Same goes for any other bots you dont want hitting your server. Of course it doesnt answer the question of what promted the bot to crawl your site especially if they were not provided with a link. It would also be interesting to know if it does this only with links posted on someones wall, ect or whether it does this with links exchanged during a chat session. The first would be reasonable, the latter would be a little suspicious imo.

Interesting find! I am going to look into this some more for sure.

Originally posted by dasysgod
Interesting find! I am going to look into this some more for sure.

Yeah, I'm rather surprised that more people haven't responded to this. The ramifications are pretty huge. Because of the amount of very personal information collected by Facebook, any bot technology being developed by them would be extremely invasive. Also, to collect information from your customers that doesn't constitute as part of the public service is a violation of privacy. Facebook doesn't need to be caught using bots.

reply to post by CodeRed3D


What issues with privacy are there? The bot would only have access to the exact same information any passing member of the public would have access to. That being the case, it's down to site admins or those who are making the information available, to ensure that access to that information is restricted only to those they wish to have access to it.

Its like leaving your lounge curtains open then complaining that passers-by keep looking in.

Originally posted by quackers
reply to post by CodeRed3D

 

What issues with privacy are there? The bot would only have access to the exact same information any passing member of the public would have access to.

That's a great question! I'd be happy to take that topic on. It's not always about the amount of information available. The danger is how you prepare it. A good example of this would be the sheer amount of cables released by Wikileaks. When people know what to look for, that's when the information is dangerous. Facebook has an amazing insight into a persons likes, habits, activities, interests, and psychological make-up. Tailor a search engine using that information, along with the personal notes, contact information, etc, a lot of info can be instantly asserted through the Facebook account and most of their contacts.

reply to post by CodeRed3D


But this isn't about a search engine in the strictest sence of the word. It's about a bot that gathers metadata. It's not quite the same thing. The bot simply gathers publicly available data from user generated content where that content is off-site. If a user posted a url to an image on your site then it's the bots job to gather any information from your site about that image. Where are the privacy implications in that? If an administrator objects bots crawling their site for such data then there are ways to prevent them from doing so. It's not a secret either. The page you linked to earlier is there to inform admins what the bot is and why it was visiting their site.

Originally posted by quackers
reply to post by CodeRed3D

 

But this isn't about a search engine in the strictest sence of the word. It's about a bot that gathers metadata. It's not quite the same thing. The bot simply gathers publicly available data from user generated content where that content is off-site. If a user posted a url to an image on your site then it's the bots job to gather any information from your site about that image. Where are the privacy implications in that? If an administrator objects bots crawling their site for such data then there are ways to prevent them from doing so. It's not a secret either. The page you linked to earlier is there to inform admins what the bot is and why it was visiting their site.

I'm sorry. I honestly can't explain this any clearer. If a website has enough personal information to do a profile on a person, create a bot to collect information connected to that profile, that information is worth millions to private intelligence firms. I just don't feel that Facebook should be using bots, especially when it pertains to the gathering of information that isn't intended for public use or to better their public service. That's where I feel people's privacy is being invaded.

By seeing the lack of popularity of this thread, most people feel the same way you do, quackers. So no offense. This is just my personal opinion on how I feel about the subject. But honestly, it is the lack of concern that will continue to make Facebook a force to be reckoned with.

so what is being said here is that facebook has a way to access private information and disperse is accordingly to a specific target audience. the possibilities of the internet are frightening to my privacy.

Originally posted by CodeRed3D

I'm sorry. I honestly can't explain this any clearer. If a website has enough personal information to do a profile on a person, create a bot to collect information connected to that profile, that information is worth millions to private intelligence firms.

That's not what the bot does though. Facebook users gladly submit their own information to facebook, it's all there on their database. They don't need a bot to gleen information when the users themselves submit that information. The bot in question gathers extra information, not personal information about users, but about external content users submit.

I just don't feel that Facebook should be using bots, especially when it pertains to the gathering of information that isn't intended for public use or to better their public service. That's where I feel people's privacy is being invaded.

As I stated, the bot gathers information that is already publicly available. Or are you suggesting that this bot hacks sites to gain information stored in private directories?

By seeing the lack of popularity of this thread, most people feel the same way you do, quackers. So no offense. This is just my personal opinion on how I feel about the subject. But honestly, it is the lack of concern that will continue to make Facebook a force to be reckoned with.

Hey we're all entitled to our opinion. I think the lack of interest is mostly down to your own misunderstanding of what this bot does. In otherwords you've got the wrong end of the stick. Facebook isnt the only place on the web that has bots that operate in this fashion. They are everywhere, and they are an efficient method of information gathering. Don't get me wrong, I'm no fan of Facebook. But the responsability for privacy lies with those supplying the information to Facebook, the end user, not some bit of code whose purpose is to gather related data for information supplied that in reality has little to do with individuals privacy.

Originally posted by quackers

That's not what the bot does though. Facebook users gladly submit their own information to facebook, it's all there on their database. They don't need a bot to gleen information when the users themselves submit that information. The bot in question gathers extra information, not personal information about users, but about external content users submit.

I or no one else submitted information about the site that the bot was crawling. The site had only been on the net for three days. I tried their search engine and found no information regarding my site. So the information the bot was gleaning was for a private Facebook database to use for advertising or whatever. I do understand what the bot is doing, Facebook is crawling sites for profiling (associating data with existing accounts), not because a user submitted a Facebook Link. And how would a link be posted on someone's Facebook page when nobody else knew about the site?

Maybe the title of this post should have been "Facebook Has a Crawling Bot". I realize I hadn't filled out the robot.txt, but I had planned on making the site public eventually. I was just surprised that Facebook would be using this type of tech.