Internet Searching and Finding the Data you need.

Data, Data everywhere….but how in the ether do you find some?

Well, that is the question isn’t it? Actually the problem these days is entirely too much of it.

More knowledge and information is now available to the common person than at any time in known human history. It’s all there, somewhere, across the world-wide network of computers, servers and databases.


This is a thread I’ve been considering making for quite some time. In fact, I feel a bit like I owe it to the ATS community. I want to start with a word here as to the reason it’s being made, if you’ll bear with me.

I left trucking in late 2010 for medical reasons as well as a personal tragedy that changed my life in dramatic ways. I initially came to ATS seeking …information, I suppose. The sheer variety of forums promises information on almost everything, but debate as well. Community was also part of it, and in short order, as I saw the diversity and size. It’s an intimidating place at first. Lol….and I’ve been a part of large online communities before. Yahoo for awhile, AOL in the 90’s, and even some of the first multi-line chat systems running the old Galacticomm software. Never quite this large with the same high tempo.

I had some skill online when I arrived, having been “on-line” in various forms since dial up modem days. Still, there was nothing to really call ‘a skill’. ATS changed that, and the value of what I was learning began to dawn on me over time. Sometime last year, I realized the research and writing opportunities I’ve been enjoying here created a very real skill. Research itself. Quick, Effective, accurate, and straight forward to explain to others, as it happens. This last thought having come as an idea quite recently.

A few recent threads have sought more experienced member’s help and guidance in how to effectively search the internet and find stuff. I’ve helped a bit, but nothing like I could have…and that inspired this thread. After all, I wouldn’t have developed the skill I have without the community here.

So, this is a multi-page and colorful thread with screen shots, many links and graphics. What I’m sharing is a plain, straight forward explanation for some of the best and most effective ways to search for what you need.

Searching is an art, and practice is rewarding. Very much so.

Still, there is no reason why everyone shouldn’t be able to find what they want quickly.

The methods are not complex or difficult, just not necessarily widely shared. Double down on that for links. (folks tend to hoard what took real work to accumulate sometimes). Some people will know some or all of what I share here. If so, please feel free to add more if you can. If not, I’m glad if I could help.

[atsimg]http://files.abovetopsecret.com/files/img/kn5244e915.png [/atsimg]

Google is your friend, if you ask nicely…

Among some recent stats I found, some stand out for this section and why it matters. All but a few search engines use Google as the backbone or infrastructure. A good %, if not majority of websites use 1 or more google related services. Some other stats?

Number of Google Searches in July 2013: 12.9 Billion

Google share of US Search Market: 67%

Number of Monthly Unique Visitors to Google Sites: 192 Million

% of online ad revenue earned by Google: 33%

(Source)

2012 Search Engine Rankings:

The bottom line is, love ‘em or hate ‘em…you do use them (realize it or not) and they do have data on you. Like it or don’t….They have data on everyone and everything else too! So why not make full use of what they make available? They are one of the best sets of simple tools, accessible to anyone and without downloads or installs needed, let alone a tech to help. So, we’ll start there.

Time….they have a time machine! !

Google has a time machine?! Why…yes. As a matter of fact, they do. On news and events research, it’s literally your single most valuable ally. These days, any number of sources of events try to swamp the search engines to insure their side or spin, spins to the top. This isn’t about spin though, but the impossibility that makes to FIND something of VALUE not related to that day’s ‘big news’. So… about that time machine…… Let’s look at Syrian Chemical Weapons and just how long those things have been around.

If you simply type in Syrian Chemical Weapons, you get this:



Let’s say someone on a thread mentioned without much to support it, Syria has had the weapons forever and even used them before, a long time ago. How would you go about finding that out? Would you type in a different search string like “Syrian past uses of Chemical Weapons”? Well…. You could do that. You could also sort a half dozen pages of modern Newsweek to the NYT’s reporting their version of what happened 30 years ago. What do if you do have the date range? You might type in a special new search term with your general date until you get it right? (That can take a really long time…and I wish I didn’t know that personally…lol).

Meet the Google Date Filtering Tool.



After you have clicked on the above in the order they appear? You come to this:



That’s pretty useful and at one time, I spent the time it requires to flip back through the months to get the range I wanted. Let me make a very useful tool, even easier. Simply click the first of the current month, like shown above. It beats dozens of clicks. Then, change the years. Like so:



Now let’s see what our little trip back in time got us, shall we?



Notice, the search term didn’t change one bit. The links it comes up with are what they appear to be and they do load to contemporary reports, written in full context of the times and perspective.

I find that far more valuable. We have hindsight to “calibrate” out the spin of a decade or 3 ago. We don’t have that to even try and guess the validity through the spin of current reporting. This is the FAST way to pull source data, when source is simply what happened with a current story …outside the current times.

You’ll notice, the articles are even diverse and give a broader picture of things for that time period.

Boolean Search Terms……wait, Boo what?

It’s an odd term, but computer nerds are odd people. Not unlike many professions, they even made up their own language in some things. Boolean is a term many people are not familiar with. It’s really just techie-speak for saying ‘Get a search to find what I want and not everything I don’t!’

How Boolean works for normal folks in Google:

Exclude a word

Add a dash (-) before a word or site to exclude all results that include that word. This is especially useful for synonyms like Jaguar the car brand and jaguar the animal.

jaguar speed -car or pandas -site:wikipedia.org

Another very useful one would be:

Search for an exact word or phrase

Use quotes to search for an exact word or set of words. This option is handy when searching for song lyrics or a line from literature.

"imagine all the people"

What the above is also handy for is searching the source of a quote or a specific piece of data, but nothing else. Say someone has a quote but didn’t source it or you want to see how many other sources carried the same story? Pull a lengthy enough stretch out for the order of words to be fairly unique and just copy it into Google with the quotes. You should, on the first screen, get other places where that sentence or string of words has been used.

In addition to the above, “AND” doesn’t work for me, but + sure does. Let’s say you are doing a report on Hiroshima, Japan and radiation info. Well, typing:

Japan Radiation

……right now would be absurd to expect to find what you want if it’s from the 40’s…unless we’re at the anniversary of course. So, type this and you’ll get a very different result:

Japan Radiation +Hiroshima

The + sign should be at the word you want required as an added term in every result. No space between them. This is useful in a good many ways.

There is more than one type of search?

Don’t laugh too quick now. Seeing that other options exist and actually using them can be a good distance apart.

Some notice I can be Johnny on the Spot for a chart, graphic or other image for a thread. Often extremely quick, since ATS Live display is often running on one of my monitors to catch interesting threads. Well.. How? I’m not web searching for websites that contain the chart I want or the graphic of interest. I am directly searching for the image itself.



If you type in, say, “US debt ceiling raises” and you get a bunch of news articles and stories which can take a half hour to go through ….without ever finding something useful to share in a format like ATS. However…… Without touching anything else, click on “Images” as shown above and see how it changes.



That came, second row and third over from the top of the results ….on the Image side of Google.

Google is absolutely relentless and voracious. It eats everything, and I mean indexing. If something is up and is connected outward in almost any way, Google will find it and index it. This includes the millions of pages of Government documents. Page by page, Graphic by graphic.

If you type in:

ST Louis Federal Reserve M Numbers

… and you do this on the Image side? You will not only get the M numbers from the St Louis Fed (Which track raw money in circulation figures, among other things..) in easy to read graphic form, but you will find Photo copies of news articles from 1913, and readable in that format, to see how it was reported when it all began.

Boolean searching

[atsimg]http://files.abovetopsecret.com/files/img/wo5244fd92.png [/atsimg]
Many have said YouTube is the second most visited site in the world. Well, that may not be right. They are actually number Three.

Source: Alexa Rankings

Tubes come in all shapes, sizes and quality. The fart lighting is in there, and still as sad in some way as it’s always been.

Other very valuable things are there as well. Such as…

Guns, Germs and Steel:



This is a very informative look in some depth and across more than one episode, for the way in which Africa and South America were conquered by the European powers. What worked, why and just HOW so few men in parties like the Spanish Conquistadors could overtake enormous numbers. It goes into disease when it worked for the Europeans and when it decimated them in different places, and why.

A casual look at the right side bar that comes up with this, shows the value of what is up along this subject line. They aren’t 5 minute vids…but I’ll tell you as someone doing it right now, a college Geography or other classes can and are made up of videos like those. In fact, some of those exact ones…and you can watch it all free if you don’t mind skipping instructor lectures. (I know… Huge loss).

Some are kinda corny:

How beer saved the world:



I also came across that in a college course and as silly as it first sounds by the title? How silly does it become to learn Egyptians were rich with an anti-biotic science recently discovered even existed, it contributed measurably to the skeletal record which shows good health and ….it was in their beer. It was deliberately and by design, as some believe. It’s a serious tape and…yes, also was a part of a college course.

Finally, some are simply useful:

Access Databases:



I’ll say this about the value of that. Some sites charge $30-$100 and more for libraries of tutorials like that is a part of, and the other series on just that one program. What I was doing in a CIS course last year had some of the tutors stumped, for being beyond what they had done. (Oh…what happened to exploring software for the joy of learning?). That was during my ‘101’ course geared toward “This is what a ribbon is” level of MS Office. (It’s important on a transcript)

All of that, and I do mean all of it, was from Youtube video. I’m currently working on making Excel, Access and my FTP program all work together, automatically on a timer to keep dozens of feeds updated on a website I’m just about done with. Again….formally? I just have a 101 class. That’s the power of YouTube as it’s exploded over the past couple years into pro-areas and high quality stuff. I knew nothing of Access a year and a half ago.

I even built a website last year for the Geography Course on Kenya with complete tutorial sets to quickly learn basic Swahili. All Youtube sourced.

For those who want to simply be the very best at searching with Google and it’s properties? Here is the source for far more than I can share.

Alison – Free Online Courses

Alison offers up to University level online courses, 100% free. They are full, real and invaluable. The above link is just for those published by Google itself, through Alison.

Advanced Google Search Course

…and that is specifically for advanced level Google search use. Depending on the line of work everyone is in, there is something there for everyone and much more than that for most.

[atsimg]http://files.abovetopsecret.com/files/img/zb5244f620.png [/atsimg]
Now we’ve all been hearing about NSA leaks and CIA scandals. NRO budgets and DIA shenanigans. The media has seemingly given the impression that it’s all intelligence related and all out to get us.

Well. No, that isn’t how it’s happening. That *IS* happening, don’t get me wrong, but it’s SO much more than that. The Alphabet Soup agencies represent only a few percent of the overall effort happening right now. I’m guessing this is where part of the money is being spent, too, although in some ways, that level of detail is near impossible to get at without a staff and a half to go through department level budgets.

What I can say? The available online data has flat out exploded in recent years. We all probably know about current proposed and past actual budgets all being listed and available at the White House:

WH budget:

(I recommend “Historical Tables” for the most useful information. They may not be passing any formal budgets since 2009…but they sure know what they have spent, right into last year and YTD)

However, there is a great deal more at this point. Some of the more useful examples:

Data.gov

…as it says on the front page. Over 98,000 data sets, galleries and with apps galore. All free. All open to use (or almost all…some Gov sites carry limited sets of private copyrighted things, but rarely in my experience and clearly marked when they are)


Bureau Of Labor Statistics

They publish the unemployment figure, but they put out much much more than that. Regularly. Numbers seem to be a little fuzzy at times these days, but it’s still useful to watch for overall trending and explanations of factors they consider responsible for each.


Index of Government Agencies

All US Government things they have been able to integrate to date, in one jump off point. Searchable and indexed from the one spot. It’s not worth going into what is there but simply to ask what is NOT there which is intended to be public? The vast majority of what the vast legion of office workers do across the Government is public and published. Much of it and more all the time, at the above link.


Government Databases

Last but not least, an index of databases. All Government of US or other variety and linking back to official sites or records in most if not all cases.

Mileage greatly varies by state

It’s most definitely not just the Feds computerizing just about everything about putting it out into the public domain. Some is downright useful. For instance, in most states you have to pay to see if someone you know is a criminal. Not so in Florida, as one example:

Florida State Inmate Records

Complete with color mugshots for the discriminating researcher.


California State Water Lease Map

California wants to make sure you know who has rights to water, where……..


Local Missouri County Active Warrant List

While a Sheriff’s office in my local area figures the warrant sheets are valid public information and with regular updates.

If you haven’t checked your own state’s available data recently, you may want to look again. New things are literally coming up weekly in some areas.

[atsimg]http://files.abovetopsecret.com/files/img/tj5244f620.png [/atsimg]
Ahhh…. I’ve saved some of the best for last here in terms of what many people want to see and know. Many here already have some of these. Some have them all. Still, a good many people sign up new around ATS all the time, so it’s sure not fair to think everyone has them.

** Warning **

For those who have never seen “Disaster Index” sites? Watch them for a few days or weeks at least, before letting the volume of reports or activity get to you. Get a feel for what normal actually is, because some of it can be shocking at first. The media focuses only on the story of the moment, and rarely outside of documentaries, do they give an indication that Anthrax is out there infecting someone frequently, for instance. They don’t mention Ebola is something of a regular occurrence in small flare ups around some local areas.

Likewise, seeing how many volcanoes are currently in eruption status can be a bit of an attention getter, if it’s not a sight one is accustomed to for what is a usual level of activity.

General

RSOE Disaster Index

A master disaster list, one might call it. Most things worth knowing about pop up there in short order, it seems. World wide.

World Market Index

While not strictly a disaster index on a daily basis? It’s a useful running look at all the major global markets on one page. It most certainly looks disastrous when more of the change column is red than black. Double digits for red in any majority? Well…. Don’t walk below any high rise buildings in the financial district that day. Falling objects may be people, in a Great Depression/Crash way.

The Disaster Center

The name says it all for this one and I keep it for the quick link access to specific things I just recall that page has, that quickly. I include it here for that reason.


Global Incident Map

This one is like RSOE, but not as subtle about things. The quake area is a secondary reference to the USGS in my experience. Also, from that site and absolutely worthy of it’s own listing here?

Disease Outbreak Index

The global index of categorized disease outbreaks and news related to them, world wide. It’s broken down by illness type for the table list down the page. This is the one that can be most shocking to some to see the first time. It’s always very busy for things happening on a global basis. That’s what one has to keep in mind about it, IMO.

Quakes

Feel something moving recently? A few sites track it very well.

USGS Quake Index – New and Improved

I particularly recommend the map. It’s really something else and as good as anything else out there at the moment. They out did themselves on that one and it’s part of the new look they are working on with other agencies too.

EMSC-CSEM (European-Mediterranean Seismological Centre)



(EMSC Map Search Tool – Very Handy)

This is essentially a European version of the USGS for the data it reports and focus it carries. It also has a very useful search feature for both mapping a simple drag box to check past quakes to their text results.

IRIS-IEB



(Not Recent – for example only)

Iris is not to be outdone and now has their own interactive global map with what the side legend claims is access to around 3 million quake events. It’s a great stop now for both current and historic reference of what’s happening, with a unique way of color coding them for depth, apart from others out there. It makes looking for general patterns easier, if nothing else.

[atsimg]http://files.abovetopsecret.com/files/img/wn5244f816.png [/atsimg]
News sites are straight forward and everyone knows the major MSM sites. For headlines and information, there are a few others worth noting as well.

Headlnes

Breaking News

Drudge Report

Associated Press WIre

Reuters Wire Service

Local to World Newspapers – Sorted to city/town

Popurls – Very diverse MSM headline collection

Feature Services

Courthouse News

Hot Air

Economic Journal

• There will be two additional sections I just didn’t get a chance to complete this evening. They’ll be coming tomorrow (Friday) at some point and dealing with Real time feeds and finding some of the more sensitive things online. Not Wikileaks level, but just short of that.

reply to post by Wrabbit2000


Thanks for the great summary of features..

Thanks, very helpful

reply to post by Wrabbit2000

Thank you kind sir. Book marked under web tools for future "how do I-". Most worthy.

I would add a tool of my own. When a new story breaks the information is scant. New threads about it are looking for updates, evidence, pics.

Google-- Search Tools-- Anytime-- Past hour-- will bring new postings of news on the web.

Click on "Images" and the pics about the subject will reflect the time they were posted like "two hours ago", "18 minutes ago". Good way to uncover new information about breaking stories. Scanning the pictures is a lot easier to the minds eye and the time stamps on the pics help narrow the search to right now.

One other I have used, helped me to refine searches in YouTube...

YouTube Search Methods

Information should be shared... gulp. I hope others will be open and share their tricks so we can all improve our techniques.

Helpful stuff. Will be a shame that it will drop off the boards etc ...

Whilst researching isn't all the forums are about (or should be about) a lot of persons on ATS are trying to do it.

Edit: Dump it in your sig?

reply to post by Pinke

I'm glad people are finding it helpful thus far. I'm looking forward to getting the other two parts up after classes today.

You're right. Searching isn't everything but it's sure an important part of the whole experience, eh?

reply to post by Wrabbit2000

Terrific thread, Wrabbit. Good quality stuff, well presented. Hats off to you! And I have to say you taught me a couple of things about using the search engines that I never knew before, like customizing the time range, for example. Verrrry useful, that.

Many thanks!

Just a note about some of the catch-all "disaster" sites like RSOE. When taking info from these places it's good as a first resource but it's often wise to cross-check elsewhere if possible. For example, RSOE takes multiple data feeds for earthquakes from places like USGS and ESMC (that you've also mentioned as primary sources), and I've seen the situation where RSOE will post two significant quakes in a location where there was in fact only one.

Often it's a case of the "feed" sites having very slightly diferent times or lat/long locations for the given quake. But because RSOE don't filter this data very fast -- if at all -- it can be very misleading. So, with quakes, I'd recommend readers here check back with the actual data sources that RSOE gets their feeds from. And that goes double before starting threads like "Two huge quakes hit [name of place] within 30 seconds!!" when a quick check with a "main" site like USGS will show there was only one.

It saves embarrassment.

reply to post by Wrabbit2000


@Wrabbit2000,

Thank you for the useful thread. It was good of you to create it. [Thumbs up]

I have a question, however:

When I go to the google search page, I don't have a "Search tools" button/option. Thought it might be because I wasn't using Chrome, so I tried it with Chrome too, but still no luck. Is there something special one has to do to get the "Search tools" function, such as login to a google account?

Anyone else having this problem?

reply to post by MrInquisitive


I'm really not sure what would cause that? We use I.E. and Firefox at the college and I use it for research there frequently. Anyone else have any ideas why the tools wouldn't show on the google screen?

I just tried logging clear out of Google stuff and the tools are still showing? Odd...

reply to post by Wrabbit2000


Thx for the reply, and the checking you did on the matter. I tried it with Safari, Chrome and Firefox (all on a Mac, hence can't try it with IE), and no luck finding the "Search tools" button. I've been doing google searches on the matter, but so far no explanation or fix.

Edit: OOPS...:

I was looking at the black "bar" header that listed all the items you had, excepting the "Search tools" button, near the top of the google search page. I just read that the "Search tools" option shows up after doing a search. Tried this out and YES, it shows up along with the other items (on a white background) just as shown in your tutorial image.

Derpy, derpy doo...

Problem solved.

Thanks again for your response, @W2000.

nvm.... problem resovled.. lol

Wow, Thanks for the refresher.
It's been too many tears since I had a search tutorial and don't normally do comprehensive searches anymore.
I'll have to start again.

reply to post by Wrabbit2000
I am bookmarking this thread forever and ever Amen.

Hey everyone.. It would be rude not to update on a thread update that hasn't been yet. So, on what I'm working on for 2 additional sections on this thread, it'll be a bit for real life intrusions that took time away from otherwise important pursuits ..like this thread.

The flip side is, I'd been debating about getting into a more elaborate concept (best way to put it) relating to live feeds and data for regular people to make useful, and for the extra time? I'm doing the whole thing for that. That should make up a bit for the delay, I hope.

I'm not sure if the new material will really need to be a Part 2 Thread or add-on to this one. It depends on how much space the new coverage takes up. It's worth it though, for the technically able on computers? It's well worth it.

As the big guy in the movie said.. 'I'll be back....'