A thought about source integrity for stories...

I'm spending the day recovering from some major problems of the last few days, and am taking the time to catch up on some side projects. Among those are a couple Vietnam and Gulf war related things.

The most useful data for this kind of research is raw data. Spreadsheet files with 10's of thousands of rows excite me to no end. Database front ends on websites or tables someone else made, make me groan. It doesn't matter who made them, if it's the head of CNN or the National Archives or whether it's a humble ATS user with the very best of intentions. Once the original dataset is modified by so much as ONE letter in ONE line? Integrity is gone. Kaput. Shot.

What I am finding to my utter dismay and sadness is that a great deal of raw data which was available as recently as 12 months ago from National Archives and other places is now 'for-profit' offerings from the same .gov sites at fees even they can't possibly defend for taxpayer funded employees of a taxpayer funded agency to be peddling like they are. Essentially? A great deal is now lost to the general public vs. the open and easy access enjoyed in the past for casual research. (Serious and funded research isn't impacted ..it just won't be free to them, either)


So what is the moral of this story? PLEASE PLEASE...if you deal with raw files, do what I do with every one I get. Make a copy. 1st thing. Prior to doing anything else. Copy Copy Copy. Forget about the original. Archive it. Never touch it again, unless needing ANOTHER copy. Whatever you do though, please please don't modify or "scrub" a large original dataset for your own uses ...then trash the rest. SOME of what may be obtained that way, may not always BE available to get again some day. Some of it may become too expensive to ever get again. The copy that is destroyed, may be the only one you'll find you could ever get over time.

* Specific examples on this are real easy and frustrating in this case. I'm looking for the complete, 100% comprehensive data record for every Naval Gunfire support mission in the Vietnam War, with details for each, running from 03/1966 to 01/1973. Plenty of references can be found, I've discovered, for how people downloaded the WHOLE thing, as recently as 2012. I find all kinds of tables ..of just specific vessels, culled from the overall set. It's neat for a trivia game. It's worthless as a research tool in that form. (sigh) more lost to the ether and profits of Government.

Something to keep in mind while researching then. Copy, Archive, keeP. CAP every thing you get, because you just may not be able to get it again.

In my more paranoid days, I used to keep offline copies of anything I came across of interest.

Fell out of the habit for some reason, but the more I think about it the more it makes sense in this day and age of NSA spying and government censor ship.

Hell, another shut down could see things like Library of congress and various other government funded information source shutting down as well. Maybe even closing indefinitely in another crisis as those seem to be less vital services.

reply to post by benrl

Yep. Same here. At one time, I had thousands of whole web pages and screen captures. I finally deleted all of them in a momentary lapse of reason. It wasn't long before I realized what a blunder that was. I had a lot of stuff in there, I've never found again.

reply to post by Wrabbit2000

Sorry Wrabbit. I got a little to the side of the topic there. I have a feeling raw datasets are going to keep getting harder and harder to come by. Your advice is good, because they're going to become a rare commodity. A lot of gems can be found in raw data. And obviously, we aren't the only ones aware of that.

Klassified
reply to post by benrl

 

Yep. Same here. At one time, I had thousands of whole web pages and screen captures. I finally deleted all of them in a momentary lapse of reason. It wasn't long before I realized what a blunder that was. I had a lot of stuff in there, I've never found again.

Yea, I felt a little stupid I left an old kindle three feet buried in my back yard filled with survival, medical, and general tech books, as well as huge amounts of fiction and history books. Had it in a lock box with a hand cranked charger and in an anti static bag.

Took me only an hour to prep and store when I got my new one, anyway so now 1200 miles away buried near my old place is a cash of info I can't reach...

I may have to do that again when I upgrade, I was really paranoid about solar flares a few years back.

Well hey there Wrabbit! Sorry you have had some challenges the past few days!!

Here is an idea... Is there a way to put out a call on ATS - a sort of "research beacon" for specific things you (or others) may need? It is possible that someone has that tucked in an old hard-drive somewhere and would happily hand you a full copy? A new thread or even a forum for sharing data (obviously within T&C)?

Just a thought. The Scholars in their secret researchery-lair might have some good ideas for that too??

Good luck to you in finding the data you need!

peace,
AB