Huge UFO Database in DOS - Someone's labour of love, can it be saved?

My endeavours are currently producing good results from the demo Db,e.g.:

"ID","Year","Unknown1","Month_Hi","Month","Day_Hi","Day","Time","Unknown2","Duration","Unknown3","Longitude","Latitude","Un known4","Location","Attributes","Description","Reference","Unknown5","Strangeness","Credibility"
"#1","1952","0x10","0x0","07","0x0","05","19:50","0xFF","3","0x0","97:27:0W","35:9:0N","0x2","OKL.","MAP|GND|AIR|CIV|SC R|NOC|OBS|SIG","SSW/NORMAN,OKLA:BBK#1390:3 SCRS HVR until COP/PVT PLANE ASKS ABOUT THEM/RADIO","88","0xF1","7","6"
"#2","1952","0xD","0x0","07","0x0","05","21:10","0xFF","2","0x0","73:25:0W","41:5:0N","0x2","CNC.","MAP|CST|CIV|NLT|NOC ","off NORWALK,CT:2/BOAT:INTENSE LO STAR BECOMES 2 SML NLTS:ZIP . UNTIL LOST/SKY","88","0xF3","5","7"
"#3","1952","0x2","0x0","07","0x0","06","00:00","0xFD","1","0x0","7:39:0E","36:57:0N","0x76","CNN.","MAP|GND|CST|CIV|SC R|FBL|NOC","BONE = ANNABA,ALGERIA:LUMINOUS SPHERE SEEN:OBS+TIME UNK:NFD","02","0x54","4","5"
"#4","1952","0x2","0x0","07","0x0","06","12:00","0xFD","3","0x0","7:5:0E","47:49:0N","0x35","HRH.","MAP|GND|CIV|SCR|NOC ","THANN,Ht RHIN,FR:2 BLUISH DISKS SEEN:OBS+TIME UNK:NFD:/JIMMY GUIEU.","02","0x54","6","6"

There are still a couple of lookup fields missing (e.g. Reference) and some unknown fields but the core data can be produced...

Of course, I have the complete Demo Db converted to CSV barring the lookups and other fields which I haven't converted. I haven't managed to spend much time on this so the above constitutes an hour's snatched work this morning, but, if anybody has anything to add, let me know...

Originally posted by SugarCube
There are still a couple of lookup fields missing (e.g. Reference) and some unknown fields but the core data can be produced...

The "print-to-file" method has produced a list which includes the Reference and some of the other "unknown" fields, but without quite a few of the fields that you have captured (particularly the attributes matrix).

So, the two sets of records complement each other wonderfully.

If the other lookup fields cannot be resolved by other means, then I think we should be able to merge the two sets of records by some method (which others are probably better able to devise than me...) - which should result in virtually the entire data being salvaged.

I've been impressed with the progress made by ATS members during the last week or two, after several years in which this database has been uncompatible with new computers.

All the best,

Isaac

reply to post by IsaacKoi


BTW, I assumed that the YMDT field is an assurance of the accuracy the date and time fields (i.e. 3 = definite, 1=guess). I don't have a description of the fields as such and so this is a guess in itself!

As such, values such as "3333" would mean a definite date and time for the sighting whereas "3331" would indicate that the time is a bit iffy.

Maybe there is a better description of this fields...

For those that may be working on the database without having the demo working, here are some screen images for the first record (when we are editing it).

Date and time accuracy:


Continent/Area:


Country/Region:


Duration:


Terrain:


Attributes:


Strangeness:


Credibility:


References:


[edit on 4/7/2010 by ArMaP]

Originally posted by SugarCube
reply to post by IsaacKoi

 

BTW, I assumed that the YMDT field is an assurance of the accuracy the date and time fields (i.e. 3 = definite, 1=guess). I don't have a description of the fields as such and so this is a guess in itself!

That's basically right.

I posted a relevant description from Larry Hatch's manual in the middle of the post above at:
www.abovetopsecret.com...

All the best,

Isaac

Is anything still needed for this? I find myself interested. I'm an amateur programmer and so could probably help with that side of it.

It would be neat if someone could make a google spreadsheet with some of the demo data, and hack together an online-searchable version. It should be possible to enter query requests and see results automatic tagged on a dynamically generated map.

googlemapsapi.blogspot.com...
googlesystem.blogspot.com...
gmaps-samples.googlecode.com...

@ Ian & szpiegel,

We basically have the data extracted, it is now just about deciding how it should be presented without compromising the original copyright and in such a form so as to ensure the sustainability of the data in a normal form (i.e. making it useful for searching and statistical analysis).

The should really be the next step. What does the 'UFO community' want from a database? What type of form should it be in? Can we integrate the core data into a form that also enable the submission of new data?

I like the idea of using this data as the seed for growing a properly administrated dataset in the public domain (i.e. internet based). Now THAT would be worth having!

reply to post by SugarCube

I don't know how things are in other countries, but in Portugal we have two types of copyright in cases like this: the database structure copyright and the data copyright. One company can create, for example, a program based on a SQL database, and they own the copyright of the structure they created, but when someone else uses that program, that person owns the copyright of the data inside the database.

That would mean that the data already in the U database may also be copyrighted, and in that case it cannot be used without a specific authorization. I don't know if that authorization was included when someone bought the program, but if it wasn't, IsaacKoi can only give away his own records, not the ones that came with the program.

reply to post by IsaacKoi
I've been following the thread and admiring how a collaborative effort can yield results. The name 'Larry Hatch' kept ringing a bell, but I guessed I was thinking of Harry Patch...the late Great War veteran. I'm sure you know who I mean.

By coincidence, I was reading the NIDS report about black triangles and they cited Larry Hatch's database as a primary source for their research.

One or two significant researchers have passed away and their archives have been lost. Receiving permission from Hatch or family to ensure the survival of the archive could be important. At the least, it could be his legacy.

I think that the 'Copyright' issue is the key at the moment... I know that there are folks out there that say 'this should be preserved regardless of copyright' but the fact remains that it is copyrighted and it is right and proper to respect that, regardless of whether it is inconvenient.

Purely from a 'prototype' perspective we know that we can save the data - so it is SAFE! The rest is leg work in getting an authorisation from the appropriate current copyright holder and I guess that we need a bit of 'group effort' to progress this aspect, in identifying the relevant 'owner' and obtaining that permission.

Can anybody help with that? We don't want to see posts just 'talking the talk' but actually 'walking the walk'!

Originally posted by SugarCube
The rest is leg work in getting an authorisation from the appropriate current copyright holder and I guess that we need a bit of 'group effort' to progress this aspect, in identifying the relevant 'owner' and obtaining that permission.

Don't worry I (and one or two others) have not been ignoring the possibility of getting consent from the copyright owner to distribution of the database. There are one or two relevant bits of information that are currently confidential, hence the lack of discussion of relevant issues in this thread. Sorry to be a bit vague here.

However, my focus has been on:

(a) getting the database to work on modern computers for those (including me) that bought a copy of the database years ago but simply could not run the application on new computers (until various ideas were put forward and successfully tested in the course of this thread). This can now be ticked off the "to do" list.

(b) extracting the data so that it cannot be used by such owners - which your work has virtually solved.

The potential sharing of that data more widely is a more ambitious project - but one that I'm certainly keen to see happen and have some hope of seeing.

All the best,

Isaac

reply to post by IsaacKoi
IsaacKoi,
Has the APRO archive been opened? The last time I heard about APRO's files, they were locked away following the demise of the organisation. They seem to share a similar significance with your current attempts to save this database.

If APRO's archive remains locked in Scottsville, it might represent an opportunity for ATS to secure open access and longevity to the files? Just a thought.

Originally posted by Kandinsky
reply to post by IsaacKoi

 

IsaacKoi,
Has the APRO archive been opened?

No, unfortunately.

The last time I heard about APRO's files, they were locked away following the demise of the organisation. They seem to share a similar significance with your current attempts to save this database.

I made a similar connection in my mind a few weeks ago and looked into the current situation regarding the APRO files. Coincidentally, there has actually been a recent development in relation to the holders of the APRO files that may represent an opporunity for CUFOS to get hold of the files. I've been meaning to email CUFOS about this for the last few weeks, but it hasn't been at the top of my ufological "to do" list (and I only have a rather limited amount of time for ufology due to work, wife, kids etc).

Anyway, I'll start a separate thread on the relevant development in relation to those files after I've emailed CUFOS.

All the best,

Isaac

[edit on 9-7-2010 by IsaacKoi]

Hi all. Sorry I was away for a while, I had a lot of work on.

I'm glad to see a few more people weighing in on this.

I am hoping to have more time and energy for this towards the end of this week. I'd like to get this fully solved if only for my own satisfaction.

I never got any more solved than I have posted in this thread in my two info posts, but with those screenshots from Armap, I think there's a good chance i can solve a couple more of the fields without too much pain if sugarcube or others don't beat me to it.

Good to hear people are still working on the copyright issues too, that's the real bottleneck I guess now, the technical side seems covered by good people.

If anyone has any queries or questions for me, feel free to U2U.

reply to post by harpsounds


We're kind of there actually harpsounds, but good on you!

The only thing left is the lookups, but since these are keyed values they basically don't need to be x-reference until a suitable database mechanism is in place (i.e. retaining keys to lookups and simply completing the lookup tables).

The data extraction tool is up and running and I believe that The Good Barrister has just about everything he needs from the technical perspective!

For anyone still interested, the Larry Hatch database has been back online for a couple of months.

*U* UFO Database

Can't really help at all, just wanted to say thanks for such a great thread.

This thread should be given more exposure, there's some great statistics on Larry Hatch's website, including stuff like UFO sightings being greater when Mars is closest to the earth:

www.larryhatch.net...

Feel free to check out more of the stats:

www.larryhatch.net...

This is still being worked on?

From reading there seems to be some copyright issues. Y'all should know that databases typically only have their structure copyrighted. Typically, all data is public domain.

It should be rather easy to design a database to contain the available data without violating any copyrights. From there you can put it on a real database engine. No offense to all those who have put in some serious hard work on this research; but, y'all don't do so well with database design.

Also, you absolutely need to make this database work on a SQL engine. Something like SQL Server will give you the ability to do Full Text searches, as well as search for any given data. It will also provide a reporting engine.

Doing this will provide you with the means to properly exploit the data, and perhaps some mining as well.

I have this "notion" that IF UFO research could have a "proper" database; the whole UFOlogy can be understood, and proven. Databases are wonderful tools, but only IF you have the tools to use one.