Huge UFO Database in DOS - Someone's labour of love, can it be saved?

Originally posted by AnthraAndromda
This is still being worked on?

Hi AnthraAndromda,

Sorry for the delay in responding to your question - my life has been a bit hectic recently.

Anyway, the current position is that, as a result of generous technical assistance on this forum, I have:

(1) Larry Hatch's database running on several of my modern computers (up to and including a 64 bit Windows 7 machine). I was also able to post a guide in this thread (and on a couple of other forums, giving credit to help from ATS members) to help other people that own Larry Hatch's database to get it running on their modern (non-DOS) computers.

(2) converted much of the data in Larry's database to various formats, including most simply and usefully for my purposes, a 1,208 page PDF file listing each entry in chronological order.

From reading there seems to be some copyright issues. Y'all should know that databases typically only have their structure copyrighted. Typically, all data is public domain.

Well, I think the person that spent a heck of a lot of time condensing and compiling information into that database (Larry Hatch) takes a different view and he has not (so far...) been willing to give permission for the free sharing of the data within his database.

I haven't given up hope that Larry can be persuaded to share his database - or at least update it and make it available online in one of various possible ways.

This evening, I've received permission to make Larry Hatch's huge UFO database ("the *U* Database") freely available online. I've been after this permission for about 7 years.



I'll include the database in various formats within the rather large (final?) item I'm planning to post to tie up various loose ends.

I'll post a link in this thread once that material is online.

a reply to: IsaacKoi

That is some patience. Wonderful to see data not lost from the public.

Good work.

originally posted by: roadgravel
That is some patience. Wonderful to see data not lost from the public.

Cheers. I have one or two UFO projects that have been underway longer. I'd like to wrap those other long-term projects up as well in the large (final?) item I'm planning.

originally posted by: IsaacKoi

This evening, I've received permission to make Larry Hatch's huge UFO database ("the *U* Database") freely available online. I've been after this permission for about 7 years.

That is simply awesome... I applaud you for saving this precious compilation of data. This is definitely the best news in ufology in a long time!

As far as I know, Larry's entries were "geotagged" so that one could map sightings across the globe and I think his DOS version already included the capability to visualize mutliple sightings on a global map (quite minimalistic in its style, but still incredible when thinking about what was accomplished back in the early days).

Imagine if that data were available in a Google maps compatible format. From that, one could expect to derive further insights, eg. was there a connection between similar sightings that occured globally within a certain period of time. That would require to limit the total amount of entries displayed on the map to those that occured in a specific timeframe (eg. all those that occurred in June 1952 or on a specific day). Maybe one could even filter additional parameters like sightings categories (nocturnal lights, daylight discs etc.). It's just an idea (and certainly a lot of work) but seeing all those cases on a global map could perhaps lead to new insights and possibly reveal inter-related sightings based on this data.

In any case, this is fantastic news. Keep up the great work!

a reply to: IsaacKoi

Great work and Great news!
I can not wait.

If given the time, I'll look at the data and see if I can be of help.

originally posted by: IsaacKoi
This evening, I've received permission to make Larry Hatch's huge UFO database ("the *U* Database") freely available online. I've been after this permission for about 7 years.

I'll include the database in various formats within the rather large (final?) item I'm planning to post to tie up various loose ends.

I'll post a link in this thread once that material is online.

Does permission extend to hosting it as an online-accessible database, ie export it to mysql and give it a user-friendly front end?

That can be arranged if permission is there.

originally posted by: EvillerBob
Does permission extend to hosting it as an online-accessible database, ie export it to mysql and give it a user-friendly front end?

I think so. The permission I've received is unrestricted, so I plan on making the raw data available so that people can do their own thing with the data.

I've previously done some quick conversions of the data to some other formats (with help from members of ATS on this thread any by email back in 2010). Don't laugh, but the format I've found most useful for quick searches during the last few years is a basic export I made of most of the fields into a searchable PDF.

Mysql is beyond me, but I think some others may want to work with the raw data to other formats.

originally posted by: IsaacKoi

originally posted by: EvillerBob
Does permission extend to hosting it as an online-accessible database, ie export it to mysql and give it a user-friendly front end?

I think so. The permission I've received is unrestricted, so I plan on making the raw data available so that people can do their own thing with the data.

I've previously done some quick conversions of the data to some other formats (with help from members of ATS on this thread any by email back in 2010). Don't laugh, but the format I've found most useful for quick searches during the last few years is a basic export I made of most of the fields into a searchable PDF.

Mysql is beyond me, but I think some others may want to work with the raw data to other formats.

mySQL isn't beyond me, hence my question

As long as there aren't going to be any permission issues, I would be interested in setting up a publicly accessible online database with a web interface.

originally posted by: EvillerBob
As long as there aren't going to be any permission issues, I would be interested in setting up a publicly accessible online database with a web interface.

The permission I've received is to allow the data to "to be shared freely" and post whatever I want, the "UFO database included".

So, I don't see any problem with you (or anyone else) setting up an online database with the data.

originally posted by: IsaacKoi

originally posted by: EvillerBob
As long as there aren't going to be any permission issues, I would be interested in setting up a publicly accessible online database with a web interface.

The permission I've received is to allow the data to "to be shared freely" and post whatever I want, the "UFO database included".

So, I don't see any problem with you (or anyone else) setting up an online database with the data.

Good news. Now we just need to wait for it to be released so I can get my hands on the data!

originally posted by: EvillerBob
Good news. Now we just need to wait for it to be released so I can get my hands on the data!

Oh, I don't want to be a tease so I'll sort out getting you the data in advance of the thread I'm planning about various loose ends.

originally posted by: IsaacKoi

originally posted by: EvillerBob
Good news. Now we just need to wait for it to be released so I can get my hands on the data!

Oh, I don't want to be a tease so I'll sort out getting you the data in advance of the thread I'm planning about various loose ends.

What would be useful would be even a single record with example data so that I can see the structure needed. All in your own time, of course, the world isn't going to end if it's not done by Christmas!

If any of the older hands are still tracking this thread, can I ask how you got the DMS from the lat/long fields? The conversion seems to be defeating me somewhat, there's some voodoo going on somewhere

originally posted by: EvillerBob
If any of the older hands are still tracking this thread, can I ask how you got the DMS from the lat/long fields? The conversion seems to be defeating me somewhat, there's some voodoo going on somewhere

In case it helps, I'll just mention that one of Nablator's posts in this thread said the following in relation to lat/long:

www.abovetopsecret.com...

Lon/Lat coordinates are encoded in two bytes, just divide by 20 the number of seconds.

I sent the full database and data to one software person a few days ago. I'm waiting to hear back from him at the moment. I don't mind sending it to others as well (or posting some of it here), but would be worried I'd be wasting other people's time duplicating effort.

(There a privacy issue in relation to one file, since it includes my real name. I've edited the file using a hex editor to change that name to my pseudonym - but the database program then says that there is a user name mismatch. I could share the full data file now and also the database file - but with my real name edited out, the database file would not execute properly until the relevant user name check issue is addressed).

In the meantime, here is a 1,209 page searchable PDF document I created with most fields exported from the database into a chronological listing of the relevant UFO cases. For bibliographic (as opposed to statistical) purposes, I've found this PDF quite useful a few times over the last few years (but felt I shouldn't post even this limited data until I got permission to do so). The AFU in Sweden will shortly be providing a permanent home for the full data in various formats.
we.tl...

originally posted by: IsaacKoi

originally posted by: EvillerBob
If any of the older hands are still tracking this thread, can I ask how you got the DMS from the lat/long fields? The conversion seems to be defeating me somewhat, there's some voodoo going on somewhere

In case it helps, I'll just mention that one of Nablator's posts in this thread said the following in relation to lat/long:

www.abovetopsecret.com...

Lon/Lat coordinates are encoded in two bytes, just divide by 20 the number of seconds.

Ahhh, missed that little gem, that might be the missing piece of the puzzle, thank you.

All the other key data seems readable from the data file - below is a record output from a php file I'm using to parse the demo data file. Forgive the formatting, it looks much nicer in a table!

[3] Record Start
Year 1952
Terrain 18 [ Mil. Base ]
Month 01
Day 01
Time 22:00
YMDT 3:3:3:2
Duration 3 minute(s)
Unknown2 b[00000000] d[0]
Longitude b[00110111 11100000] [W] d[14304]
Latitude b[00100000 01111111] [N] d[8319]
CONCTR 0 / 1 [ NAM / CAN ]
Location ONT
Attributes 10100011: 00000000 : 00000001 : 10000000
Text NORTH BAY,ON:2 ORG SCRS ZIGZAG ALL OVR BRAND-NEW RCAF AIRFIELD:SUPERSONIC
Reference 172 [ KEHOE,Donald: FLYING SAUCERS FROM OUTER SPACE; Henry Holt,NY 1953 HC. ]
Pinpoint 128
STRACRE 6 / 9

I'm going to use this as a basis for exporting the data to a database. The hard work of identifying the fields was already done earlier in the thread, thankfully.

originally posted by: EvillerBob
I'm going to use this as a basis for exporting the data to a database. The hard work of identifying the fields was already done earlier in the thread, thankfully.

I've sent you a U2U.

I don't want to waste your time duplicating work when I'd already asked someone to look at similar issues, but if you are working on them anyway then you may as well have a bit more material to work from.

originally posted by: IsaacKoi

originally posted by: EvillerBob
I'm going to use this as a basis for exporting the data to a database. The hard work of identifying the fields was already done earlier in the thread, thankfully.

I've sent you a U2U.

I don't want to waste your time duplicating work when I'd already asked someone to look at similar issues, but if you are working on them anyway then you may as well have a bit more material to work from.

I like interesting projects. I'm a sad old man who enjoys stuff like this, even if it doesn't go anywhere

I think I've replied. The U2Us here don't seem to like me very much...

Evillerbob asked me to post some sample records from the database to compare to the reverse engineered data he has extracted (and fill in some gaps).

Here are some sample screen shots of the last few records, just as examples. If anyone needs anything more, just ask.