Huge UFO Database in DOS - Someone's labour of love, can it be saved?

I can't understand anything to this thread, but this is so exciting !

Good luck guys, and good thread OP !

Well this here is some good info on anyone who is trying to run old windows 3.1 things. Personally i hate virtual machines with a passion which is why i still have my old rigs. But for those who do not have old comps check this out.
vogons.zetafleet.com...

Originally posted by Xtraeme
Isaac would you do me a favor and diff the full version database Z.RND against the trial copy? I want to make sure the two aren't significantly different before trying to write a little CLI exporter.

Sure - with basic enough instructions I'm happy to do this sort of thing.

I'm not asking you to give me anything that would cause a copyright violation

Fine - I don't see any problem at all sharing the basic sort of detail you are asking form.

once you've done the diff (I recommend using Beyond Compare), just select "Session" from the toolbar and then click "Hex Compare Info." You should see something like,

xxxx same byte(s)
yyyy left orphan byte(s)
zzzz right orphan byte(s)

Just copy and paste that information here on to ATS.

Ok, I've run the relevant functions from Beyond Compare) and get the following:

108227 same byte(s)
12395 left orphan byte(s)
1916777 right orphan byte(s)
4782 difference byte(s)

I hope that helps you in some way.

I've had a glance at a comparison of the hex code in the demo .rnd and the full .rnd file - apart from the obvious huge different in the number of records and some apparent updating of some of the information in some of the record, there appears to be one consistent diffent: When Beyond Compare is set to highlight differences, every few lines (presumably every record) of the full .rnd file has 5 pairs of additional numbers which shows as gaps in the comparison version of the demo .rnd file - e.g. "03 00 E7 03 00".

For example, I posted in a post above the data seen using HxD in relation to an example mentioning Annaba (since a screen shot had been posted of that record). For comparison, the similar extract of the HxD information from the demo .rnd file is as follows:

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00006EA0 32 20 53 4D 4C 20 4E 4C 54 53 3A 5A 49 50 20 18 2 SML NLTS:ZIP .
00006EB0 20 55 4E 54 49 4C 20 4C 4F 53 54 2F 53 4B 59 00 UNTIL LOST/SKY.
00006EC0 88 F3 57 A0 07 02 07 06 00 FD 01 00 9F FA FB 19 ˆóW .....ý..Ÿúû.
00006ED0 76 43 4E 4E 00 47 00 21 80 00 00 00 00 42 4F 4E vCNN.G.!€....BON
00006EE0 45 20 3D 20 41 4E 4E 41 42 41 2C 41 4C 47 45 52 E = ANNABA,ALGER
00006EF0 49 41 3A 4C 55 4D 49 4E 4F 55 53 20 53 50 48 45 IA:LUMINOUS SPHE
00006F00 52 45 20 53 45 45 4E 3A 4F 42 53 2B 54 49 4D 45 RE SEEN:OBS+TIME
00006F10 20 55 4E 4B 3A 4E 46 44 00 00 00 00 00 00 00 00 UNK:NFD........
00006F20 00 00 00 00 00 00 00 00 00 00 00 02 54 45 A0 07 ............TE .
00006F30 02 07 06 48 FD 03 00 05 FB 9F 21 35 48 52 48 00 ...Hý...ûŸ!5HRH.
00006F40 43 00 01 80 00 00 00 00 54 48 41 4E 4E 2C 48 74 C..€....THANN,Ht
00006F50 20 52 48 49 4E 2C 46 52 3A 32 20 42 4C 55 49 53 RHIN,FR:2 BLUIS
00006F60 48 20 44 49 53 4B 53 20 53 45 45 4E 3A 4F 42 53 H DISKS SEEN:OBS

The differences highlighted by Beyond Compare are :

Shortly before the bit which is ASCII is BONE = ANNABA, you can see in the the full version the sequence in Hex "9D FF E7 03 00" (with the relevant characters in ASCII) with the numbers 19 and 76 either side - while the demo just jumps from 19 to 76.

Similarly, near the end of the extract there is a sequence in the the full version in Hex "9D FF E7 03 00" which is omitted from the demo

All the best,

Isaac

[edit on 23-6-2010 by IsaacKoi]

Originally posted by Arbitrageur

For a few dollars you can buy an old Windows 98 PC that will run DOS, I think there was one at the local salvation army store for $20.

[edit on 21-6-2010 by Arbitrageur]

I have a few W98 machines also. They still run fine, and I use them to control some of my house with x10 technology. The X10 S/W that I like to use does not work well on O/S's past W98. (If it ain't broke, why fix it!)

FYI: Having a DOS based machine running in DOS means that there is no Windows overhead and the response time of the machine is not hindered by Bill Gates overlystuffed graphical OS's.

If needed, I can check out the S/W on one of these machines, and report back.

I am sure the EXE can be decompiled into pseudo-source code. It may not look pretty, but the old DOS apps (CPU's) didn't have as many instructions available for them to use - which becomes a benefit for us now. (Need the SW next!)

-E2

Originally posted by nablator
I like this hex editor very much: XVI32.
www.chmaas.handshake.de...

Thanks for that link and, more importantly, the basic pointers you supplied on the contents of the hex and ascii information.

I'll download that hex editor so that I can look at the decimal values of some of the non-text characters, but it sounds like you (and one or two others) are well ahead of me in this game.

All the best,

Isaac

With thanks for the comments so far (particularly the references to Virtual PCs - something that was knew to me...), I am pleased to report that I now have the full version of Larry Hatch's database working flawlessly on a fairly modern computer using Windows Vista Business.

After numerous dead-ends (some arising from trying to use my main computer, which has Windows 7, which is already incompatible with the approach outlined below), the approach that worked was:

(1) Installing Microsoft's Virtual PC 2007 from www.microsoft.com... onto a laptop I own which still has Windows Vista Business on it.

(2) Adding Windows 95 within that virtual PC, using the instructions at this link:
www.youtube.com...

(3) Saving the old installed files from the floppy disk (which I've passed on from old machine to new machine several times, without having had a floppy drive for a few computer generations...) into an .ISO file using MiniDVDSoft's free ISO creation software at this link:
www.minidvdsoft.com...

(5) Running Window95 within Virtual PC 2007 then capturing the .ISO image of the relevant files, copying them into a directory on the virtual C: hard drive (into a new directory, "UFO").

(6) Running the u.exe file from that new UFO directory.


Anyone with Windows Vista Business should be able to duplicate the above steps (without having duplicate my exploration of dead-ends...) in a fairly short span of time.

Now that I have a fully working version of the database on a modern computer, I can start:

(1) Using it for the research that originally caused me to post this thread;

(2) Try taking forward a couple of the other approaches that were suggested in this thread, with a view to exporting the data into a format that does not require the use of a virtual computer (which has quite a few problems, particularly since Windows 7 is already incompatible with the above approach).

Using the full database application, I've tried the "export ASCII" option referred to by others above. Unfortunately, this only relates to a fairly modest number of the fields covered by the database.

I think the next step is to try using the "print" option to "print" to a text file. I should be able to try this in the next day or two.

So, thanks again for the help so far.

All the best,

Isaac

Originally posted by IsaacKoi
I think the next step is to try using the "print" option to "print" to a text file. I should be able to try this in the next day or two.

That's what I was going to suggest. You can use a PDF printer like this one:

sourceforge.net...

Which apparently works on Win 95. But beware: if the software exports only a part of the data into a text file is probable that it would only print those as well. You should try though just to be sure.

One small tip: you can drag files from your host OS (Windows Vista) into the desktop on your virtual PC (Win 95) to copy files between them, you don't really need to create an ISO file.

Glad you found virtual PCs, they are really useful!

Due to the copyright issues, I'm a more than a bit hesitant about emailing the relevant files to anyone

Preservation of history is more important than "copyright". This is the internet, the internet is about file sharing. Share those files because you never know, if one day you lose that data you can get it back from one of the friends you shared it with right away.

Do the right thing, sharing is caring. Preservation comes from multiple, unending copies.

Run an old DOS application on a new environment?
Sounds like a perfect utilization for VMware Player, VMware Workstation or VMware server.

I run old DOS applications under virtual machines all the time. All you need is VMware player (free download) and images of old DOS installation disks to get started.

PS: My guess is the database will turn out to be dBase-derived, or ISAM or bTreive-based.

[edit on 2010-6-28 by EnhancedInterrogator]

reply to post by NASA is ASAN


That depends on the opinion of the creator of the database, not on what we think.

Copyright is a real issue in a case like this.

Originally posted by ArMaP
That depends on the opinion of the creator of the database, not on what we think.

Copyright is a real issue in a case like this.

Wrong, no it is not. Preserving history is more important than worrying about the remote possibility that someone is going to sue you for $20 (which is so remote it's not even worth thinking about). You have a terrible attitude. Sharing is caring and it's what is going to make the world a better place. Or hell, just let the disk rot and disappear into the past, god forbid you let your conscious get in the way and break a copyright law.

It's akin to buying a movie and having a bunch of friends come over to watch it. Are you going to turn your friends away because of a copyright law?

Those laws are completely out-dated, please get with the times and realize the greater good comes from sharing. Or don't and disappear into the past; we already have most of the world sharing now anyways.

Or TS, just do this, share it with me (discreetly of course), then I will share it with everyone else. If the copyright holder wants to beef with anyone, he can come beef with me so I can laugh in his face for being a d-bag.

[edit on 28-6-2010 by NASA is ASAN]

Originally posted by NASA is ASAN
You have a terrible attitude. Sharing is caring and it's what is going to make the world a better place.

It's not a terrible attitude, sharing is only caring when you share your own work, sharing other people's work without their approval is not caring for what they think.

Originally posted by ArMaP

Originally posted by NASA is ASAN
You have a terrible attitude. Sharing is caring and it's what is going to make the world a better place.

It's not a terrible attitude, sharing is only caring when you share your own work, sharing other people's work without their approval is not caring for what they think.

Excellent, nitpick one comment and disregard the message of the rest of the post, you're a stand up guy.

Isn't this database from the 1980s and the creator of it dead? So that's it, you're worried about a 30 year old copyright owned by a deceased person? *facepalm*

I'm sure he cares if his work lives on or not, so help it live on. How old are you, by the way? You have a very outdated mindset, I'm going to guess 40+.

And one last thing, if the owner of the copyright is against this database living on forever, then that guy is extremely selfish and has no business owning the copyright. He doesn't deserve to own it if he's going to so @ss backwards in his thinking. The fact of the matter is, no one is going to buy it ever again, it's in the public domain now.

[edit on 28-6-2010 by NASA is ASAN]

Personally, I would dump the data file to UNIX, write an awk script to extract the text and dump it, analyse the data and normalise the main entities, add a few entities for extra details and recreate a new database schema that the data can be loaded into.

Forget 'products' per se, once you have the schema and the 'table' dumps and primary references the data can be loaded onto any database system you want...

reply to post by IsaacKoi


Hi isaack

There is another way that might be quicker than deciphering the non ASCII values, which is to write a tsr that inserts keypresses into the keyboard buffer to move between records and read the onscreen text out of video mem and dump to file.

O

[edit on 28/6/2010 by LightFantastic]

reply to post by LightFantastic


awk is entirely suited to this process, trust me, dump to text file and run it through a uitable awk script and you'll have the free text in no time. It is then just a case or normalising the data.

Originally posted by SugarCube
reply to post by LightFantastic

 

awk is entirely suited to this process, trust me, dump to text file and run it through a uitable awk script and you'll have the free text in no time. It is then just a case or normalising the data.

The problem isn't extracting the free text, it is that the rest of the data is stored differently.

If he just wanted the free text he could just open it in Word with the recover free text option.

[edit on 28/6/2010 by LightFantastic]

I performed a cursory internet search and found some references at the following location:

NICAP

This is a very simple list of sightings apparently taken from the *U* Db and it shows the kind of data entities available:

#14523: 1990/2/11 23:30 15 83:24:0W 41:54:40N 3333 NAM USA MCH 7 6
MONROE,MI:1 OBS/CAR:SCR-DLT CIRCLES FERMI NUCL.FCLTYROPS "SPARKS" ALL OVR
Ref# 160 MUFON UFO JOURNAL (by issue no.) Issue No. 268 TOWN &CITY

#15021: 1991/10/12 13:0 1 30:16:0E 51:17:0N 3331 EEU UKR KIE 7 7
CHERNOBYL,UKR:SCR FOUND/FOTO/PWR PLANT 1 DAY AFTER NEAR NUCLEAR INCIDENT..
Ref# 121 GOOD,Timothy: ALIEN UPDATE : Page No. 120 TOWN &CITY

#15465: 1993/3/31 2:20 1 4:46:40E 45:27:0N 3333 WEU FRN ISR 7 8
St.CLAIR-du-RHONE,FR:2 COPS:SLNT 45M CGR W/WINDOWS:HVRS:BLUE RAYS:>>SE/20M alt
Ref# 194 LUMIERES dans la NUIT.(Paris) Issue No. 318 TOWN &CITY

Using the references on the site as a guide, the data can be extracted. Maybe I missed the link in the thread, but is the *U* database available somewhere on the internet? Most of the links I tried to follow were dead-ends.

reply to post by SugarCube


You can download a demo - the location is detailed in this thread. You have to get it from Archive.org. The format isn't exactly the same between the demo data and the full version. Isaack has the full version on floppy.



[edit on 28/6/2010 by LightFantastic]

reply to post by LightFantastic


The point is, using an awk script the discrepancies in the data formats can be processed and taken care of, if you want to extract 18K odd record entities * n field entities using Word, feel welcome... :-)

This is a 'trivial' task, just dump it to text 'en clair' (warts and all) and make it available. You can have the data back in CSV, XML, whatever form within a couple of days depending on how much tea is consumed...