Huge UFO Database in DOS - Someone's labour of love, can it be saved?

Originally posted by IsaacKoi
This looks VERY promising.

Following up on your post a few minutes ago, I quickly downloaded the first free hex editor that I found via Google, i.e. HxD:

mh-nexus.de...

HxD installed in seconds. Opening the .vce using that program just resulted in gibberish being displayed, but the .exe and RND files contain readable text. The data seems to be in the latter file (i.e. the .rnd file).

Xtraeme - Since your basic idea looks promising, could you expand on where to go next?

Just downloaded the demo and looking through the various files it's pretty clear the data is stored in the RND. It seems to follow a well-defined structure and (thankfully) isn't stream based. Running `file` shows it's a generic `data` file, nothing matching any magic entries like DB2. So it's custom. Later when I have more time I'll load up IDA Pro to see if I can find out where it reads in the file and get a sense of what each field represents.

From my 15 minutes looking over this I'd say it's likely all the data can be converted to a more open format without too much pain just using the demo, but it'd probably take at least a dedicated couple of weeks to create a proper exporter.

[edit on 21-6-2010 by Xtraeme]

Originally posted by Xtraeme
it's pretty clear the data is stored in the RND. It seems to follow a well-defined structure and (thankfully) isn't stream based.

So, this continues to look VERY promising.

From my 15 minutes looking over this I'd say it's likely all the data can be converted to a more open format without too much pain just using the demo, but it'd likely take at least a dedicated couple of weeks to create a proper exporter.

I've found the "IDA Pro" software you mentioned online and it looks like there are various free versions to try:
www.hex-rays.com...

I'll have a look myself later tonight, but I'm afraid my computer skills are probably not up to this sort of challenge...

All the best,

Isaac

I have tried downloading the demo however I cannot even run the extraction due to my windows version. Windows 7 x64. If you happen to get permission to distribute this application for the purposes of updating it then please send me a U2U with a link to the files already extracted. I really would only need the database storage file.

I can take the database storage file and run various processes on the data to figure out what storage scheme was used, and to then easily create a CSV file, which can then be imported into any current database application software, such as Access, MySQL, MSSQL, excel, etc.

Unfortunately without the consent of the copyright holder I am unable to touch anything from the application. I signed the ACM code of ethics and am bound to adhere to it.

Good luck in your efforts!

I love a technical challenge.

I will take a look.

A quick look at the Z.RND file shows that:
- it is not compressed or encrypted,
- each record is a fix size, 107 bytes, with short texts in ASCII and some binary information (type? date? country code?).
Offset 0 : CDh (file type check ?)
Offset 1 : 7 (record follows)
Record #1: 106 bytes
Offset 108 : 7 (record follows)
Record #2: 106 bytes
Offset 215 : 7 (record follows)
Record #3: 106 bytes
Offset 322 : 7 (record follows)
...
Record 1172: 106 bytes
EOF

That's very promising. 1172*107 bytes in the file.

The other file, VM.VCE has 6 bytes records, obviously, 12127*6 bytes.

12127 is not a multiple of 1172, a pity.

It may be possible to figure out the meaning of the binary data in each record from a few samples. If someone can get the software working and do a screen copy in DosBox or otherwise it would be nice. I had a fully working DosBox with Windows on a now defunct machine a few years ago. Now I'm not sure if I still have a copy somewhere of Windows 3.1 and DOS. Probably not.

[edit on 2010-6-21 by nablator]

Hope it helps:

The program has an option to export to ASCII, so I think if you can run it you can export the data.

I am running it on a virtual Windows 95 computer in Virtual PC. Virtual PC is free, but you need to install the operating system.

Edit: changed "cant run it" to "can run it".

[edit on 21/6/2010 by ArMaP]

For those of you who have this working, simply move to the last record and record the data onscreen. Now exit the program, change a single byte prior to each text string in the last record in the file by adding or subtracting one from the hex value and save. Start the software again, move to the last record and see what has changed.

Avoid changing 0x00s that occur after a text string. Repeat until you know what each byte is for. Two or more bytes might affect a single number. Once you know the function of each byte in the record it is a trivial task to write a program to extract the data to a more common format.

Hope this helps


[edit on 22/6/2010 by LightFantastic]

Originally posted by nablator
- each record is a fix size, 107 bytes, with short texts in ASCII and some binary information
...
It may be possible to figure out the meaning of the binary data in each record from a few samples.

So, it looks like we can do this, but it's going to take a bit of effort (and probably more technical skill than I have on my own).

I haven't played with technical problems for a couple of decades, but I'll start the ball rolling.

As I mentioned above, I've downloaded the hex editor HxD from THIS LINK and used it to open the .rnd file. I can read some text on the right and also see additional characters and hex values, but I'm not entirely sure I understand all that I'm seeing.

In particular, I'm not sure if the hex values (I think) that take up most of the screen are additional information or simply a different way of encoding/representing the ASCII characters on the right.

Since I'm far more comfortable using Microsoft Word to look at information, I exported the .rnd file to a richtext file using one of the options in HxD. I can then search for keywords etc using the Word functions that I'm used to using.

Originally posted by Radiobuzz

Ok, the screen shot at the above link supplied by Radiobuzz for one of the records has several elements.

(1) On the bottom in red is the case summary for record number 265, a sighting of a luminous sphere in Annaba, Algeria.

(2) On the top right are basic details, including the date, source, and rating of strangeness and credibility (in hex scales).

(3) On the top left is a matrix of three digit codes. The codes are explained in the downloadable manual. (In my copy, Appendix 2 gives details of each code in the "Attributes Matrix" - presumably the downloadable demo version includes the same information). These appear to toggle between on and off, so may presumably be represented in the data by a single digit. I'll avoid posting an explanation of all the individual codes set out in the manual, but the rows are as follows:

Row #1 ( top row ) Location of the OBSERVER.
Row #2: Miscellaneous details and features.
Row 3: Type of UFO / Craft.
Row 4 Aliens! Monsters! ( sorry, no religious figures. )
Row 5 Apparent UFO/Occupant activities.
Row 6: Places visited and things affected.
Row 7 Evidence and special effects:
Row 8 Miscellaneous details.

Anyway, searching the exported .rnd hex file (of the full version, not just the demo that others playing with this problem are probably using) for Annaba I found two hits, one of which relates to the screen shot at the link above.

Since I'm not sure where the record begins and then, I'll post a bit of the material either side of the text referring to Annaba.

The offset numbers will presumably be different for those of you working on the demo version of the .rnd file, since that demo contains far fewer records.

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

0003E1E0 4D 4C 20 4E 4C 54 53 3A 5A 49 50 20 18 20 55 4E ML NLTS:ZIP . UN

0003E1F0 54 49 4C 20 4C 4F 53 54 2F 53 4B 59 00 88 F3 47 TIL LOST/SKY.ˆóG

0003E200 A0 07 02 07 06 00 FD 01 00 9F FA FB 19 9D FF E7 .....ý..Ÿúû..ÿç

0003E210 03 00 76 43 4E 4E 00 47 00 21 80 00 00 00 00 42 ..vCNN.G.!€....B

0003E220 4F 4E 45 20 3D 20 41 4E 4E 41 42 41 2C 41 4C 47 ONE = ANNABA,ALG

0003E230 45 52 49 41 3A 4C 55 4D 49 4E 4F 55 53 20 53 50 ERIA:LUMINOUS SP

0003E240 48 45 52 45 20 53 45 45 4E 3A 4F 42 53 2B 54 49 HERE SEEN:OBS+TI

0003E250 4D 45 20 55 4E 4B 3A 4E 46 44 00 00 00 00 00 00 ME UNK:NFD......

0003E260 00 00 00 00 00 00 00 00 00 00 00 00 00 02 54 45 ..............TE

0003E270 A0 07 02 07 06 48 FD 03 00 05 FB 9F 21 9D FF E7 ....Hý...ûŸ!.ÿç

0003E280 03 00 35 48 52 48 00 43 00 01 80 00 00 00 00 54 ..5HRH.C..€....T

0003E290 48 41 4E 4E 2C 48 74 20 52 48 49 4E 2C 46 52 3A HANN,Ht RHIN,FR:

0003E2A0 32 20 42 4C 55 49 53 48 20 44 49 53 4B 53 20 53 2 BLUISH DISKS S

The above screen relates to a sighting at Quantico involving an object emitting a red light.

Searching for the text "emits red" in the exported hex file finds:

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

0005C9C0 45 52 53 3A 43 49 41 20 52 50 54 00 00 89 72 6A ERS:CIA RPT..‰rj

0005C9D0 A1 07 12 0C 15 7F FF AA 00 5D 36 18 1B 09 00 00 ¡.....ÿª.]6.....

0005C9E0 00 00 02 56 52 47 00 A3 00 09 80 00 80 00 43 51 ...VRG.£..€.€.CQ

0005C9F0 55 41 4E 54 49 43 4F 2C 56 41 3A 42 42 4B 23 31 UANTICO,VA:BBK#1

0005CA00 36 35 34 3A 4D 41 52 49 4E 45 53 3A 55 4E 4B 2E 654:MARINES:UNK.

0005CA10 52 4F 55 4E 44 20 4F 42 4A 20 4C 41 4E 44 53 3A ROUND OBJ LANDS:

0005CA20 45 4D 49 54 53 20 52 45 44 20 4C 49 54 45 3A 42 EMITS RED LITE:B

0005CA30 41 43 4B 20 30 31 4A 41 4E 35 34 00 00 D2 1C 8A ACK 01JAN54..Ò.Š

0005CA40 A1 07 10 0C 38 30 FF 05 00 3B 52 10 17 70 00 E7 ¡...80ÿ..;R..p.ç

0005CA50 03 00 02 43 4C 46 00 B1 00 01 80 00 00 00 03 6F ...CLF.±..€....o

0005CA60 76 72 20 45 4C 20 43 41 4A 4F 4E 2C 43 41 3A 32 vr EL CAJON,CA:2


All the best,

Isaac


[edit on 22-6-2010 by IsaacKoi]

Originally posted by IsaacKoi
or simply a different way of encoding/representing the ASCII characters on the right.

Exactly. The problem with editing the file using Word is that it will change change the number of bits on the file so most probably the file won't work with the original software anymore. That's why you have to use an hexadecimal editor if you want to edit a file, so you modify without adding or taking anything important out.

Originally posted by Radiobuzz
The problem with editing the file using Word is that it will change change the number of bits on the file so most probably the file won't work with the original software anymore.

But I'm not trying to edit the file - I'm simply using Word since I'm at home using its search functions and display options.

I'm only looking at the data exported from the hex editor to see if I can match up the text and other characters with the data for the sample records.

However, I'm not sure what to look at - in relation to hex data I've cut and pasted above in relation to the sample records:

(1) Is that all I need to look at or do I need to look at, for example, some sort of display of the data in binary form rather than hex for some or all the relevant record?

(2) Do I only need to look at the text and characters in the final column on the right? I'm not sure if the data in all the other columns to the left of that final column merely display the same information in a different format or whether they contain additional information (and, if so, which parts relate to which record)?

If anyone has pointers or links to a relevant resource, I'm happy to spend more of my own time on this - but I'm probably struggling with some aspects that may well be blindingly obvious to some of the more technically minded members of ATS.

All the best,

Isaac

Originally posted by IsaacKoi
(1) Is that all I need to look at or do I need to look at, for example, some sort of display of the data in binary form rather than hex for some or all the relevant record?

Hex is what a hex viewer/editor displays. Conversions to decimal are needed too. You can do them in Windows' calc program.

(2) Do I only need to look at the text and characters in the final column on the right? I'm not sure if the data in all the other columns to the left of that final column merely display the same information in a different format or whether they contain additional information (and, if so, which parts relate to which record)?
ASCII text is only useful for the readable description of each case. The rest needs to be figured out in terms of numbers and codes. What you see in hexadecimal is the same information as the ASCII text. 32 decimal = 20 hex = space ASCII for instance.

If anyone has pointers or links to a relevant resource, I'm happy to spend more of my own time on this - but I'm probably struggling with some aspects that may well be blindingly obvious to some of the more technically minded members of ATS.

Yes. There are a few easily identifiable fields that are identical (in decimal) to the plain text version in the screen capture, such as duration. Others are more mysterious. This extremely compact way to encode data is difficult to decipher. I don't understand dates yet. Lon/Lat coordinates are encoded in two bytes, just divide by 20 the number of seconds. Other fields are less obvious because they are codes, there is one for each country, etc. They can be identified by comparison between records... a large number of records, as there are many different codes.

All the best,

Nab

Here's a site for you to decode hex data into some other useful forms:

home2.paulschou.net...

I've copied the data between the records expecting to find numbers that would correspond to the record's numbers but it doesn't appear to be the reason. In fact I have no idea what they are for. Someone with disassembly knowledge should look them up.

I'm running Windows XP Professional 3, and I can see it just about fine. Of course I made some additions to the original package (eg: it runs office 97 - prefer it over the newer stuff!)

Should I get the OK, i recon I could have a bloody good go and save some of it

I like this hex editor very much: XVI32.
www.chmaas.handshake.de...

Once I did scripts with it to 1337 haxoring on DLL files. Aaaah... the joys of disassembly. Also did many binary formats exploration (like JPEG) and reverse engineering (CAD and GIS formats).

There are many others hex editors just as good maybe but this one displays decimal/hexadecimal value on the status bar (very useful) and has a data inspector for number conversions. Many other useful features : copy / paste in hex, block operations (insert, delete ...)

Instead of slowly and painfully figuring out the meaning of each byte in the file, why not capture screens (as text, if possible in DOSBOX?). If it is only a matter of typing a key thousands of times and capturing the screen to get all record, it can be automated.

This is only needed if there is no way to export the database as readable text. ArMaP wrote something about that if I remember correctly.

[edit on 2010-6-22 by nablator]

reply to post by nablator


That's the one I use.

I don't know if it's a problem on my side or if the demo does not allows changes to the database, can anyone confirm it, please?

Thanks in advance.

reply to post by IsaacKoi


Isaac would you do me a favor and diff the full version database Z.RND against the trial copy? I want to make sure the two aren't significantly different before trying to write a little CLI exporter.

I'm not asking you to give me anything that would cause a copyright violation (please don't post a patch-tarball), once you've done the diff (I recommend using Beyond Compare), just select "Session" from the toolbar and then click "Hex Compare Info." You should see something like,

xxxx same byte(s)
yyyy left orphan byte(s)
zzzz right orphan byte(s)

Just copy and paste that information here on to ATS.

Thanks!

reply to post by ArMaP


I could do it just fine. What I can't do is export to ASCII from within the software. The print function is taken out of the demo version, unless you're talking about some other option I haven't seen.

reply to post by Radiobuzz


OK, thanks.

I thought that I had edited a record on Monday but I wasn't sure, I must be doing something wrong now.