Huge UFO Database in DOS - Someone's labour of love, can it be saved?

originally posted by: javarome
a reply to: EvillerBob
...unless you want to allow storing multiple analysis of the same case, and even some "factual" data can differ/be challenged in different analysis. But, once again, this is just my two cents.

And that two cents is spot on the money, as with your other comments. That's exactly what I want to achieve. The main data will always be LH and that will be presented as priority, but I think there is merit in allowing other people to add their own case notes and assessments to incidents, specifically because they may have a different take on things.

Having said that, there is an additional layer for the future to develop - some kind of curating or oversight, rather than blindly accepting anything from anybody. That is a discussion for another day, however.

At the moment, the SQL database is providing great leverage in terms of searching against the data for things that don't fit, which is why I prioritised it before the analysis. I'm aiming to post up interesting results as I go along in case others might spot patterns that I'm missing.

For example, we're currently treating the location code as 4 bytes, even though a scan of the database confirms that the 4th bit is never used - and I seem to recall that the manual specifies a three character limit as well. So we have the already "unknown" bytes AND byte 4 of location that are essentially unused bytes, while the pinpoint data is being split over some rather random shared locations instead of using that free space. Room for expansion? Quite possibly. It just seems odd.

originally posted by: javarome
a reply to: EvillerBob
...some another allows to interpret the whole bitset differently (i.e. as volume+issue) instead of page number...

And I'll find the little blighter if it kills me

a reply to: javarome

Can you create an XML file?

Unknown 1 - the 4 bits that are shared with the month bits.

This is another puzzle. There are approx 75 records that have something set for these 4 bits. I say approx as I was manually counting and might have skipped a few, but say +/- 3 records.

Out of those records, the vast majority have the terrain set for 17 - in other words, they are incidents that are located in "Space". There are approx 13 records that do not conform to this, however, so despite the clear bias towards space events it seems like there must be more to it.

My initial thought was tagging. The original program has the ability for the user to "tag" records of particular interest. It is not clear whether this facility is persistent (ie if you restart the program the tag remains) but I have been expecting to possibly find some bit set for tagged records. The record pattern so far is what I would consider to be consistent with this behaviour (someone focusing on a specific type of event with a small handful of other events they find interesting)... but having checked with the source of the data it appears unlikely and I would find it hard to believe the author would distribute a copy of the database with his personal tags in place. Not impossible, of course.

So, one more mystery for us. I'll not post up all 75 records but I find it intriguing.

Isaac, if you have the opportunity and ability to check the bytes in a specific record, it would interesting to see a comparison of a record before and after tagging. Because I love making more work for you to do

Alternatively, if you could confirm whether the following records have a green background, which (according to the manual) would indicate that they are tagged:

#13, #40, #80, #100, #120

That would be most helpful indeed.

originally posted by: ArMaP
a reply to: javarome

Can you create an XML file?

If you don't mind it being a bit rough around the edges, I can provide an XML file for you. Just need to set up a convenient way to share it!

a reply to: ArMaP

I just added XML export. Not sure it is well formed, though.

The files are here.

originally posted by: javarome
a reply to: ArMaP

I just added XML export. Not sure it is well formed, though.

The files are here.

I don't think it is, at least Excel complained about the file's structure.

originally posted by: javarome
a reply to: ArMaP

I just added XML export. Not sure it is well formed, though.

The files are here.

If you are working on a Linux platform, you can use xmllint through the command line to verify it. This is what I do, as trying to validate a massive XMl file via the browser just kills my computer

Be aware that the text field in the record contains things that are recognised as control characters and will break XML.

If it helps, this is what I needed to add (in PHP) to make the XML file happy. Assuming ATS filtering doesn't play havoc with trying to post code. God forbid we ever try to end a sentence with window.

$textdata will hold the string taken from the record file, after replacing specific characters:

$textdata = preg_replace('/[x00-x09x0Bx0Cx0E-x1F]/', ' ', $dbrecord->[$key]['data']);

Now wrap up everything else in a UTF8-happy format, as the XML is UTF8 encoded:

$textdata = utf8_encode($textdata);

$textdata now hold that string that is written out to XML.

if you find a cleaner way of doing this, please share, I'm not too happy with having to strip data out but ran into continuous issues with malformed XML until I reached the above combination.

I was also wondering whether the control characters might be linked to the page/volume/etc issue that we have, but it seems illogical to have a character limit for the text field and then arbitrarily reduce this in specific circumstances (n order to allow space for those characters) without indicating it in the manual. I'm planning to revisit this after exhausting the other options, just in case.

Edited to add: Hey! ATS fixed the ridiculous window issue!

Edited again: Though I do notice that it's changing the curly brackets to square brackets. [$key] should be using curly brackets! ATS, you really need to add a code tag of some kind if you want to sandbox this stuff.

a reply to: EvillerBob

Thanks Bob. Yes, I guess this was the problem (at least). The lib I used escaped characters such as tag opening and closing but did not handle invisible characters such as 0x18 and 0x19. I replaced those two by spaces, as they seem to be the only weird chars present in descriptions.

ArMap (or whoever wants to test XML) could you try again with the latest file?
Thanks.

Invalid character, Line 6, Position 256.

Flying shield beams, year 322 (sic) (actually 332), Alexander the Great during the Siege of Tyre.

a reply to: javarome

Running it through xmllint returns approx 3200 parse errors, all of which involve similar errors. I'll post up the first few:

u.xml:6: parser error : PCDATA invalid Char value 25
ntinent>SIEGE/ALEXANDER the GREATFLYING SHIELD BEAMS
^
u.xml:75: parser error : PCDATA invalid Char value 25
andWestern EuropeVERTICAL CLOUD-CGR GOES
^
u.xml:130: parser error : PCDATA invalid Char value 24
inent>Western EuropeMANY OBSFBL BOUNCES
^
u.xml:130: parser error : PCDATA invalid Char value 25
ent>Western EuropeMANY OBSFBL BOUNCES +

The caret doesn't appear to be aligning properly on ATS which makes it a tiny bit pointless I suppose, but it's picking up lots of CAN and EM control codes. I think you'll need to strip out 0x8 (CAN) and 0x9 (EM) at least, possibly some others that aren't showing up in Notepad++ for me.

First, my apologies as the file you tested did not contain my fixes (was 3 days old).

I think you'll need to strip out 0x8 (CAN) and 0x9 (EM) at least.

Apparently x09 (tabulation) is not forbidden in XML. I updated the forbidden characters according to the XML Spec (but limited to the MS-DOS 0-FF range) and validated the first records of the new generated file (validating all records is too big) in an online validator.

Thanks for your feedback.

Does anyone have the raw files on a server somewhere? The links to post first generation files mentioned in posts are bad.

a reply to: roadgravel

You can find the raw files here.

a reply to: javarome

Thanks

originally posted by: javarome
First, my apologies as the file you tested did not contain my fixes (was 3 days old).

I think you'll need to strip out 0x8 (CAN) and 0x9 (EM) at least.

Apparently x09 (tabulation) is not forbidden in XML. I updated the forbidden characters according to the XML Spec (but limited to the MS-DOS 0-FF range) and validated the first records of the new generated file (validating all records is too big) in an online validator.

Thanks for your feedback.

That's my error, I mean 0x18 and 0x19, which are CAN and EM respectively - also known as ASCII 24 and 25, which is what the error messages had identified.

Apologies for the confusion!

Hi there,

Is the latest XMl file any better?

By the way, I just added a simple record find capability.

originally posted by: javarome
Hi there,

Is the latest XMl file any better?

By the way, I just added a simple record find capability.

The file dated 31 March 2017 passed xmllint without error, I'd say that you have cracked it in terms of xml compliance

a reply to: EvillerBob

Thank you Bob. Now I should focus on decoding those damn volume/issues/pages info.

originally posted by: javarome
Hi there,

Is the latest XMl file any better?

Yes, Excel 2007 imported the file without any problems.