Canadian disclosure: “UFO Found” and other documents/photos

Oh well, it's not saving as many images as I expected, which means there's another naming convention I should use. So, this ain't happening tonight...

Jeff

Originally posted by redzareptile
reply to post by Echtelion

 

Demons and evil underworld spirts desguised as light beings from distant planets?

Extrapolate further on your theory please.

A bit late to counter-reply, but just in case...

There's a very good case, that's been already discussed at least a bit in the "UFOs and paranormal" section of ATS, that the aliens aren't really, and may in fact turn out to be either extradimensional beings, or simply evolved humans from the future. Both assumptions can be easily explained through quantum physics.

In either cases, this would explain why these beings could be witnessed all through human history, with little technological evolution from one era where sightings have occurred, to another. As an example, the reported "signs" and enlightened, moving "clouds" in the Bible correspond exactly to some modern-era descriptions of UFOs, even though both accounts have thousands of years of distance from each other.

Angels and demons could very well be these beings, differing mostly on their intents towards other helping or enslaving (or even destroying) mankind, and this is much substantial with the ancient legends of the Annunaki (in Sumerian texts), the Elohim and the Nephilim (in ancient Semitic texts), superior races of beings who were either servants of a higher cosmic authority, or antagonists of it. As a matter of fact, I don't see how these beings could be anything else than this. One could say that these were mere fictions created by the Church to scare the masses, but then WHY do these fictions have similar traits from one culture to another? Why there's always stuff about winged serpents, flying wheels and lights in the skies, among other things?

It makes sense that if these UFOs are technology used by these supernatural races, still flying in our skies, the most powerful human authorities on Earth might rather not talk too much about their presence -and potential influence in human affairs- than disclosing it from the start, for their own power's sake.

There is major gain in secrecy, both in terms of protection and capitalization of power, although that's how capitalist politicians of all kinds see things...

I got the feeling that only once the State has fallen in the Western world, among its rubble will we discover its biggest secrets.

Originally posted by idealord
Now spidering the Canadian site and downloading all of the images. I'm capturing all the data I can think of, every record and all data on each of the pages.

...
spreadsheets.google.com...

Hi Jeff,

As with the spreadsheet you created which we have been discussing on the Massive UFO disclosure in USA : A challenge for ATS, this looks very promising. (Incidentally this is the approach I briefly mentioned in a response to Kandinsky above, but didn't have time earlier today to explain what you have been doing on that other thread).

The main query I have in relation to your draft spreadsheet in relation to the Canadian documents is whether it is possible to (automatically) incorporate the reference numbers beginning with an E that are used in relation to each individual image, e.g. at the URL below:
data2.collectionscanada.gc.ca...

As you may have seen in this thread (particularly in Section 2 above), the images are stored in blocks of numbers beginning with an e (such as the example blocks below) and I (and presumably some others) have downloaded the vast bulk of the ??9,500?? pages of images and saved them with file names that use those E numbers.

For example, the URLs for the images end with the numbers below:

* from /e/e110/e002744278.jpg to /e/e110/e002749999.jpg

* from /e/e111/e002750306.jpg to /e/e111/e002751233.jpg

I can't see these E numbers in your spreadsheet.

Unless I'm missing something, I think we will either:

(1) need to add those numbers (ideally using some automatic method...) to your spreadsheet; or
(2) redownload the images and store them with other reference numbers which DO appear in your draft spreadsheet.

I'd prefer option (1) if possible...

What do you think Jeff?

All the best,

Isaac

Great minds... that was my going to be today's plan, Issac! I don't really understand how this database works (I suspect it's keeping multiple data across reference #'s because it's insinuating paging) and when I run my current spider it doesn't save as many images as needed.

So I was going to add the e-url data, like you suggest to the database and to the file name. Should get that rolling today. That'll also let the good folks who have been crawling the image system to confirm and correlate my data.

Jeff

So far I've grabbed a total of 8,766 .JPG's from these blocks of URL's

from /e/e110/e002744278.jpg to /e/e110/e002749999.jpg

from /e/e111/e002750000.jpg to /e/e111/e002750209.jpg :Posted by freelance_zenarchist

from /e/e111/e002750306.jpg to /e/e111/e002751233.jpg

from /e112/e002799600.jpg to /e112/e002799811.jpg :Posted by freelance_zenarchist

from /e/e120/e002996914.jpg to /e/e120/e002996931.jpg

from /e/e120/e002997366.jpg to /e/e120/e002997414.jpg :Posted by ArMaP

from /e/e120/e002997420.jpg to /e/e120/e002997675.jpg :Posted by freelance_zenarchist

from /e/e120/e002999664.jpg to /e/e120/e002999999.jpg

from /e/e121/e003000000.jpg to /e/e121/e003001631.jpg

Has anyone found more than this?



Also I just came across these, they might come in handy, they have short descriptions typed out, and in some cases the whole document is retyped (could be good for the ones that are hard to read) -

ufo-joe.tripod.com...

ufo-joe.tripod.com...

ufo-joe.tripod.com...

Originally posted by freelance_zenarchist
Has anyone found more than this?

Not me, but the method Jeff has been developing in the past day or two (which has mostly been discussed on the USA thread, but mentioned above as well) may well give us a complete list - with indexing information - so we can then quickly identify any missing blocks of URLs and add download those images.

So, I'd wait a day or two and see if Jeff can get his method working before carrying on our more time consuming search.

I'm not expecting a whole lot more images (may be a few hundred), but the indexing information should be useful. (I have ideas for cross-referencing that indexing information fairly quickly with the official file numbers, which are the numbers used in books and in a few outlines online - but let's leave that until Jeff's spreadsheet is produced...).

All the best,

Isaac

I'm about half way there, FWIW. 4331 records processed. Strangely, I only have 667 images. Now, either I've got bugs, or there are a lot of dupes.

Latest data:

parnasse.com...

The file-naming convention is set up to avoid dupes. I'm combining the initial recordID with the subsequent recordid and then adding the e-file name.

Something like this:

canadian-ufo-13019-4394-10591-S-940-5-e111-e002750324

I'm going to add some code to see if it's dupes that are the problem - that is - my file naming convention is causing different files to be named the same and thus to be over-written.

I'm going to stop the run and you guys can take a look at the data it's output already, maybe correlate it with your block-based approach?

Yikes, figured it out...

The Canadian search results themselves cite the same image over and over. I sorted those 4500 records and take a look at Column G (sorting on Column G).

Column G shows lots of redundancies...

Heh.

Jeff

Originally posted by idealord
I'm about half way there, FWIW. 4331 records processed. Strangely, I only have 667 images. Now, either I've got bugs, or there are a lot of dupes.

Latest data:

parnasse.com...

...
I'm going to stop the run and you guys can take a look at the data it's output already, maybe correlate it with your block-based approach?

Hi Jeff,

By removing part of the text in Column G (i.e. find the first part of the URL and replace with nothing) of your spreadsheet, I've sorted by E numbers (the number that appears at the end of the URL for the jpg images).

I think your number of images (667) is about right for the number of different E numbers in your sample output. Some E numbers are repeated dozens of times.

Once the duplication has been eliminated and your method run for the rest of the records, it should only take a few minutes to identify the missing blocks of URLs by repeating this type of sort and comparison with known block numbers.

All the best,

Isaac

Yeah, that's pretty much what I was thinking. I'll go back to the run, starting where I left off. We really don't know where these search results leave off I'm guessing? The 9500 records - that's 9500 images? So maybe to get metadata on all of that, I need to run the search results to 20000 records or more?

Running again, at least it wasn't my bug...

Jeff

Originally posted by idealord
We really don't know where these search results leave off I'm guessing? The 9500 records - that's 9500 images?

Hi Jeff,

We don't know.

I'd found a press release which referred to "9,500 files" - which I've presumed actually is "9,500 pages", since my limited efforts had found about 8,000 pages and subsequent efforts by others have found several hundred more pages.

I would not take the figure 9,500 as a precise figure and it may be that we find that we have ALL, or at least nearly all, the images already. I'm certainly not expecting to find thousands and thousands of more pages.

Of course, even if the spreadsheet does not identify any more pages, it will still be useful for indexing and organisation purposes.

So maybe to get metadata on all of that, I need to run the search results to 20000 records or more?

I'd guess more, since some of the E numbers appeared dozens of times in the spreadsheet generated using your method. But since duplicate rows can be easily eliminated, I don't think this is a problem (other than the time it takes up on your computer...).

All the best,

Isaac

Originally posted by x5deadlyvenoms
Isaac, I've taken a look at this documentation from the Canadians and it's incredibly interesting. It prompted me to visit your website, and I must say there is quite the wealth of information.

Hi x5deadlyvenoms,

Many thanks for the kind words.

I saw your website is going to go into full-gear this December,

Well, probably... I've pushed it back repeatedly over the last year or two. It's not the highest priority in my life.

I wanted to know what kind of benefits users would have access to if they register. Are you looking for a discussion outlet with the registration? a Forum? I know you post on ATS a great deal and was wondering what you were planning. Will there be opportunities for users to upload information as well?

I'm certainly not planning on competing with ATS! I haven't got the time, patience or technical ability to create and moderate a forum.

I'm happy participating here, and adding some resources to my own website. This is not to say that I think ATS is perfect. A lot of dross gets posted here. Also, I don't agree with every single decision the moderators and owners make (I recently had one of my own threads 404ed and a request that it be reinstated, and locked if necessary but with a note from me giving my reasons for wanting the thread to exist - ignored) but I doubt I could do a better job even if I had the time (which I don't...).

I did have in mind allowing users (or certain users) the option to add to, or edit, some or all pages on my website. I'm not too sure about that and how it would work in practice. Again, I don't have much time to moderate edits - but I can see the benefit of allowing some people to add to my framework.

Frankly, I have other interests (plus a family, not to mention a job) which mean that at the moment I'm not sure how much more I'll bother with my own website.

All the best,

Isaac

Originally posted by idealord
We really don't know where these search results leave off I'm guessing?

I stopped looking into this because I don't trust the search results that much.

The 9500 records - that's 9500 images?

Usually, what archivists talk about is documents, they never say how may pages the document has, so in this case, 9500 records can represent a completely different number of images.

PS: archivists use linear metres as a measurement to the size of an archive, but we don't have that either on this case, if we had we could use a simple rule to turn linear metres into pages, although that wouldn't tell us if both pages had information or if the information was only on one page.

First time I've heard about this. Thanks for bringing it to our attention. It certainly looks legitimate as far as I am concerned.

reply to post by TheComte


So, I just want to know, "are they going to eat us? and are they from space or time traverlers?", its probably

going to be scary or not.

All I know is there isn't any room for them here. So, it will be a take over job.

Hi Isaac (& helper(s))

How many gigs of images/ data do you all have so far?

Speaking from my spidering, not even 100 MB yet.

Issac, I'm thinking of running a spider that will numerically attempt to slurp down every image. If they don't show up in the search results data, at least we've got it.

Should be super easy to write, it's just a counter and a getter....

Jeff

Originally posted by Dalbeck
Hi Isaac (& helper(s))

How many gigs of images/ data do you all have so far?

I'll check when I'm at home.

I had just over 8,000 images of the Canadian documents before starting this thread and they did not take up that much room on my computer. (I didn't even have to move them to an external drive). Also, the number of pages is fairly small compared to, say, my collection of over 1,000 UFO books.

All the best,

Isaac

Originally posted by idealord
Issac, I'm thinking of running a spider that will numerically attempt to slurp down every image. If they don't show up in the search results data, at least we've got it.

Hi Jeff,

I think you may have problems with counting approach.

It's been a week or two since I put a bit of effort into finding the images, but from memory the collection at, say, E121 would shift at a certain point from UFO documents to, say, some Canadian government department records from 1910... Unless we know the start and end points for the E121 etc parts of the collection, you may end up a LOT of irrelevant images.

Since my guess is that we are now probably only missing a few hundred pages (at most), I'd draw a line under our efforts once you've run your spreadsheet mechanism to get as many URLs as possible - even with some duplication of numbers (and I'd be happy to run that script from my computer, if you give me detailed enough instructions...) - use that spreadsheet to identify any further blocks that we can (which should only take a few minutes) and then I'm happy to email the Canadian archives and invite them to confirm that we have all the relevant images or provide any further URLs. I think that email can show that a number of people have put a considerable amount of effort into finding relevant images and indexing information, so I'd hope we get a bit of sympathy...

So, other than running your spreadsheet generator for a bit longer (or allowing me to do so), I wouldn't spend much longer on trying on obtaining any fairly small number of further Canadian documents.

If you would like to put your talents and techniques to better use, I have a couple of ideas in relation to other govnerment collections of UFO documents for which your spreadsheet generating technique should (I think) be fairly easy to adopt... But let's get the Canadian spreadsheet finished first!

All the best,

Isaac

Sounds good... I still haven't seen any saucer pics out of the few images I gleaned from my approach!

How many runs would you suggest. I'm up to 9100 now and was thinking of going up to 50000. But I was just poking around and I see no difference in the search results for this URL:

www.collectionscanada.gc.ca...

Starting with 20000

And this URL:

www.collectionscanada.gc.ca...

I'd be happy to send you my scripts if you wanted to verify my approach with another professional, but there's no way you could run these Perl scripts without installing Cygwin and all of the supporting libraries. Actually, I could send you those... but it's very much like going back into the old MSDOS days except with these great libraries that make writing Spiders a breeze.

Curious about your other ideas about governmental databases. FYI, a few systems are not very happy with people that hit their databases with spiders or bots. Mine aren't aggressive, but it freaks people out and I don't want to get arrested for 'Hacking' when we're just automating our searches!

Jeff