Massive UFO disclosure in USA : A challenge for ATS

just curious....there's something that has been bugging me for quite some time.....did anyone stop for a minute to think that just because some idiot at the FBI stamped "BOGUS" all over the MAJIC documents, that they really are fake? and the so-called "experts" out there that claim they're fake, might have been lying to us, just to throw us off? think about it.

Originally posted by Hilltaker
this:

www.httrack.com...

Will sort all your problems.

Thanks Hilltaker, but I don't see how.

I'd love to learn I'm missing something simple - but someone will have to spell it out for me.

A few people have confidently stated in this thread that I can simply use tools such as those found at www.surfoffline.com... or www.httrack.com... , but (as I have said a few times now)

(1) the problem with tools that allow you to download a webpage or entire website is that only a small part of the ABSOLUTELY HUGE footnote.com relates to UFOs.

(2) while some of the relevant tools allow you to specify a part of a website to be downloaded, the UFO images are not (so far as I've been able to tell so far...) in a directory such as footnote.com/UFO/*.*

While I am happy to buy one or more hard drives to download these files, downloading the ENTIRE huge footnote.com website (as opposed to the capacity needed to download about 130,000 UFO documents which forms a small part of that website) may be FAR beyond a realistic goal.

At the moment, I don't see how the tools at surfoffline.com and httrack.com could be used to download these documents in an efficient manner.

Again, let me stress : I'd LOVE to be wrong, but I need a bit of help from someone that can spell out how these tools can be used in the context I've outlined above.

All the best,

Isaac

reply to post by IsaacKoi


I used the tool multiple times and it is rather easy. If you open up the tool the guide will show you the page indexed or should do so. There you simply browse what you need? Thanks for trying.

PS: I am at work and have no change to look at this until I get internet myself around 240811.

Thanks,

Originally posted by IsaacKoi

Originally posted by zorgon
Did you ever do that thread on the Canadian files? I believe you said you had a quick d/l for that one?

I should finish that thread in the next day or two.

Er, I'm going to have to revise that estimate upwards by a few days.

Having downloaded most of Canadian documents (if only the USA documents were as easy...) and looked at quite a few of them, the thread I was drafting has grown quite a bit in the last couple of days and I need to look into a few things a bit more (particularly documents and photos relating to the recovery in Canada in the late 1960s of an item that, according to the Canadian documents, "likely formed part of a vehicle that travelled in outer space"...).

All the best,

Isaac

Hey Isaac,

The material isn't easily accessible to normal spidering tools because everything's obscured by the flash app. Even grabbing the flash object and looking through the binary doesn't help much because it appears they were smart enough to keep all the path information server-side.

The only good way I can see to automate downloading the content is to write a screen-scraping macro. Not exactly ideal, but it would work. Basically you'd have to use something like AutoIt ( www.autoitscript.com... ) to:

1. mouse over or tab to the "Download" widget and send a KEYDOWN message (starting from www.footnote.com...)
   
2. mouse-over "Entire Image" and send a WM_LBUTTONDOWN
   
3. attempt to write the current file in a given directory by clicking 'save'
   
   1. if this fails (check the task list a second or two later to see if "Select location for download..." is still active), then
      
   2. tab to the "Confirm Save As" dialog and send a KEYDOWN message for "No."
      
   3. create a new directory starting from 00000 and increment the value. After that in the "Select location for download" dialog add the new URI in the "File name" field as: ..\0000X\filename.jpg (or just type in the directory and send two keydowns for "Save" -- this will regenerate the filename)
   
   
4. wait some x seconds for the download to complete (or simply monitor the filesize of the new item and wait for it to stop incrementing).
   
5. mouse-over to "Next Image" and send a WM_LBUTTONDOWN
   
6. And repeat ...

Implementing a script that follows these rules should let you download all the content. Though it'll take approximately 10.5 (~7 sec/doc) to 14.9 days (~10 sec/doc) of dedicated computer time to grab it all (~7 sec/doc * 129,000 doc = 903,000 sec*doc/doc; 903,00 sec / 60 sec/min = 15,050 mins; 15,050 mins / 60 mins/hr = 250.83 hrs; 250.83 hrs / 24 hrs/day = 10.45 days)

Cheers,
-Xt

Originally posted by zorgon

Originally posted by dilly1
And so what if it says three small bodies. You all want to believe aliens have visited earth(recently).

Did you even read the posts and Isaac's thread showing that document was a HOAX?

Perhaps you will find this interesting.
IsaacKoi said in his thread that that FBI document is not a fake but a genuine document but that it was a part of a hoax because he is convinced that the Silas Newton affair was exposed as a hoax DECADES ago.

Originally posted by IsaacKoi

Silas Newton affair exposed as a hoax DECADES ago

The hard part in understanding the Guy Hottel memo is linking it to the Silas Newton affair. This was done in Section D. I’ll just give a very brief overview of the Silas Newton affair.

[atsimg]http://files.abovetopsecret.com/images/member/a1e99e332b0e.jpg[/atsimg]

Silas Newton gave a lecture at the University of Denver on 8 March 1950. That lecture became the foundation of a very popular UFO book by Frank Scully, “Behind the Flying Saucers”, published later in 1950 (available online here).

[atsimg]http://files.abovetopsecret.com/images/member/9d1d66015e8b.jpg[/atsimg]

In his book, Scully gave details of a story told to him by Silas Newton and a scientist friend (“Dr Gee”) about several crashed UFOs. The first of these saucers allegedly went down near Aztec, New Mexico.

Silas Newton was introduced at the relevant lecture at the University of Denver by a Mr George Koehler, i.e. the “Coulter” in the article investigated by the Air Force and which formed the basis of Hottel’s memo. Frank Scully told Cahn that the “Dr Gee” in Scully’s book was in fact Leo Gebauer – i.e. the individual Koehler named as Koehler’s source when interviewed by the Air Force. Thus, the sources of the Hottel memo and Scully’s book are clearly intimately connected.

For the whole post you can look here.

reply to post by IsaacKoi


The coincidence is that I did got that book from Frank Scully in my possession back then in a second hand bookstore and asked later on John Lear in 2008 if he knew about that book and if had read it and what his opinion about it was.

Aug 16, 2008, 9:00am, spacevisitor wrote:

Hello John.

Sorry if I ask this question in the wrong thread but there is no other then this one so far from you.
I buy some years back in a second hand bookstore the book “Behind the flying Saucers” from Frank Scully.
I assume you know it and perhaps you read it once.
If that's so, my question to you is.
I read some chapters in it and thought that there is information in it that really could be valid in my opinion.
But it is the opinion of many others [as usual] that it is only good for burning on a stake.
I am very interested in your opinion, so will and could you tell me what your impression of that book is?

Here is what John said.

Frank's book was accurate in every way.

After Frank's death his wife lived in Palm Springs. Of course she was bitter about the whole thing as she knew it was all true. I sent her a message through mutual friends not to take it personally.

Thanks.

And because of what IsaacKoi said there I become very curious if those accusations could/were indeed to be true and I ask him this question.

Apr 14, 2011, 2:23pm, spacevisitor wrote:

Hi John,

I am shore you will raise your eyebrows now, because this is as you can see an older post.
But I have again a question I want to ask you about it.

It has all to do with the chapter in that book called, “The Mystery of the University of Denver”, and the next chapter was called “What the Scientist Said”.

In this link is something posted about that all and about Frank Scully which I find quite surprising because I cannot believe that it is true, so, my question to you is, are you willing to take the time to read it and let me know what your opinion is about it.

reply to post by IsaacKoi

 

I hope the link works out, if not let me know.

Thanks in advance.

This is what John said.

I read all the letters. Whoever made the post trashing the Scully story and everything about it is less than knowledgable.

I talked to Bill Steinman last night and we actually had discussed the whole story.

The guy who was writing the post was trying to confirm that the whole story was a hoax and had completely bought into the disinfo that the incident was a indeed a hoax.

So, what is the real truth behind that all?
For what I know now about that subject, I go for the time being for Johns view.

Sorry IsaacKoi.

well...that puts a new spin on things....i'm starting to think that a lot of documents that have been dismissed as fake, may very well be the real thing

Originally posted by moonweed
well...that puts a new spin on things....i'm starting to think that a lot of documents that have been dismissed as fake, may very well be the real thing

This is the primary goal of debunkers Confuddle the audience

Originally posted by spacevisitor
So, what is the real truth behind that all?
For what I know now about that subject, I go for the time being for Johns view.

According to the FBI files...

Silas M. Newton — filed under: Unexplained Phenomenon Silas Newton (1887-1972) was a wealthy oil producer and con-man who claimed that he had a gadget that could detect minerals and oil. He was cited as an authority in Frank Scully’s book Behind the Flying Saucers, a work that claimed to report on several UFO crashes in the area of New Mexico. In 1950, Newton said that a flying saucer crashed on land he leased in the Mojave Desert; however, he revised his claim in 1952, saying he never saw a flying saucer but had only repeated comments he heard from others. These files detail the FBI’s investigations into Newton’s fraudulent activities between 1951 and 1970.

vault.fbi.gov...

So... my question is...

is Dana Katherine Scully related to Frank Scully?



ETA:

In case anyone is interested

FBI files on Albert Einstein

vault.fbi.gov...

Originally posted by moonweed
just curious....there's something that has been bugging me for quite some time.....did anyone stop for a minute to think that just because some idiot at the FBI stamped "BOGUS" all over the MAJIC documents, that they really are fake? and the so-called "experts" out there that claim they're fake, might have been lying to us, just to throw us off? think about it.

Well DUH... I thought EVERYONE knew that the FBI lies to us

Ya want proof? Here ya go

According to the FBI...

Extra-Sensory Perception — filed under: Unexplained Phenomenon ESP is considered a perception of information about events beyond what may be discerned through the five physical senses or deduced from past experience or knowledge. This release consists of cross references to ESP found in FBI files from 1957 to 1960. Several of the documents concern William Foos, a proponent of ESP. Others concern claims that ESP could be used in espionage investigations. The FBI found no scientific support for this or other claims and did not pursue the matters raised in these references.

vault.fbi.gov...

But at the same time the CIA ran "Project Stargate" from 1968 to 1995 when Congress ruled it wasn't nice to snoop in peoples brains (after that they just went to private corporations that were not under Congressional scrutiny.. )

reply to post by Xtraeme

Originally posted by Xtraeme
The only good way I can see to automate downloading the content is to write a screen-scraping macro. Not exactly ideal, but it would work. Basically you'd have to use something like AutoIt ( www.autoitscript.com... )

Well, now we are cooking...

This looks VERY promising. I think it is by far the best suggestion in this thread. MANY thanks.

I've never seen that sort of tool before, but I can see how it may work in this context.

Mmm.

Time to get done to business...

All the best,

Isaac

reply to post by spacevisitor

Originally posted by spacevisitor
So, what is the real truth behind that all?
For what I know now about that subject, I go for the time being for Johns view.

Sorry IsaacKoi.

That's fine, although it would be nice if John had actually presented some evidence in support of his claims...

reply to post by IsaacKoi

Happy to help. I spent a couple of minutes seeing if I could fish out the details from the breadcrumb trail.

[atsimg]http://files.abovetopsecret.com/images/member/08cbdee948a4.png[/atsimg]

I've only checked through a couple of the _javascript files, but it appears you can make a call to dynamically get the year, month, and subject. Personally I'd configure the script so when it creates the directories they would be named like so:

00000 - 1945.11 - Tom's River, New Jersey (2853)

Just remember to sanitize the input (strip characters like \, /, ;). Otherwise when it goes to create the directory it'll throw an error and the script will come to a screeching halt.

Originally posted by Xtraeme
I've only checked through a couple of the _javascript files, but it appears you can make a call to dynamically get the year, month, and subject. ... Just remember to sanitize the input (strip characters like , /,

.

Er, ok. (You just lost me with the technobabble a bit there, but it sounds good.)

Hopefully now that we have a solid way forward, a few other technically minded people will pitch in as well.

Since some people new to the thread may not see your very helpful post, I may take advantage of a new thread I'll be starting soon (about the easily-downloadable Canadian files) to point people at your input.

All the best,

Isaac

reply to post by IsaacKoi

Sounds good to me. If you'd like this weekend (assuming I don't get stuck with any extra work) I'll write a script and send it over to you. As I said before it will take quite a while for it to grab all the files. Basically when it's executing you really won't be able to use the computer it's operating on because the macro will essentially be overriding the normal user input. So any human input or interruptions will break the script.

Also each of the jpg's are usually about 1 MB. So you're going to want to have about 150 to 200 GBs free. I'd recommend adding in a few extra steps where the script compresses the files into an archive format like a zip. This way you can access the files via a CBR viewer like Comical. It'll make viewing and managing the files a heck of lot easier.

Originally posted by Xtraeme
If you'd like

Mmm. Let me think about that for a moment... Yes, I'd like that VERY much.

Your idea is looking better and better to me!

(assuming I don't get stuck with any extra work)

If other developments (work or otherwise) get in the way, I'll quite understand. We have to keep this hobby of ours in perspective...

Basically when it's executing you really won't be able to use the computer it's operating on because the macro will essentially be overriding the normal user input. So any human input or interruptions will break the script.

Ok - I'll sort something out if that's the way it has to be - e.g. dedicating a laptop to the task for a week or two. (If the macro could be paused it would be a bit easier, then I could let my main home computer work on it when I'm in the office or tucked up in bed and then pause it during the evenings etc).

Also each of the jpg's are usually about 1 MB. So you're going to want to have about 150 to 200 GBs free.

No problem.

I'd recommend adding in a few extra steps to the script where it compresses all the files into an archive format like a zip. This way you can access the files via a CBR viewer like Comical. It'll make viewing and managing the files a heck of lot easier.

I know zip files are smaller but I hadn't heard of them (or any other archive format) making it easier to manage files. I'll have a look at this "Comical" thingamabob...

All the best,

Isaac

Originally posted by zorgon

Originally posted by spacevisitor
So, what is the real truth behind that all?
For what I know now about that subject, I go for the time being for Johns view.

According to the FBI files...

Silas M. Newton — filed under: Unexplained Phenomenon Silas Newton (1887-1972) was a wealthy oil producer and con-man who claimed that he had a gadget that could detect minerals and oil. He was cited as an authority in Frank Scully’s book Behind the Flying Saucers, a work that claimed to report on several UFO crashes in the area of New Mexico. In 1950, Newton said that a flying saucer crashed on land he leased in the Mojave Desert; however, he revised his claim in 1952, saying he never saw a flying saucer but had only repeated comments he heard from others. These files detail the FBI’s investigations into Newton’s fraudulent activities between 1951 and 1970.

vault.fbi.gov...

So... my question is...

is Dana Katherine Scully related to Frank Scully?

Why do you ask me that question and why do you post a link to the FBI files when you said in a reply to moonweed that you thought that EVERYONE knew that the FBI lies to us.?

Originally posted by zorgon

Originally posted by moonweed
just curious....there's something that has been bugging me for quite some time.....did anyone stop for a minute to think that just because some idiot at the FBI stamped "BOGUS" all over the MAJIC documents, that they really are fake? and the so-called "experts" out there that claim they're fake, might have been lying to us, just to throw us off? think about it.

Well DUH... I thought EVERYONE knew that the FBI lies to us

Ya want proof? Here ya go

According to the FBI...

Extra-Sensory Perception — filed under: Unexplained Phenomenon ESP is considered a perception of information about events beyond what may be discerned through the five physical senses or deduced from past experience or knowledge. This release consists of cross references to ESP found in FBI files from 1957 to 1960. Several of the documents concern William Foos, a proponent of ESP. Others concern claims that ESP could be used in espionage investigations. The FBI found no scientific support for this or other claims and did not pursue the matters raised in these references.

vault.fbi.gov...

But at the same time the CIA ran "Project Stargate" from 1968 to 1995 when Congress ruled it wasn't nice to snoop in peoples brains

(after that they just went to private corporations that were not under Congressional scrutiny..

)

reply to post by IsaacKoi

After a little thought I can easily enough have the script remember which page it's on. That should allow you to stop and resume the macro. Also even though CBR's are easy to construct (basically it just collapses all the images into a single archive file with some metadata about the contents -- check out ComicRack to see some more advanced features). I think we'd both prefer to have the files in a format that's text searchable. Maybe if I can find a good way to automate converting the files into the DjVu or PDF format that would be preferable. However this will obviously add considerable overhead because the computer will have to spend a large amount of time OCR'ing the documents. If we do it this way I wouldn't be surprised if it takes a little over a month to complete. Also it'll very likely double the space requirements (400 GBs of space would be preferable).

It's starting to sound like a fun little project.

Originally posted by Xtraeme
reply to post by IsaacKoi

 

After a little thought I can easily enough have the script remember which page it's on. That should allow you to stop and resume the macro.

Better and better...

check out ComicRack to see some more advanced features).

Will do.

I think we'd both prefer to have the files in a format that's text searchable.

That would be ideal. It would add a lot of value to the stored material.

If we do it this way I wouldn't be surprised if it takes a little over a month to complete.

I can live with that.

Also it'll very likely double the space requirements (400 GBs of space would be preferable).

Given the price of 2 TB external hard drives, this isn't a problem.

It's starting to sound like a fun little project.

The outcome - particularly if the OCR angle can be made to work - certainly sounds would be fun to use. GETTING to that outcome is less fun as far as I'm concerned, so I'm REALLY glad someone likes this sort that part of the project!

Onwards and upwards...

All the best,

Isaac

reply to post by Xtraeme



That actually sounds like more work than just clicking the Download button 129,000 times in a row. Maybe I'm not understanding how the script works but it sounds like you would have to manually visit each page to execute the script and grab the document. Remember we have no URL or directory to direct the script to.