Help ATS with a contribution via PayPal:
learn more

HTTrack wildcard help? Downloading Australian UFO documents

page: 2
1
<< 1   >>

log in

join

posted on Mar, 25 2013 @ 02:20 PM
link   
I'm now at over 10,000 pages of UFO files downloaded from the Australian National Archives...

I'm looking into getting permission from their National Archives to share the searchable PDF files that I've made. If I can't get that permission, I'll just do a thread like my initial one on the Canadian UFO documents (i.e. outlining how to download the document images and giving some highlights):
www.abovetopsecret.com...

I know this would not be as useful as sharing the searchable PDF files and would involve a bit more effort for people wanting to download the material. Subject to sorting out the copyright permission, I'd prefer to share the searchable PDF files, in the same way that I've done recently with other material e.g. collections of FBI documents and (after I got the permission of the Canadian government) the Canadian UFO files:
www.abovetopsecret.com...
www.abovetopsecret.com...

edit on 25-3-2013 by IsaacKoi because: (no reason given)




posted on Apr, 18 2013 @ 10:11 AM
link   

Originally posted by IsaacKoi
I'm looking into getting permission from their National Archives to share the searchable PDF files that I've made.


After another email exchange, I received the relevant permission on behalf of the Australian Government last night. I've uploaded the relevant PDF files of Australian UFO documents (containing over 10,000 pages) to a free file storage website and now just need to find the time to finish writing/researching a thread highlighting some of the more interesting documents in that collection (well, more interesting to me anyway
).

Thanks again for the help and input given in this thread. I'll be sure to give credit for the assistance when I post the relevant thread, which hopefully should be within the next week.
edit on 18-4-2013 by IsaacKoi because: (no reason given)



posted on Apr, 19 2013 @ 04:06 PM
link   

Originally posted by IsaacKoi
The code I've used is:



@echo off
:beginning
cls

set cnt=1
set /p url= What is the base URL (i.e. before the number that changes)?
set /p end= How many pages are in the relevant file?

:downloadloop
echo %cnt%
wget64 -O "C
ownloadswgetimagesimage-%cnt%.jpg" "%url%%cnt%"
set /a cnt=%cnt%+1
if %cnt% GTR %end% goto finishquestion
goto downloadloop
:end

:finishquestion
set /p finished= Have you finished downloading (yes/no)?
If "%finished%" == "no" goto beginning

:close



To enable me to download higher resolution images, I would now like to run the same code but to add "&T=p" as a suffix to the URLs generated by the above code, i.e. so that the URL includes &T=r after the number generated by the cnt loop.

I presume I can just define a suffix ("suf") and then change the URL in the wget line of the code to include the suf variable (e.g.: wget64 -O "C
ownloadswgetimagesimage-%cnt%.jpg" "%url%%cnt%%suf%"), but I'm not sure of the format for defining the "suf" variable given that it includes an equals sign.

I've tried using the line below but that didn't work:
set suf="&T=p"

It's been a couple of decades since I last tried doing this sort of thing in a batch file, so I'd be grateful if someone can refresh my memory on how to deal with this sort of issue.

(Perhaps I should be using some sort of symbol and/or code number in place of the equal sign within the "&T=p"??)

EDIT - I've just been looking at ASCII encoding in URLs and think the relevant definition of the suf variable should be something like:
%26T%3dp

I'll play around with this, unless someone knows the answer off the top of their head.

ANOTHER EDIT: Well, by changing the definition of the "suf" (suffix) variable so that it requires a user input, I can just type &T=p and the code then works. I'd rather not have to do this each time (since I'm not re-downloading more than 70 files) but I can live with that rather than worry too much about the correct way to define the variable within the batch file itself. Quick and dirty works for me...


edit on 19-4-2013 by IsaacKoi because: (no reason given)
edit on 19-4-2013 by IsaacKoi because: (no reason given)



posted on Apr, 24 2013 @ 08:51 AM
link   

Originally posted by IsaacKoi
I received the relevant permission on behalf of the Australian Government last night. I've uploaded the relevant PDF files of Australian UFO documents (containing over 10,000 pages) to a free file storage website and now just need to find the time to finish writing/researching a thread highlighting some of the more interesting documents in that collection (well, more interesting to me anyway
).


With thanks again for the assistance in this thread, I've now posted a thread about the archive of searchable PDF documents:
Oz X-Files : “authenticated sighting”, “appear to be inexplicable”, USA “very interested"





new topics

top topics
 
1
<< 1   >>

log in

join