wget to download multiple folders from https url
I am using following line of wget to download dataset.
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/
--header "Authorization: Bearer <your appKey>" -P <target dir>
Now, I want to download multiple folders from same URL (i.e. https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672579/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672580/ etc)
I am using for loop in shell script for a same as following:
for i in 50672578 50672579 50672580
do
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/$i/
--header "Authorization: Bearer <your appKey>" -P <target dir>
done
But this gives me the following error:
Proxy request sent, awaiting a response... 400 Bad Request
2018-11-12 12:18:34 ERROR 400: Bad Request.
Individually it is working fine but not via for-loop.
Kindly request to indicate my mistakes.
Thank you in advance
shell for-loop wget
asked Nov 12 at 6:50
Piyushkumar Patel
154
add a comment |
I am using following line of wget to download dataset.
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/
--header "Authorization: Bearer <your appKey>" -P <target dir>
Now, I want to download multiple folders from same URL (i.e. https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672579/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672580/ etc)
I am using for loop in shell script for a same as following:
for i in 50672578 50672579 50672580
do
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/$i/
--header "Authorization: Bearer <your appKey>" -P <target dir>
done
But this gives me the following error:
Proxy request sent, awaiting a response... 400 Bad Request
2018-11-12 12:18:34 ERROR 400: Bad Request.
Individually it is working fine but not via for-loop.
Kindly request to indicate my mistakes.
Thank you in advance
shell for-loop wget
asked Nov 12 at 6:50
Piyushkumar Patel
154
Can you runwgetwith "-v" and post the output here?
– Samuel
Nov 12 at 7:01
add a comment |
I am using following line of wget to download dataset.
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/
--header "Authorization: Bearer <your appKey>" -P <target dir>
Now, I want to download multiple folders from same URL (i.e. https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672579/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672580/ etc)
I am using for loop in shell script for a same as following:
for i in 50672578 50672579 50672580
do
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/$i/
--header "Authorization: Bearer <your appKey>" -P <target dir>
done
But this gives me the following error:
Proxy request sent, awaiting a response... 400 Bad Request
2018-11-12 12:18:34 ERROR 400: Bad Request.
Individually it is working fine but not via for-loop.
Kindly request to indicate my mistakes.
Thank you in advance
shell for-loop wget
asked Nov 12 at 6:50
Piyushkumar Patel
154
I am using following line of wget to download dataset.
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/
--header "Authorization: Bearer <your appKey>" -P <target dir>
Now, I want to download multiple folders from same URL (i.e. https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672578/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672579/; https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/50672580/ etc)
I am using for loop in shell script for a same as following:
for i in 50672578 50672579 50672580
do
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/$i/
--header "Authorization: Bearer <your appKey>" -P <target dir>
done
But this gives me the following error:
Proxy request sent, awaiting a response... 400 Bad Request
2018-11-12 12:18:34 ERROR 400: Bad Request.
Individually it is working fine but not via for-loop.
Kindly request to indicate my mistakes.
Thank you in advance
shell for-loop wget
shell for-loop wget
asked Nov 12 at 6:50
Piyushkumar Patel
154
asked Nov 12 at 6:50
Piyushkumar Patel
154
edited Nov 12 at 6:57
asked Nov 12 at 6:50
Piyushkumar Patel
154
asked Nov 12 at 6:50
Piyushkumar Patel
154
asked Nov 12 at 6:50
Piyushkumar Patel
154
154
Can you runwgetwith "-v" and post the output here?
– Samuel
Nov 12 at 7:01
add a comment |
Can you runwgetwith "-v" and post the output here?
– Samuel
Nov 12 at 7:01
Can you run
wget with "-v" and post the output here?– Samuel
Nov 12 at 7:01
Can you run
wget with "-v" and post the output here?– Samuel
Nov 12 at 7:01
add a comment |
2 Answers
2
active
oldest
votes
There is no need to use a for loop you can express the list of urls as follows:
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/{50672578,50672579,50672580}/
answered Nov 13 at 7:32
gopy
1637
add a comment |
Can you assign URL's in array like below and read them in for loop
ARRAY=(URL1 URL2... URLN)
answered Nov 12 at 8:25
Jithin Babu
215
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53257137%2fwget-to-download-multiple-folders-from-https-url%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
There is no need to use a for loop you can express the list of urls as follows:
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/{50672578,50672579,50672580}/
answered Nov 13 at 7:32
gopy
1637
add a comment |
There is no need to use a for loop you can express the list of urls as follows:
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/{50672578,50672579,50672580}/
answered Nov 13 at 7:32
gopy
1637
add a comment |
There is no need to use a for loop you can express the list of urls as follows:
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/{50672578,50672579,50672580}/
answered Nov 13 at 7:32
gopy
1637
There is no need to use a for loop you can express the list of urls as follows:
https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/{50672578,50672579,50672580}/
answered Nov 13 at 7:32
gopy
1637
edited Nov 13 at 9:46
answered Nov 13 at 7:32
gopy
1637
answered Nov 13 at 7:32
gopy
1637
answered Nov 13 at 7:32
gopy
1637
1637
add a comment |
add a comment |
Can you assign URL's in array like below and read them in for loop
ARRAY=(URL1 URL2... URLN)
answered Nov 12 at 8:25
Jithin Babu
215
add a comment |
Can you assign URL's in array like below and read them in for loop
ARRAY=(URL1 URL2... URLN)
answered Nov 12 at 8:25
Jithin Babu
215
add a comment |
Can you assign URL's in array like below and read them in for loop
ARRAY=(URL1 URL2... URLN)
answered Nov 12 at 8:25
Jithin Babu
215
Can you assign URL's in array like below and read them in for loop
ARRAY=(URL1 URL2... URLN)
answered Nov 12 at 8:25
Jithin Babu
215
answered Nov 12 at 8:25
Jithin Babu
215
answered Nov 12 at 8:25
Jithin Babu
215
answered Nov 12 at 8:25
Jithin Babu
215
215
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53257137%2fwget-to-download-multiple-folders-from-https-url%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you run
wgetwith "-v" and post the output here?– Samuel
Nov 12 at 7:01