How to get s3 objects in a loop?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
1
In Node.js I'm attempting to retrieve objects by looping through an array using the fs.createReadStream and fs.createWriteStream methods.
AWS documentation shows how to retrieve a single object with
s3.getObject(params).createReadStream().pipe(file);
But with params and s3 set as
const params = { Bucket:'user_events' };
const s3 = new AWS.S3();
When I call my function:
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let file = eventsArray[i];
params.Key = file;
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(params).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
delete params.Key;
return;
}
});
});
}
The output produces:
--> finding files from s3...
0 9 'harry_test_audio_09.wav'
1 9 'harry_test_audio_08.wav'
2 9 'harry_test_audio_07.wav'
3 9 'harry_test_audio_06.wav'
4 9 'harry_test_audio_05.wav'
5 9 'harry_test_audio_04.wav'
6 9 'harry_test_audio_03.wav'
7 9 'harry_test_audio_02.wav'
8 9 'harry_test_audio_01.wav'
6 -- file added: harry_test_audio_03.wav
8 -- file added: harry_test_audio_01.wav
7 -- file added: harry_test_audio_02.wav
0 -- file added: harry_test_audio_09.wav
5 -- file added: harry_test_audio_04.wav
1 -- file added: harry_test_audio_08.wav
3 -- file added: harry_test_audio_06.wav
4 -- file added: harry_test_audio_05.wav
2 -- file added: harry_test_audio_07.wav
-- success! --
Which produces 9 files with correct names each with only the content of the first file inside.
I also tried using stream.on('finish' ... and stream.on('end' ... with similar results.
What am I doing wrong?
javascript node.js amazon-web-services amazon-s3 fs
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
add a comment |
1
In Node.js I'm attempting to retrieve objects by looping through an array using the fs.createReadStream and fs.createWriteStream methods.
AWS documentation shows how to retrieve a single object with
s3.getObject(params).createReadStream().pipe(file);
But with params and s3 set as
const params = { Bucket:'user_events' };
const s3 = new AWS.S3();
When I call my function:
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let file = eventsArray[i];
params.Key = file;
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(params).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
delete params.Key;
return;
}
});
});
}
The output produces:
--> finding files from s3...
0 9 'harry_test_audio_09.wav'
1 9 'harry_test_audio_08.wav'
2 9 'harry_test_audio_07.wav'
3 9 'harry_test_audio_06.wav'
4 9 'harry_test_audio_05.wav'
5 9 'harry_test_audio_04.wav'
6 9 'harry_test_audio_03.wav'
7 9 'harry_test_audio_02.wav'
8 9 'harry_test_audio_01.wav'
6 -- file added: harry_test_audio_03.wav
8 -- file added: harry_test_audio_01.wav
7 -- file added: harry_test_audio_02.wav
0 -- file added: harry_test_audio_09.wav
5 -- file added: harry_test_audio_04.wav
1 -- file added: harry_test_audio_08.wav
3 -- file added: harry_test_audio_06.wav
4 -- file added: harry_test_audio_05.wav
2 -- file added: harry_test_audio_07.wav
-- success! --
Which produces 9 files with correct names each with only the content of the first file inside.
I also tried using stream.on('finish' ... and stream.on('end' ... with similar results.
What am I doing wrong?
javascript node.js amazon-web-services amazon-s3 fs
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
you can use "list objects" method present in s3client. I am not sure about the way to write a code in node.js. But in java, there is a method listObjects(ListObjectsRequest) and in ListObjectsRequest you can set bucket name and prefix. With the help of this you can get list of objects present in your s3 bucket. After that you can fetch Object summaries. Each object summary contains one key from that bucket. you can loop object summaries and use getObject on each key derived from that loop.
– Suyash
May 22 '18 at 12:32
Most likely when something like this happens in Javascript it has to do something with closures. I didn't debug your code, but it sounds like this is the problem. Read more about this on for example: decembersoft.com/posts/…
– Jørgen
May 22 '18 at 13:38
Thanks @Suyash, butlistObjects();only returns the names and metadata of the objects in s3, while I'm trying to write the actual files to tmp/
– JT Houk
May 22 '18 at 13:42
u may want to check whether the readstream.pipe blocks or whether its async with independ lifecycle events indicating 'onData' , onEnd... if its the latter then u would need to alter the code to be async to wait til each stream / each fileOut finishes inside the loop
– Robert Rowntree
May 22 '18 at 14:02
using listObjects() you will get object of ObjectsListings from which you can get list of object summaries. Each object summary contains the metadata which contains the s3 object key. So using that key and bucket name with getObject() you can retrieve actual object from s3 bucket. You want to fetch all the objects from s3 bucket, right?
– Suyash
May 22 '18 at 14:10
add a comment |
1
1
1
In Node.js I'm attempting to retrieve objects by looping through an array using the fs.createReadStream and fs.createWriteStream methods.
AWS documentation shows how to retrieve a single object with
s3.getObject(params).createReadStream().pipe(file);
But with params and s3 set as
const params = { Bucket:'user_events' };
const s3 = new AWS.S3();
When I call my function:
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let file = eventsArray[i];
params.Key = file;
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(params).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
delete params.Key;
return;
}
});
});
}
The output produces:
--> finding files from s3...
0 9 'harry_test_audio_09.wav'
1 9 'harry_test_audio_08.wav'
2 9 'harry_test_audio_07.wav'
3 9 'harry_test_audio_06.wav'
4 9 'harry_test_audio_05.wav'
5 9 'harry_test_audio_04.wav'
6 9 'harry_test_audio_03.wav'
7 9 'harry_test_audio_02.wav'
8 9 'harry_test_audio_01.wav'
6 -- file added: harry_test_audio_03.wav
8 -- file added: harry_test_audio_01.wav
7 -- file added: harry_test_audio_02.wav
0 -- file added: harry_test_audio_09.wav
5 -- file added: harry_test_audio_04.wav
1 -- file added: harry_test_audio_08.wav
3 -- file added: harry_test_audio_06.wav
4 -- file added: harry_test_audio_05.wav
2 -- file added: harry_test_audio_07.wav
-- success! --
Which produces 9 files with correct names each with only the content of the first file inside.
I also tried using stream.on('finish' ... and stream.on('end' ... with similar results.
What am I doing wrong?
javascript node.js amazon-web-services amazon-s3 fs
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
In Node.js I'm attempting to retrieve objects by looping through an array using the fs.createReadStream and fs.createWriteStream methods.
AWS documentation shows how to retrieve a single object with
s3.getObject(params).createReadStream().pipe(file);
But with params and s3 set as
const params = { Bucket:'user_events' };
const s3 = new AWS.S3();
When I call my function:
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let file = eventsArray[i];
params.Key = file;
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(params).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
delete params.Key;
return;
}
});
});
}
The output produces:
--> finding files from s3...
0 9 'harry_test_audio_09.wav'
1 9 'harry_test_audio_08.wav'
2 9 'harry_test_audio_07.wav'
3 9 'harry_test_audio_06.wav'
4 9 'harry_test_audio_05.wav'
5 9 'harry_test_audio_04.wav'
6 9 'harry_test_audio_03.wav'
7 9 'harry_test_audio_02.wav'
8 9 'harry_test_audio_01.wav'
6 -- file added: harry_test_audio_03.wav
8 -- file added: harry_test_audio_01.wav
7 -- file added: harry_test_audio_02.wav
0 -- file added: harry_test_audio_09.wav
5 -- file added: harry_test_audio_04.wav
1 -- file added: harry_test_audio_08.wav
3 -- file added: harry_test_audio_06.wav
4 -- file added: harry_test_audio_05.wav
2 -- file added: harry_test_audio_07.wav
-- success! --
Which produces 9 files with correct names each with only the content of the first file inside.
I also tried using stream.on('finish' ... and stream.on('end' ... with similar results.
What am I doing wrong?
javascript node.js amazon-web-services amazon-s3 fs
javascript node.js amazon-web-services amazon-s3 fs
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
edited May 22 '18 at 13:33
JT Houk
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
asked May 22 '18 at 11:52
JT HoukJT Houk
5310
5310
you can use "list objects" method present in s3client. I am not sure about the way to write a code in node.js. But in java, there is a method listObjects(ListObjectsRequest) and in ListObjectsRequest you can set bucket name and prefix. With the help of this you can get list of objects present in your s3 bucket. After that you can fetch Object summaries. Each object summary contains one key from that bucket. you can loop object summaries and use getObject on each key derived from that loop.
– Suyash
May 22 '18 at 12:32
Most likely when something like this happens in Javascript it has to do something with closures. I didn't debug your code, but it sounds like this is the problem. Read more about this on for example: decembersoft.com/posts/…
– Jørgen
May 22 '18 at 13:38
Thanks @Suyash, butlistObjects();only returns the names and metadata of the objects in s3, while I'm trying to write the actual files to tmp/
– JT Houk
May 22 '18 at 13:42
u may want to check whether the readstream.pipe blocks or whether its async with independ lifecycle events indicating 'onData' , onEnd... if its the latter then u would need to alter the code to be async to wait til each stream / each fileOut finishes inside the loop
– Robert Rowntree
May 22 '18 at 14:02
using listObjects() you will get object of ObjectsListings from which you can get list of object summaries. Each object summary contains the metadata which contains the s3 object key. So using that key and bucket name with getObject() you can retrieve actual object from s3 bucket. You want to fetch all the objects from s3 bucket, right?
– Suyash
May 22 '18 at 14:10
add a comment |
you can use "list objects" method present in s3client. I am not sure about the way to write a code in node.js. But in java, there is a method listObjects(ListObjectsRequest) and in ListObjectsRequest you can set bucket name and prefix. With the help of this you can get list of objects present in your s3 bucket. After that you can fetch Object summaries. Each object summary contains one key from that bucket. you can loop object summaries and use getObject on each key derived from that loop.
– Suyash
May 22 '18 at 12:32
Most likely when something like this happens in Javascript it has to do something with closures. I didn't debug your code, but it sounds like this is the problem. Read more about this on for example: decembersoft.com/posts/…
– Jørgen
May 22 '18 at 13:38
Thanks @Suyash, butlistObjects();only returns the names and metadata of the objects in s3, while I'm trying to write the actual files to tmp/
– JT Houk
May 22 '18 at 13:42
u may want to check whether the readstream.pipe blocks or whether its async with independ lifecycle events indicating 'onData' , onEnd... if its the latter then u would need to alter the code to be async to wait til each stream / each fileOut finishes inside the loop
– Robert Rowntree
May 22 '18 at 14:02
using listObjects() you will get object of ObjectsListings from which you can get list of object summaries. Each object summary contains the metadata which contains the s3 object key. So using that key and bucket name with getObject() you can retrieve actual object from s3 bucket. You want to fetch all the objects from s3 bucket, right?
– Suyash
May 22 '18 at 14:10
you can use "list objects" method present in s3client. I am not sure about the way to write a code in node.js. But in java, there is a method listObjects(ListObjectsRequest) and in ListObjectsRequest you can set bucket name and prefix. With the help of this you can get list of objects present in your s3 bucket. After that you can fetch Object summaries. Each object summary contains one key from that bucket. you can loop object summaries and use getObject on each key derived from that loop.
– Suyash
May 22 '18 at 12:32
you can use "list objects" method present in s3client. I am not sure about the way to write a code in node.js. But in java, there is a method listObjects(ListObjectsRequest) and in ListObjectsRequest you can set bucket name and prefix. With the help of this you can get list of objects present in your s3 bucket. After that you can fetch Object summaries. Each object summary contains one key from that bucket. you can loop object summaries and use getObject on each key derived from that loop.
– Suyash
May 22 '18 at 12:32
Most likely when something like this happens in Javascript it has to do something with closures. I didn't debug your code, but it sounds like this is the problem. Read more about this on for example: decembersoft.com/posts/…
– Jørgen
May 22 '18 at 13:38
Most likely when something like this happens in Javascript it has to do something with closures. I didn't debug your code, but it sounds like this is the problem. Read more about this on for example: decembersoft.com/posts/…
– Jørgen
May 22 '18 at 13:38
Thanks @Suyash, but
listObjects(); only returns the names and metadata of the objects in s3, while I'm trying to write the actual files to tmp/– JT Houk
May 22 '18 at 13:42
Thanks @Suyash, but
listObjects(); only returns the names and metadata of the objects in s3, while I'm trying to write the actual files to tmp/– JT Houk
May 22 '18 at 13:42
u may want to check whether the readstream.pipe blocks or whether its async with independ lifecycle events indicating 'onData' , onEnd... if its the latter then u would need to alter the code to be async to wait til each stream / each fileOut finishes inside the loop
– Robert Rowntree
May 22 '18 at 14:02
u may want to check whether the readstream.pipe blocks or whether its async with independ lifecycle events indicating 'onData' , onEnd... if its the latter then u would need to alter the code to be async to wait til each stream / each fileOut finishes inside the loop
– Robert Rowntree
May 22 '18 at 14:02
using listObjects() you will get object of ObjectsListings from which you can get list of object summaries. Each object summary contains the metadata which contains the s3 object key. So using that key and bucket name with getObject() you can retrieve actual object from s3 bucket. You want to fetch all the objects from s3 bucket, right?
– Suyash
May 22 '18 at 14:10
using listObjects() you will get object of ObjectsListings from which you can get list of object summaries. Each object summary contains the metadata which contains the s3 object key. So using that key and bucket name with getObject() you can retrieve actual object from s3 bucket. You want to fetch all the objects from s3 bucket, right?
– Suyash
May 22 '18 at 14:10
add a comment |
1 Answer
1
active
oldest
votes
1
I encountered the same problem and it was due to bad closure in the loop.
The solution is to create a copy of params that is not shared among all the iterations.
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let fileParams = {
Bucket: 'user_events',
Key: eventsArray[i]
}
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(fileParams).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
return;
}
});
}
});
}
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f50467006%2fhow-to-get-s3-objects-in-a-loop%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
I encountered the same problem and it was due to bad closure in the loop.
The solution is to create a copy of params that is not shared among all the iterations.
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let fileParams = {
Bucket: 'user_events',
Key: eventsArray[i]
}
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(fileParams).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
return;
}
});
}
});
}
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
add a comment |
1
I encountered the same problem and it was due to bad closure in the loop.
The solution is to create a copy of params that is not shared among all the iterations.
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let fileParams = {
Bucket: 'user_events',
Key: eventsArray[i]
}
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(fileParams).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
return;
}
});
}
});
}
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
add a comment |
1
1
1
I encountered the same problem and it was due to bad closure in the loop.
The solution is to create a copy of params that is not shared among all the iterations.
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let fileParams = {
Bucket: 'user_events',
Key: eventsArray[i]
}
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(fileParams).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
return;
}
});
}
});
}
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
I encountered the same problem and it was due to bad closure in the loop.
The solution is to create a copy of params that is not shared among all the iterations.
function gets3Objects(eventsArray) {
console.log('--> finding files from s3...');
const arrLen = eventsArray.length;
let iter = 0;
s3.listObjects(params, (err, data) => {
for (let i = 0; i < arrLen; i += 1) {
let fileParams = {
Bucket: 'user_events',
Key: eventsArray[i]
}
let fileOut = fs.createWriteStream(`./tmp/${file}`);
let stream = s3.getObject(fileParams).createReadStream().pipe(fileOut);
console.log(i, arrLen, eventsArray[i]);
stream.on('close', () => {
iter += 1;
console.log(`${i} -- file added: ${eventsArray[i]}`);
if (iter === arrLen) {
console.log('-- success! --');
return;
}
});
}
});
}
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
answered Nov 16 '18 at 15:45
SputnikSputnik
13815
13815
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f50467006%2fhow-to-get-s3-objects-in-a-loop%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
you can use "list objects" method present in s3client. I am not sure about the way to write a code in node.js. But in java, there is a method listObjects(ListObjectsRequest) and in ListObjectsRequest you can set bucket name and prefix. With the help of this you can get list of objects present in your s3 bucket. After that you can fetch Object summaries. Each object summary contains one key from that bucket. you can loop object summaries and use getObject on each key derived from that loop.
– Suyash
May 22 '18 at 12:32
Most likely when something like this happens in Javascript it has to do something with closures. I didn't debug your code, but it sounds like this is the problem. Read more about this on for example: decembersoft.com/posts/…
– Jørgen
May 22 '18 at 13:38
Thanks @Suyash, but
listObjects();only returns the names and metadata of the objects in s3, while I'm trying to write the actual files to tmp/– JT Houk
May 22 '18 at 13:42
u may want to check whether the readstream.pipe blocks or whether its async with independ lifecycle events indicating 'onData' , onEnd... if its the latter then u would need to alter the code to be async to wait til each stream / each fileOut finishes inside the loop
– Robert Rowntree
May 22 '18 at 14:02
using listObjects() you will get object of ObjectsListings from which you can get list of object summaries. Each object summary contains the metadata which contains the s3 object key. So using that key and bucket name with getObject() you can retrieve actual object from s3 bucket. You want to fetch all the objects from s3 bucket, right?
– Suyash
May 22 '18 at 14:10