read files from hdfs location using spring batch 4 java config
0
I want read all the files from HDFS location and process files sequentially using spring batch.Currently I was using the MultiResourceItemReader to read the files from local file system and processing it.
spring spring-batch
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
add a comment |
0
I want read all the files from HDFS location and process files sequentially using spring batch.Currently I was using the MultiResourceItemReader to read the files from local file system and processing it.
spring spring-batch
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
This answer may help you: stackoverflow.com/a/52774679/5019386
– Mahmoud Ben Hassine
Nov 13 '18 at 7:32
Possible duplicate of Reading file from HDFS using Spring batch
– Mahmoud Ben Hassine
Nov 13 '18 at 7:33
add a comment |
0
0
0
I want read all the files from HDFS location and process files sequentially using spring batch.Currently I was using the MultiResourceItemReader to read the files from local file system and processing it.
spring spring-batch
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
I want read all the files from HDFS location and process files sequentially using spring batch.Currently I was using the MultiResourceItemReader to read the files from local file system and processing it.
spring spring-batch
spring spring-batch
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
asked Nov 13 '18 at 6:16
P RajeshP Rajesh
64
64
This answer may help you: stackoverflow.com/a/52774679/5019386
– Mahmoud Ben Hassine
Nov 13 '18 at 7:32
Possible duplicate of Reading file from HDFS using Spring batch
– Mahmoud Ben Hassine
Nov 13 '18 at 7:33
add a comment |
This answer may help you: stackoverflow.com/a/52774679/5019386
– Mahmoud Ben Hassine
Nov 13 '18 at 7:32
Possible duplicate of Reading file from HDFS using Spring batch
– Mahmoud Ben Hassine
Nov 13 '18 at 7:33
This answer may help you: stackoverflow.com/a/52774679/5019386
– Mahmoud Ben Hassine
Nov 13 '18 at 7:32
This answer may help you: stackoverflow.com/a/52774679/5019386
– Mahmoud Ben Hassine
Nov 13 '18 at 7:32
Possible duplicate of Reading file from HDFS using Spring batch
– Mahmoud Ben Hassine
Nov 13 '18 at 7:33
Possible duplicate of Reading file from HDFS using Spring batch
– Mahmoud Ben Hassine
Nov 13 '18 at 7:33
add a comment |
1 Answer
1
active
oldest
votes
0
l have Read the files from HDFS Location to local file system and spring batch read the files from the local file system.
//Read the files from the hdfs to local file system
private Resource getMultipleResourceItemreader() {
ArrayList<Resource> resource = new ArrayList<Resource>();
org.apache.hadoop.conf.Configuration configuration= new org.apache.hadoop.conf.Configuration();
configuration.set("fs.defaultFS", "hdfs://localhost:9000");
configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
System.setProperty("HADOOP_USER_NAME", "xxxx");
System.setProperty("hadoop.home.dir", "D:\rajesh\softwares\winutils");
FileSystem fs;
try {
fs = FileSystem.get(URI.create("hdfs://localhost:9000"), configuration);
FileStatus files = fs.listStatus(new Path("hdfsfilelocation"));
for (int i=0;i<files.length;i++){
//resource.add( context.getResource(files[i].getPath().toString()));
fs.copyToLocalFile(files[i].getPath(), new Path(batchConfigurationProperties.getCsvFilePath()));
deleteTempFile(batchConfigurationProperties.getCsvFilePath(),".crc");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
File configJsonDirectory = new File(batchConfigurationProperties.getCsvFilePath());
File csvFileList = configJsonDirectory.listFiles();
for (File file : csvFileList) {
if (file.isFile()) {
resource.add(new FileSystemResource(file.getPath()));
}
}
return resource.toArray(new Resource[resource.size()]);
}
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53274899%2fread-files-from-hdfs-location-using-spring-batch-4-java-config%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
l have Read the files from HDFS Location to local file system and spring batch read the files from the local file system.
//Read the files from the hdfs to local file system
private Resource getMultipleResourceItemreader() {
ArrayList<Resource> resource = new ArrayList<Resource>();
org.apache.hadoop.conf.Configuration configuration= new org.apache.hadoop.conf.Configuration();
configuration.set("fs.defaultFS", "hdfs://localhost:9000");
configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
System.setProperty("HADOOP_USER_NAME", "xxxx");
System.setProperty("hadoop.home.dir", "D:\rajesh\softwares\winutils");
FileSystem fs;
try {
fs = FileSystem.get(URI.create("hdfs://localhost:9000"), configuration);
FileStatus files = fs.listStatus(new Path("hdfsfilelocation"));
for (int i=0;i<files.length;i++){
//resource.add( context.getResource(files[i].getPath().toString()));
fs.copyToLocalFile(files[i].getPath(), new Path(batchConfigurationProperties.getCsvFilePath()));
deleteTempFile(batchConfigurationProperties.getCsvFilePath(),".crc");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
File configJsonDirectory = new File(batchConfigurationProperties.getCsvFilePath());
File csvFileList = configJsonDirectory.listFiles();
for (File file : csvFileList) {
if (file.isFile()) {
resource.add(new FileSystemResource(file.getPath()));
}
}
return resource.toArray(new Resource[resource.size()]);
}
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
add a comment |
0
l have Read the files from HDFS Location to local file system and spring batch read the files from the local file system.
//Read the files from the hdfs to local file system
private Resource getMultipleResourceItemreader() {
ArrayList<Resource> resource = new ArrayList<Resource>();
org.apache.hadoop.conf.Configuration configuration= new org.apache.hadoop.conf.Configuration();
configuration.set("fs.defaultFS", "hdfs://localhost:9000");
configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
System.setProperty("HADOOP_USER_NAME", "xxxx");
System.setProperty("hadoop.home.dir", "D:\rajesh\softwares\winutils");
FileSystem fs;
try {
fs = FileSystem.get(URI.create("hdfs://localhost:9000"), configuration);
FileStatus files = fs.listStatus(new Path("hdfsfilelocation"));
for (int i=0;i<files.length;i++){
//resource.add( context.getResource(files[i].getPath().toString()));
fs.copyToLocalFile(files[i].getPath(), new Path(batchConfigurationProperties.getCsvFilePath()));
deleteTempFile(batchConfigurationProperties.getCsvFilePath(),".crc");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
File configJsonDirectory = new File(batchConfigurationProperties.getCsvFilePath());
File csvFileList = configJsonDirectory.listFiles();
for (File file : csvFileList) {
if (file.isFile()) {
resource.add(new FileSystemResource(file.getPath()));
}
}
return resource.toArray(new Resource[resource.size()]);
}
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
add a comment |
0
0
0
l have Read the files from HDFS Location to local file system and spring batch read the files from the local file system.
//Read the files from the hdfs to local file system
private Resource getMultipleResourceItemreader() {
ArrayList<Resource> resource = new ArrayList<Resource>();
org.apache.hadoop.conf.Configuration configuration= new org.apache.hadoop.conf.Configuration();
configuration.set("fs.defaultFS", "hdfs://localhost:9000");
configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
System.setProperty("HADOOP_USER_NAME", "xxxx");
System.setProperty("hadoop.home.dir", "D:\rajesh\softwares\winutils");
FileSystem fs;
try {
fs = FileSystem.get(URI.create("hdfs://localhost:9000"), configuration);
FileStatus files = fs.listStatus(new Path("hdfsfilelocation"));
for (int i=0;i<files.length;i++){
//resource.add( context.getResource(files[i].getPath().toString()));
fs.copyToLocalFile(files[i].getPath(), new Path(batchConfigurationProperties.getCsvFilePath()));
deleteTempFile(batchConfigurationProperties.getCsvFilePath(),".crc");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
File configJsonDirectory = new File(batchConfigurationProperties.getCsvFilePath());
File csvFileList = configJsonDirectory.listFiles();
for (File file : csvFileList) {
if (file.isFile()) {
resource.add(new FileSystemResource(file.getPath()));
}
}
return resource.toArray(new Resource[resource.size()]);
}
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
l have Read the files from HDFS Location to local file system and spring batch read the files from the local file system.
//Read the files from the hdfs to local file system
private Resource getMultipleResourceItemreader() {
ArrayList<Resource> resource = new ArrayList<Resource>();
org.apache.hadoop.conf.Configuration configuration= new org.apache.hadoop.conf.Configuration();
configuration.set("fs.defaultFS", "hdfs://localhost:9000");
configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
System.setProperty("HADOOP_USER_NAME", "xxxx");
System.setProperty("hadoop.home.dir", "D:\rajesh\softwares\winutils");
FileSystem fs;
try {
fs = FileSystem.get(URI.create("hdfs://localhost:9000"), configuration);
FileStatus files = fs.listStatus(new Path("hdfsfilelocation"));
for (int i=0;i<files.length;i++){
//resource.add( context.getResource(files[i].getPath().toString()));
fs.copyToLocalFile(files[i].getPath(), new Path(batchConfigurationProperties.getCsvFilePath()));
deleteTempFile(batchConfigurationProperties.getCsvFilePath(),".crc");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
File configJsonDirectory = new File(batchConfigurationProperties.getCsvFilePath());
File csvFileList = configJsonDirectory.listFiles();
for (File file : csvFileList) {
if (file.isFile()) {
resource.add(new FileSystemResource(file.getPath()));
}
}
return resource.toArray(new Resource[resource.size()]);
}
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
edited Nov 14 '18 at 7:03
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
answered Nov 14 '18 at 6:58
P RajeshP Rajesh
64
64
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
add a comment |
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
how to read the file and process from hdfs location using flatfileItemreader with out copying into local file system from hdfs
– P Rajesh
Nov 20 '18 at 8:04
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53274899%2fread-files-from-hdfs-location-using-spring-batch-4-java-config%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
This answer may help you: stackoverflow.com/a/52774679/5019386
– Mahmoud Ben Hassine
Nov 13 '18 at 7:32
Possible duplicate of Reading file from HDFS using Spring batch
– Mahmoud Ben Hassine
Nov 13 '18 at 7:33