how can i read file fastest way in c++ ? to send data to 'Vertex Buffer Object
i have 4-5 gigabyte ".obj" files like:
Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.
files contents these :
v 16.418303 40.112064 0.078153
vt 0.445198 0.462720
vn 0.264392 0.654428 0.708394
f 27/72/32 38/73/43 23/74/13
they are classic texts
and i am reading each them like this:
char text[10000][60];
ifstream in(filename);
if(!in.is_open())
{
return false;
}
while(!in.eof())
{
in.getline(text[i],60);
i++;
}
after read one folder i am sending datas to 'VBO'.
all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.
how can i make it faster ?
if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.
c++
edited Nov 12 at 4:02
eyllanesc
72.7k93055
asked Nov 12 at 3:05
Emre Kaya
103
add a comment |
i have 4-5 gigabyte ".obj" files like:
Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.
files contents these :
v 16.418303 40.112064 0.078153
vt 0.445198 0.462720
vn 0.264392 0.654428 0.708394
f 27/72/32 38/73/43 23/74/13
they are classic texts
and i am reading each them like this:
char text[10000][60];
ifstream in(filename);
if(!in.is_open())
{
return false;
}
while(!in.eof())
{
in.getline(text[i],60);
i++;
}
after read one folder i am sending datas to 'VBO'.
all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.
how can i make it faster ?
if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.
c++
edited Nov 12 at 4:02
eyllanesc
72.7k93055
asked Nov 12 at 3:05
Emre Kaya
103
So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31
Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46
If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10
thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19
add a comment |
i have 4-5 gigabyte ".obj" files like:
Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.
files contents these :
v 16.418303 40.112064 0.078153
vt 0.445198 0.462720
vn 0.264392 0.654428 0.708394
f 27/72/32 38/73/43 23/74/13
they are classic texts
and i am reading each them like this:
char text[10000][60];
ifstream in(filename);
if(!in.is_open())
{
return false;
}
while(!in.eof())
{
in.getline(text[i],60);
i++;
}
after read one folder i am sending datas to 'VBO'.
all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.
how can i make it faster ?
if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.
c++
edited Nov 12 at 4:02
eyllanesc
72.7k93055
asked Nov 12 at 3:05
Emre Kaya
103
i have 4-5 gigabyte ".obj" files like:
Sword_1 folder has 2000 obj files , Sword_2 folder has 2000, Dagger_1 folder has 2000 ... etc totaly i have 4-5 gigabyte ".obj" files.
files contents these :
v 16.418303 40.112064 0.078153
vt 0.445198 0.462720
vn 0.264392 0.654428 0.708394
f 27/72/32 38/73/43 23/74/13
they are classic texts
and i am reading each them like this:
char text[10000][60];
ifstream in(filename);
if(!in.is_open())
{
return false;
}
while(!in.eof())
{
in.getline(text[i],60);
i++;
}
after read one folder i am sending datas to 'VBO'.
all files take 10 minutes to complete.
when i make multithreading with '3 thread' (my cpu 4core 4 thread) , then same process take 3-4 minutes to complete.
but still too long.
how can i make it faster ?
if i need to use binary, how can i read binary with c++ ? can you make example for me :) because i don't know anything about binaries. and if i made binary i needed to convert 'char' format ? and if i need to convert, convert process will take same minutes to complete ? sorry for bad english.
c++
c++
edited Nov 12 at 4:02
eyllanesc
72.7k93055
asked Nov 12 at 3:05
Emre Kaya
103
edited Nov 12 at 4:02
eyllanesc
72.7k93055
asked Nov 12 at 3:05
Emre Kaya
103
edited Nov 12 at 4:02
eyllanesc
72.7k93055
edited Nov 12 at 4:02
eyllanesc
72.7k93055
edited Nov 12 at 4:02
eyllanesc
72.7k93055
72.7k93055
asked Nov 12 at 3:05
Emre Kaya
103
asked Nov 12 at 3:05
Emre Kaya
103
asked Nov 12 at 3:05
Emre Kaya
103
103
So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31
Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46
If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10
thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19
add a comment |
So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31
Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46
If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10
thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19
So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31
So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31
Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46
Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46
If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10
If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10
thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19
thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19
add a comment |
1 Answer
1
active
oldest
votes
There is no "precise final answer to your question", once it depends on a couple variables here.
C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).
Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.
Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)
Google for it and learn a little more.
About reading binary data in C++ is easy:
std::ifstream file("somedata.obj", ios::binary);
if (file)
{
char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!
}
But again, this is not enough. You need to wisely interpret the data you just read.
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255477%2fhow-can-i-read-file-fastest-way-in-c-to-send-data-to-vertex-buffer-object%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
There is no "precise final answer to your question", once it depends on a couple variables here.
C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).
Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.
Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)
Google for it and learn a little more.
About reading binary data in C++ is easy:
std::ifstream file("somedata.obj", ios::binary);
if (file)
{
char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!
}
But again, this is not enough. You need to wisely interpret the data you just read.
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
add a comment |
There is no "precise final answer to your question", once it depends on a couple variables here.
C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).
Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.
Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)
Google for it and learn a little more.
About reading binary data in C++ is easy:
std::ifstream file("somedata.obj", ios::binary);
if (file)
{
char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!
}
But again, this is not enough. You need to wisely interpret the data you just read.
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
add a comment |
There is no "precise final answer to your question", once it depends on a couple variables here.
C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).
Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.
Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)
Google for it and learn a little more.
About reading binary data in C++ is easy:
std::ifstream file("somedata.obj", ios::binary);
if (file)
{
char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!
}
But again, this is not enough. You need to wisely interpret the data you just read.
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
There is no "precise final answer to your question", once it depends on a couple variables here.
C++ defines the standard library "interfaces" but it doesn't precisely define the underlying implementation. It means that getline() depends on the compiler you are using and the operating system as well. So you would have to test this comparing with other ways your operating system offers to you (example, using fopen() directly, or open() in Linux/Unix environments, or OpenFile() in Windows).
Reading binaries would probably make you software faster, once you don't spend time converting the strings to numbers. One (among many) motivations for glTF.
Reading binaries is easy. But you should have a "strategy" to parse it effiently to solve your specific problem. Considering that you are trying to parse a model, I would highly suggest you to look for glTF. In glTF you can have the vertices of your 3D model in a binary format in sequence ready to feed your VBO... :-)
Google for it and learn a little more.
About reading binary data in C++ is easy:
std::ifstream file("somedata.obj", ios::binary);
if (file)
{
char buffer[1024];
file.read(buffer, sizeof(buffer));
// buffer has you binary data... you can interpret it!
}
But again, this is not enough. You need to wisely interpret the data you just read.
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
answered Nov 12 at 3:50
Wagner Patriota
3,9681840
3,9681840
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53255477%2fhow-can-i-read-file-fastest-way-in-c-to-send-data-to-vertex-buffer-object%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
So, you have millions tiny files? I'm afraid it is impossible to do reading of files faster. If you concatenate all tiny files into a big one using some unique separator the program will work much faster.
– S.M.
Nov 12 at 3:31
Have you looked into using a memory mapped file?
– user4942583
Nov 12 at 3:46
If you have many small files and you're reading from a HDD rather than an SSD, you'll be waiting a lot for the head to be in the right place. Using AIO can help with this.
– o11c
Nov 12 at 4:10
thanks for answers guys i think i need to do another method like said wagner
– Emre Kaya
Nov 12 at 4:19