how to read an extensive file without getting memory exception?
up vote
-1
down vote
favorite
I have a constructor for generate a Sparse Matrix after read a file.
The method works correctly if the sparse matrix resultant not are big.
Nevertheless, I want to read a big file (arround at 500.000 lines).
How to read an extensive file without getting memory exception?
The matrix it's declarate of this form: vector<vector<int>> matrix;
MatriuSparse::MatriuSparse(const string nomFitxer) {
fstream fitxer;
fitxer.open(nomFitxer);
int m_col;
int m_row;
int dimension = m_Nlinies;
while (!fitxer.eof())
{
fitxer >> m_row >> m_col;
if (dimension <= m_row|| dimension <= m_col) {
if (m_col < m_row) {
dimension = m_row;
dimension++;
}
if (m_row < m_col) {
dimension = m_col;
dimension++;
}
matrix.resize(dimension);
for (int i = 0; i < dimension; i++) {
matrix[i].resize(dimension);
}
matrix[m_row][m_col] = 1;
}
matrix[m_row][m_col] = 1;
}
m_Nlinies = dimension;
m_Ncol = m_Nlinies;
fitxer.close();
}
c++ vector sparse-matrix adjacency-matrix
asked Nov 11 at 0:47
desirée rodríguez
23
add a comment |
up vote
-1
down vote
favorite
I have a constructor for generate a Sparse Matrix after read a file.
The method works correctly if the sparse matrix resultant not are big.
Nevertheless, I want to read a big file (arround at 500.000 lines).
How to read an extensive file without getting memory exception?
The matrix it's declarate of this form: vector<vector<int>> matrix;
MatriuSparse::MatriuSparse(const string nomFitxer) {
fstream fitxer;
fitxer.open(nomFitxer);
int m_col;
int m_row;
int dimension = m_Nlinies;
while (!fitxer.eof())
{
fitxer >> m_row >> m_col;
if (dimension <= m_row|| dimension <= m_col) {
if (m_col < m_row) {
dimension = m_row;
dimension++;
}
if (m_row < m_col) {
dimension = m_col;
dimension++;
}
matrix.resize(dimension);
for (int i = 0; i < dimension; i++) {
matrix[i].resize(dimension);
}
matrix[m_row][m_col] = 1;
}
matrix[m_row][m_col] = 1;
}
m_Nlinies = dimension;
m_Ncol = m_Nlinies;
fitxer.close();
}
c++ vector sparse-matrix adjacency-matrix
asked Nov 11 at 0:47
desirée rodríguez
23
The problem is not the code that reads the file. The problem is that your matrix type stores all values of the matrix (even zeros). For large matrices, for example, 500'000 rows and columns, storing all values takes about 233 GiB. You may want to use a sparse matrix type to hold your matrix.
– pschill
Nov 11 at 0:59
How can I get it to not save the 0 values?
– desirée rodríguez
Nov 11 at 1:14
2
You use the term sparse matrix, but you do not seem to understand what one is. Start your research by familiarizing yourself. Unrelated Why is iostream::eof inside a loop condition considered wrong?
– user4581301
Nov 11 at 2:09
add a comment |
up vote
-1
down vote
favorite
up vote
-1
down vote
favorite
I have a constructor for generate a Sparse Matrix after read a file.
The method works correctly if the sparse matrix resultant not are big.
Nevertheless, I want to read a big file (arround at 500.000 lines).
How to read an extensive file without getting memory exception?
The matrix it's declarate of this form: vector<vector<int>> matrix;
MatriuSparse::MatriuSparse(const string nomFitxer) {
fstream fitxer;
fitxer.open(nomFitxer);
int m_col;
int m_row;
int dimension = m_Nlinies;
while (!fitxer.eof())
{
fitxer >> m_row >> m_col;
if (dimension <= m_row|| dimension <= m_col) {
if (m_col < m_row) {
dimension = m_row;
dimension++;
}
if (m_row < m_col) {
dimension = m_col;
dimension++;
}
matrix.resize(dimension);
for (int i = 0; i < dimension; i++) {
matrix[i].resize(dimension);
}
matrix[m_row][m_col] = 1;
}
matrix[m_row][m_col] = 1;
}
m_Nlinies = dimension;
m_Ncol = m_Nlinies;
fitxer.close();
}
c++ vector sparse-matrix adjacency-matrix
asked Nov 11 at 0:47
desirée rodríguez
23
I have a constructor for generate a Sparse Matrix after read a file.
The method works correctly if the sparse matrix resultant not are big.
Nevertheless, I want to read a big file (arround at 500.000 lines).
How to read an extensive file without getting memory exception?
The matrix it's declarate of this form: vector<vector<int>> matrix;
MatriuSparse::MatriuSparse(const string nomFitxer) {
fstream fitxer;
fitxer.open(nomFitxer);
int m_col;
int m_row;
int dimension = m_Nlinies;
while (!fitxer.eof())
{
fitxer >> m_row >> m_col;
if (dimension <= m_row|| dimension <= m_col) {
if (m_col < m_row) {
dimension = m_row;
dimension++;
}
if (m_row < m_col) {
dimension = m_col;
dimension++;
}
matrix.resize(dimension);
for (int i = 0; i < dimension; i++) {
matrix[i].resize(dimension);
}
matrix[m_row][m_col] = 1;
}
matrix[m_row][m_col] = 1;
}
m_Nlinies = dimension;
m_Ncol = m_Nlinies;
fitxer.close();
}
c++ vector sparse-matrix adjacency-matrix
c++ vector sparse-matrix adjacency-matrix
asked Nov 11 at 0:47
desirée rodríguez
23
asked Nov 11 at 0:47
desirée rodríguez
23
asked Nov 11 at 0:47
desirée rodríguez
23
asked Nov 11 at 0:47
desirée rodríguez
23
asked Nov 11 at 0:47
desirée rodríguez
23
23
The problem is not the code that reads the file. The problem is that your matrix type stores all values of the matrix (even zeros). For large matrices, for example, 500'000 rows and columns, storing all values takes about 233 GiB. You may want to use a sparse matrix type to hold your matrix.
– pschill
Nov 11 at 0:59
How can I get it to not save the 0 values?
– desirée rodríguez
Nov 11 at 1:14
2
You use the term sparse matrix, but you do not seem to understand what one is. Start your research by familiarizing yourself. Unrelated Why is iostream::eof inside a loop condition considered wrong?
– user4581301
Nov 11 at 2:09
add a comment |
The problem is not the code that reads the file. The problem is that your matrix type stores all values of the matrix (even zeros). For large matrices, for example, 500'000 rows and columns, storing all values takes about 233 GiB. You may want to use a sparse matrix type to hold your matrix.
– pschill
Nov 11 at 0:59
How can I get it to not save the 0 values?
– desirée rodríguez
Nov 11 at 1:14
2
You use the term sparse matrix, but you do not seem to understand what one is. Start your research by familiarizing yourself. Unrelated Why is iostream::eof inside a loop condition considered wrong?
– user4581301
Nov 11 at 2:09
The problem is not the code that reads the file. The problem is that your matrix type stores all values of the matrix (even zeros). For large matrices, for example, 500'000 rows and columns, storing all values takes about 233 GiB. You may want to use a sparse matrix type to hold your matrix.
– pschill
Nov 11 at 0:59
The problem is not the code that reads the file. The problem is that your matrix type stores all values of the matrix (even zeros). For large matrices, for example, 500'000 rows and columns, storing all values takes about 233 GiB. You may want to use a sparse matrix type to hold your matrix.
– pschill
Nov 11 at 0:59
How can I get it to not save the 0 values?
– desirée rodríguez
Nov 11 at 1:14
How can I get it to not save the 0 values?
– desirée rodríguez
Nov 11 at 1:14
2
2
You use the term sparse matrix, but you do not seem to understand what one is. Start your research by familiarizing yourself. Unrelated Why is iostream::eof inside a loop condition considered wrong?
– user4581301
Nov 11 at 2:09
You use the term sparse matrix, but you do not seem to understand what one is. Start your research by familiarizing yourself. Unrelated Why is iostream::eof inside a loop condition considered wrong?
– user4581301
Nov 11 at 2:09
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53244846%2fhow-to-read-an-extensive-file-without-getting-memory-exception%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
The problem is not the code that reads the file. The problem is that your matrix type stores all values of the matrix (even zeros). For large matrices, for example, 500'000 rows and columns, storing all values takes about 233 GiB. You may want to use a sparse matrix type to hold your matrix.
– pschill
Nov 11 at 0:59
How can I get it to not save the 0 values?
– desirée rodríguez
Nov 11 at 1:14
2
You use the term sparse matrix, but you do not seem to understand what one is. Start your research by familiarizing yourself. Unrelated Why is iostream::eof inside a loop condition considered wrong?
– user4581301
Nov 11 at 2:09