Newspaper library

As an absolute newbie on the topic of using python, I stumbled over a few difficulties using the newspaper library extension. My goal is to use the newspaper extension on a regular basis to download all new articles of a German news website called "tagesschau" and all articles from CNN to build a data stack I can analyze in a few years.
If I got it right I could use the following commands to download and scrape all articles into the python library.

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total

news_pool.join()`

If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?

Thanks for your help.

edited Nov 16 '18 at 9:23

SiHa

3,31961632

asked Nov 13 '18 at 21:02

Philipp Schulz

132

add a comment |

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total

news_pool.join()`

If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?

Thanks for your help.

edited Nov 16 '18 at 9:23

SiHa

3,31961632

asked Nov 13 '18 at 21:02

Philipp Schulz

132

add a comment |

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total

news_pool.join()`

If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?

Thanks for your help.

edited Nov 16 '18 at 9:23

SiHa

3,31961632

asked Nov 13 '18 at 21:02

Philipp Schulz

132

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total

news_pool.join()`

If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?

Thanks for your help.

python python-newspaper

edited Nov 16 '18 at 9:23

SiHa

3,31961632

asked Nov 13 '18 at 21:02

Philipp Schulz

132

edited Nov 16 '18 at 9:23

SiHa

3,31961632

asked Nov 13 '18 at 21:02

Philipp Schulz

132

edited Nov 16 '18 at 9:23

SiHa

3,31961632

edited Nov 16 '18 at 9:23

SiHa

3,31961632

edited Nov 16 '18 at 9:23

SiHa

3,31961632

asked Nov 13 '18 at 21:02

Philipp Schulz

132

asked Nov 13 '18 at 21:02

Philipp Schulz

132

asked Nov 13 '18 at 21:02

Philipp Schulz

132

add a comment |

2 Answers
2

active

oldest

votes

The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2)

news_pool.join()



for i in range (tagesschau_paper.size()): 

    with open("tagesschau_paper{}.html".format(i), "w") as file:

    file.write(tagesschau_paper.articles[i].html)

Note: news_pool doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size(), it results to 0. You have to import and use Source instead.

The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.

edited Nov 16 '18 at 8:57

answered Nov 16 '18 at 8:49

Mrinal Roy

713

Thanks for youre help :)

– Philipp Schulz
Nov 16 '18 at 11:19

I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`

– Philipp Schulz
Nov 18 '18 at 11:53

Thanks for the update. I didn't need to set the encoding.

– Mrinal Roy
Nov 21 '18 at 2:20

add a comment |

-1

You can use pickle to save objects outside of python and reopen them later:

file_Name = "testfile"

# open the file for writing

fileObject = open(file_Name,'wb') 



# this writes the object news_pool to the

# file named 'testfile'

pickle.dump(news_pool,fileObject)   



# here we close the fileObject

fileObject.close()

# we open the file for reading

fileObject = open(file_Name,'r')  

# load the object from the file into var news_pool_reopen

news_pool_reopen = pickle.load(fileObject)

answered Nov 13 '18 at 22:41

Turtalicious

524

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of news_pool changes (which seems likely given the time span OP specified and the fact that newspaper seems to be in active development), then pickle.load will likely fail. See, for example, this explanation.

– Paul Brodersen
Nov 13 '18 at 23:07

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53289440%2fnewspaper-library%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2)

news_pool.join()



for i in range (tagesschau_paper.size()): 

    with open("tagesschau_paper{}.html".format(i), "w") as file:

    file.write(tagesschau_paper.articles[i].html)

Note: news_pool doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size(), it results to 0. You have to import and use Source instead.

The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.

edited Nov 16 '18 at 8:57

answered Nov 16 '18 at 8:49

Mrinal Roy

713

Thanks for youre help :)

– Philipp Schulz
Nov 16 '18 at 11:19

I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`

– Philipp Schulz
Nov 18 '18 at 11:53

Thanks for the update. I didn't need to set the encoding.

– Mrinal Roy
Nov 21 '18 at 2:20

add a comment |

The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2)

news_pool.join()



for i in range (tagesschau_paper.size()): 

    with open("tagesschau_paper{}.html".format(i), "w") as file:

    file.write(tagesschau_paper.articles[i].html)

Note: news_pool doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size(), it results to 0. You have to import and use Source instead.

The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.

edited Nov 16 '18 at 8:57

answered Nov 16 '18 at 8:49

Mrinal Roy

713

Thanks for youre help :)

– Philipp Schulz
Nov 16 '18 at 11:19

I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`

– Philipp Schulz
Nov 18 '18 at 11:53

Thanks for the update. I didn't need to set the encoding.

– Mrinal Roy
Nov 21 '18 at 2:20

add a comment |

The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2)

news_pool.join()



for i in range (tagesschau_paper.size()): 

    with open("tagesschau_paper{}.html".format(i), "w") as file:

    file.write(tagesschau_paper.articles[i].html)

Note: news_pool doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size(), it results to 0. You have to import and use Source instead.

The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.

edited Nov 16 '18 at 8:57

answered Nov 16 '18 at 8:49

Mrinal Roy

713

The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....

import newspaper

from newspaper import news_pool



tagesschau_paper = newspaper.build('http://tagesschau.de')

cnn_paper = newspaper.build('http://cnn.com')



papers = [tagesschau_paper, cnn_paper]

news_pool.set(papers, threads_per_source=2)

news_pool.join()



for i in range (tagesschau_paper.size()): 

    with open("tagesschau_paper{}.html".format(i), "w") as file:

    file.write(tagesschau_paper.articles[i].html)

Note: news_pool doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size(), it results to 0. You have to import and use Source instead.

The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.

edited Nov 16 '18 at 8:57

answered Nov 16 '18 at 8:49

Mrinal Roy

713

edited Nov 16 '18 at 8:57

answered Nov 16 '18 at 8:49

Mrinal Roy

713

answered Nov 16 '18 at 8:49

Mrinal Roy

713

answered Nov 16 '18 at 8:49

Mrinal Roy

713

Thanks for youre help :)

– Philipp Schulz
Nov 16 '18 at 11:19

I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`

– Philipp Schulz
Nov 18 '18 at 11:53

Thanks for the update. I didn't need to set the encoding.

– Mrinal Roy
Nov 21 '18 at 2:20

add a comment |

Thanks for youre help :)

– Philipp Schulz
Nov 16 '18 at 11:19

I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`

– Philipp Schulz
Nov 18 '18 at 11:53

Thanks for the update. I didn't need to set the encoding.

– Mrinal Roy
Nov 21 '18 at 2:20

Thanks for youre help :)

– Philipp Schulz
Nov 16 '18 at 11:19

I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`

– Philipp Schulz
Nov 18 '18 at 11:53

Thanks for the update. I didn't need to set the encoding.

– Mrinal Roy
Nov 21 '18 at 2:20

add a comment |

-1

You can use pickle to save objects outside of python and reopen them later:

file_Name = "testfile"

# open the file for writing

fileObject = open(file_Name,'wb') 



# this writes the object news_pool to the

# file named 'testfile'

pickle.dump(news_pool,fileObject)   



# here we close the fileObject

fileObject.close()

# we open the file for reading

fileObject = open(file_Name,'r')  

# load the object from the file into var news_pool_reopen

news_pool_reopen = pickle.load(fileObject)

answered Nov 13 '18 at 22:41

Turtalicious

524

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of news_pool changes (which seems likely given the time span OP specified and the fact that newspaper seems to be in active development), then pickle.load will likely fail. See, for example, this explanation.

– Paul Brodersen
Nov 13 '18 at 23:07

add a comment |

-1

You can use pickle to save objects outside of python and reopen them later:

file_Name = "testfile"

# open the file for writing

fileObject = open(file_Name,'wb') 



# this writes the object news_pool to the

# file named 'testfile'

pickle.dump(news_pool,fileObject)   



# here we close the fileObject

fileObject.close()

# we open the file for reading

fileObject = open(file_Name,'r')  

# load the object from the file into var news_pool_reopen

news_pool_reopen = pickle.load(fileObject)

answered Nov 13 '18 at 22:41

Turtalicious

524

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of news_pool changes (which seems likely given the time span OP specified and the fact that newspaper seems to be in active development), then pickle.load will likely fail. See, for example, this explanation.

– Paul Brodersen
Nov 13 '18 at 23:07

add a comment |

-1

You can use pickle to save objects outside of python and reopen them later:

file_Name = "testfile"

# open the file for writing

fileObject = open(file_Name,'wb') 



# this writes the object news_pool to the

# file named 'testfile'

pickle.dump(news_pool,fileObject)   



# here we close the fileObject

fileObject.close()

# we open the file for reading

fileObject = open(file_Name,'r')  

# load the object from the file into var news_pool_reopen

news_pool_reopen = pickle.load(fileObject)

answered Nov 13 '18 at 22:41

Turtalicious

524

You can use pickle to save objects outside of python and reopen them later:

file_Name = "testfile"

# open the file for writing

fileObject = open(file_Name,'wb') 



# this writes the object news_pool to the

# file named 'testfile'

pickle.dump(news_pool,fileObject)   



# here we close the fileObject

fileObject.close()

# we open the file for reading

fileObject = open(file_Name,'r')  

# load the object from the file into var news_pool_reopen

news_pool_reopen = pickle.load(fileObject)

answered Nov 13 '18 at 22:41

Turtalicious

524

answered Nov 13 '18 at 22:41

Turtalicious

524

answered Nov 13 '18 at 22:41

Turtalicious

524

answered Nov 13 '18 at 22:41

Turtalicious

524

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of news_pool changes (which seems likely given the time span OP specified and the fact that newspaper seems to be in active development), then pickle.load will likely fail. See, for example, this explanation.

– Paul Brodersen
Nov 13 '18 at 23:07

add a comment |

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of news_pool changes (which seems likely given the time span OP specified and the fact that newspaper seems to be in active development), then pickle.load will likely fail. See, for example, this explanation.

– Paul Brodersen
Nov 13 '18 at 23:07

Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of news_pool changes (which seems likely given the time span OP specified and the fact that newspaper seems to be in active development), then pickle.load will likely fail. See, for example, this explanation.

– Paul Brodersen
Nov 13 '18 at 23:07

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Ndtyjky