Beautiful Soup (code was working) now only returns certain tags
I was scraping this website a few weeks back but some of the code no longer works.
This code returns everything as expected:
from bs4 import BeautifulSoup
import requests
url = 'https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/'
res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml")
soup.find_all('div', {'class': 'hUMQK _3JPYB'})
However, when doing a find all on the code below it now returns an empty list and it used to work. It appears nothing has changed in the web pages HTML
soup.find_all('div', {'class': '_3h0tU'})
HTML image of 'div' tag
python html web-scraping beautifulsoup lxml
asked Nov 13 '18 at 22:09
slingoslingo
32
add a comment |
I was scraping this website a few weeks back but some of the code no longer works.
This code returns everything as expected:
from bs4 import BeautifulSoup
import requests
url = 'https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/'
res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml")
soup.find_all('div', {'class': 'hUMQK _3JPYB'})
However, when doing a find all on the code below it now returns an empty list and it used to work. It appears nothing has changed in the web pages HTML
soup.find_all('div', {'class': '_3h0tU'})
HTML image of 'div' tag
python html web-scraping beautifulsoup lxml
asked Nov 13 '18 at 22:09
slingoslingo
32
So what is your actual question please?
– QHarr
Nov 13 '18 at 22:19
Why does the last line of code return an empty list? The tag appears to still be on the site, and it worked perfectly fine a couple of weeks ago.
– slingo
Nov 13 '18 at 22:21
Hmmm yes I think that could be the problem. When I load the page you can see a delay in which items are populated. The items that still load fine populate right away and all the items that no longer work there is a visible delay when loaded. Thank you
– slingo
Nov 13 '18 at 22:32
add a comment |
I was scraping this website a few weeks back but some of the code no longer works.
This code returns everything as expected:
from bs4 import BeautifulSoup
import requests
url = 'https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/'
res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml")
soup.find_all('div', {'class': 'hUMQK _3JPYB'})
However, when doing a find all on the code below it now returns an empty list and it used to work. It appears nothing has changed in the web pages HTML
soup.find_all('div', {'class': '_3h0tU'})
HTML image of 'div' tag
python html web-scraping beautifulsoup lxml
asked Nov 13 '18 at 22:09
slingoslingo
32
I was scraping this website a few weeks back but some of the code no longer works.
This code returns everything as expected:
from bs4 import BeautifulSoup
import requests
url = 'https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/'
res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml")
soup.find_all('div', {'class': 'hUMQK _3JPYB'})
However, when doing a find all on the code below it now returns an empty list and it used to work. It appears nothing has changed in the web pages HTML
soup.find_all('div', {'class': '_3h0tU'})
HTML image of 'div' tag
python html web-scraping beautifulsoup lxml
python html web-scraping beautifulsoup lxml
asked Nov 13 '18 at 22:09
slingoslingo
32
asked Nov 13 '18 at 22:09
slingoslingo
32
edited Nov 13 '18 at 22:15
slingo
asked Nov 13 '18 at 22:09
slingoslingo
32
asked Nov 13 '18 at 22:09
slingoslingo
32
asked Nov 13 '18 at 22:09
slingoslingo
32
32
So what is your actual question please?
– QHarr
Nov 13 '18 at 22:19
Why does the last line of code return an empty list? The tag appears to still be on the site, and it worked perfectly fine a couple of weeks ago.
– slingo
Nov 13 '18 at 22:21
Hmmm yes I think that could be the problem. When I load the page you can see a delay in which items are populated. The items that still load fine populate right away and all the items that no longer work there is a visible delay when loaded. Thank you
– slingo
Nov 13 '18 at 22:32
add a comment |
So what is your actual question please?
– QHarr
Nov 13 '18 at 22:19
Why does the last line of code return an empty list? The tag appears to still be on the site, and it worked perfectly fine a couple of weeks ago.
– slingo
Nov 13 '18 at 22:21
Hmmm yes I think that could be the problem. When I load the page you can see a delay in which items are populated. The items that still load fine populate right away and all the items that no longer work there is a visible delay when loaded. Thank you
– slingo
Nov 13 '18 at 22:32
So what is your actual question please?
– QHarr
Nov 13 '18 at 22:19
So what is your actual question please?
– QHarr
Nov 13 '18 at 22:19
Why does the last line of code return an empty list? The tag appears to still be on the site, and it worked perfectly fine a couple of weeks ago.
– slingo
Nov 13 '18 at 22:21
Why does the last line of code return an empty list? The tag appears to still be on the site, and it worked perfectly fine a couple of weeks ago.
– slingo
Nov 13 '18 at 22:21
Hmmm yes I think that could be the problem. When I load the page you can see a delay in which items are populated. The items that still load fine populate right away and all the items that no longer work there is a visible delay when loaded. Thank you
– slingo
Nov 13 '18 at 22:32
Hmmm yes I think that could be the problem. When I load the page you can see a delay in which items are populated. The items that still load fine populate right away and all the items that no longer work there is a visible delay when loaded. Thank you
– slingo
Nov 13 '18 at 22:32
add a comment |
1 Answer
1
active
oldest
votes
As
I don't know if website changed how populated but works if time is allowed for page to load. Using selenium retrieves just fine. Seems likely that content is JS loaded.
from selenium import webdriver
URL = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/"
d = WebDriver.Chrome()
d.get(URL)
for item in d.find_elements_by_css_selector("._3h0tU"):
Print(item.Text)
d.quit()
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290255%2fbeautiful-soup-code-was-working-now-only-returns-certain-tags%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
As
I don't know if website changed how populated but works if time is allowed for page to load. Using selenium retrieves just fine. Seems likely that content is JS loaded.
from selenium import webdriver
URL = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/"
d = WebDriver.Chrome()
d.get(URL)
for item in d.find_elements_by_css_selector("._3h0tU"):
Print(item.Text)
d.quit()
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
add a comment |
As
I don't know if website changed how populated but works if time is allowed for page to load. Using selenium retrieves just fine. Seems likely that content is JS loaded.
from selenium import webdriver
URL = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/"
d = WebDriver.Chrome()
d.get(URL)
for item in d.find_elements_by_css_selector("._3h0tU"):
Print(item.Text)
d.quit()
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
add a comment |
As
I don't know if website changed how populated but works if time is allowed for page to load. Using selenium retrieves just fine. Seems likely that content is JS loaded.
from selenium import webdriver
URL = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/"
d = WebDriver.Chrome()
d.get(URL)
for item in d.find_elements_by_css_selector("._3h0tU"):
Print(item.Text)
d.quit()
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
As
I don't know if website changed how populated but works if time is allowed for page to load. Using selenium retrieves just fine. Seems likely that content is JS loaded.
from selenium import webdriver
URL = "https://www.sportsbookreview.com/betting-odds/nfl-football/consensus/"
d = WebDriver.Chrome()
d.get(URL)
for item in d.find_elements_by_css_selector("._3h0tU"):
Print(item.Text)
d.quit()
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
edited Nov 13 '18 at 22:37
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
answered Nov 13 '18 at 22:29
QHarrQHarr
31.8k82042
31.8k82042
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290255%2fbeautiful-soup-code-was-working-now-only-returns-certain-tags%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
So what is your actual question please?
– QHarr
Nov 13 '18 at 22:19
Why does the last line of code return an empty list? The tag appears to still be on the site, and it worked perfectly fine a couple of weeks ago.
– slingo
Nov 13 '18 at 22:21
Hmmm yes I think that could be the problem. When I load the page you can see a delay in which items are populated. The items that still load fine populate right away and all the items that no longer work there is a visible delay when loaded. Thank you
– slingo
Nov 13 '18 at 22:32