CRAWL QUESTION: a question abount requests
I am trying to set up random proxies when using the request, and I have encountered some problems.this is my code:
import requests
import random
pool = ['220.186.175.252:4216','106.110.39.106:4232']
proxy={'https':random.choice(pool)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
and the result:
{"origin": "112.10.164.203"}
it doesn't work, so I try to change my proxy, I thought maybe it use http instead of https, I change proxy to this:
proxy={'https':random.choice(pool)}
and unfortunately I get a ERROR:
requests.exceptions.ProxyError: HTTPConnectionPool(host='106.110.39.106', port=4232): Max retries exceeded with url: http://httpbin.org/ip (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
SO, I have two questions:
1.How to set the random proxy in requests
2.why i change the proxy get this ERROR
i am so appricate if you can solve my problems!
python-requests
asked Nov 12 '18 at 15:06
苏世杰
253
add a comment |
I am trying to set up random proxies when using the request, and I have encountered some problems.this is my code:
import requests
import random
pool = ['220.186.175.252:4216','106.110.39.106:4232']
proxy={'https':random.choice(pool)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
and the result:
{"origin": "112.10.164.203"}
it doesn't work, so I try to change my proxy, I thought maybe it use http instead of https, I change proxy to this:
proxy={'https':random.choice(pool)}
and unfortunately I get a ERROR:
requests.exceptions.ProxyError: HTTPConnectionPool(host='106.110.39.106', port=4232): Max retries exceeded with url: http://httpbin.org/ip (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
SO, I have two questions:
1.How to set the random proxy in requests
2.why i change the proxy get this ERROR
i am so appricate if you can solve my problems!
python-requests
asked Nov 12 '18 at 15:06
苏世杰
253
add a comment |
I am trying to set up random proxies when using the request, and I have encountered some problems.this is my code:
import requests
import random
pool = ['220.186.175.252:4216','106.110.39.106:4232']
proxy={'https':random.choice(pool)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
and the result:
{"origin": "112.10.164.203"}
it doesn't work, so I try to change my proxy, I thought maybe it use http instead of https, I change proxy to this:
proxy={'https':random.choice(pool)}
and unfortunately I get a ERROR:
requests.exceptions.ProxyError: HTTPConnectionPool(host='106.110.39.106', port=4232): Max retries exceeded with url: http://httpbin.org/ip (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
SO, I have two questions:
1.How to set the random proxy in requests
2.why i change the proxy get this ERROR
i am so appricate if you can solve my problems!
python-requests
asked Nov 12 '18 at 15:06
苏世杰
253
I am trying to set up random proxies when using the request, and I have encountered some problems.this is my code:
import requests
import random
pool = ['220.186.175.252:4216','106.110.39.106:4232']
proxy={'https':random.choice(pool)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
and the result:
{"origin": "112.10.164.203"}
it doesn't work, so I try to change my proxy, I thought maybe it use http instead of https, I change proxy to this:
proxy={'https':random.choice(pool)}
and unfortunately I get a ERROR:
requests.exceptions.ProxyError: HTTPConnectionPool(host='106.110.39.106', port=4232): Max retries exceeded with url: http://httpbin.org/ip (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))
SO, I have two questions:
1.How to set the random proxy in requests
2.why i change the proxy get this ERROR
i am so appricate if you can solve my problems!
python-requests
python-requests
asked Nov 12 '18 at 15:06
苏世杰
253
asked Nov 12 '18 at 15:06
苏世杰
253
asked Nov 12 '18 at 15:06
苏世杰
253
asked Nov 12 '18 at 15:06
苏世杰
253
asked Nov 12 '18 at 15:06
苏世杰
253
253
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You did right, why you got this error is that your proxy does not support http request. You need to know what kind of protocol does it support before you use it. free-proxy-list
And this is the way i define a random proxy
import requests
import random
https = ['220.186.175.252:4216','106.110.39.106:4232']
http = ["169.50.180.250:3128"]
proxy={'https':random.choice(https),"http":random.choice(http)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53264928%2fcrawl-question-a-question-abount-requests%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You did right, why you got this error is that your proxy does not support http request. You need to know what kind of protocol does it support before you use it. free-proxy-list
And this is the way i define a random proxy
import requests
import random
https = ['220.186.175.252:4216','106.110.39.106:4232']
http = ["169.50.180.250:3128"]
proxy={'https':random.choice(https),"http":random.choice(http)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
add a comment |
You did right, why you got this error is that your proxy does not support http request. You need to know what kind of protocol does it support before you use it. free-proxy-list
And this is the way i define a random proxy
import requests
import random
https = ['220.186.175.252:4216','106.110.39.106:4232']
http = ["169.50.180.250:3128"]
proxy={'https':random.choice(https),"http":random.choice(http)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
add a comment |
You did right, why you got this error is that your proxy does not support http request. You need to know what kind of protocol does it support before you use it. free-proxy-list
And this is the way i define a random proxy
import requests
import random
https = ['220.186.175.252:4216','106.110.39.106:4232']
http = ["169.50.180.250:3128"]
proxy={'https':random.choice(https),"http":random.choice(http)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
You did right, why you got this error is that your proxy does not support http request. You need to know what kind of protocol does it support before you use it. free-proxy-list
And this is the way i define a random proxy
import requests
import random
https = ['220.186.175.252:4216','106.110.39.106:4232']
http = ["169.50.180.250:3128"]
proxy={'https':random.choice(https),"http":random.choice(http)}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
test_url = "http://httpbin.org/ip" # a url test ip
response = requests.get(url=test_url,headers=headers,proxies=proxy)
text = response.text
print(text)
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
answered Nov 13 '18 at 7:30
kcorlidy
2,1702318
2,1702318
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
add a comment |
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
thank you kocrlidy,I also found the same problem yesterday, so I try to get some proxies from different ways.and I found the proxies I used before lack of effectiveness, the problem is not on me,but on the agent.
– 苏世杰
Nov 14 '18 at 8:06
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53264928%2fcrawl-question-a-question-abount-requests%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
