toysrus.com seems to be expecting some headers like Accept and Accept-Encoding, make sure to include those in your CURL request.
s
stlproinc
said
over 6 years ago
Thanks nestor! I'm pretty new to this. Can you briefly explain how you discovered that? Was it trial and error or something else that I should have noticed? Or, if you don't have the time, is there resource you would recommend to get me up to speed on diagnosing things like this?
s
stlproinc
said
over 6 years ago
Just for anyone else that runs into this, the following curl command worked:
Sorry for replying so late here, I simply checked the request headers sent to the website in my browser's developer tools and added them to the curl command.
stlproinc
I just set up my Crawlera account and was testing the setup. I ran the commands
curl -x proxy.crawlera.com:8010 -U <API_KEY>: --cacert crawlera-ca.crt https://www.toysrus.com curl -x proxy.crawlera.com:8010 -U <API_KEY>: --insecure https://www.toysrus.com
Both commands timed out with the error message "Timeout processing HTTP stream". I did the following command and it worked:
curl -x proxy.crawlera.com:8010 -U <API_KEY>: --cacert crawlera-ca.crt https://www.amazon.com
Do I have something set up incorrectly or is toysrus.com just denying access?
Thanks in advance
Hi,
toysrus.com seems to be expecting some headers like Accept and Accept-Encoding, make sure to include those in your CURL request.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
Hi,
toysrus.com seems to be expecting some headers like Accept and Accept-Encoding, make sure to include those in your CURL request.
stlproinc
Thanks nestor! I'm pretty new to this. Can you briefly explain how you discovered that? Was it trial and error or something else that I should have noticed? Or, if you don't have the time, is there resource you would recommend to get me up to speed on diagnosing things like this?
stlproinc
Just for anyone else that runs into this, the following curl command worked:
joevu-stlpro
For more info
1 person likes this
nestor
Sorry for replying so late here, I simply checked the request headers sent to the website in my browser's developer tools and added them to the curl command.
1 person likes this
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 389 topics