My spider works properly when I test it on my my local machine.
When I run my spider on Scrapy Cloud, the site returns a page meant for a different region. This could be due to the IP address presented by Scrapy Cloud.
How could I make the spider run on Scrapy Cloud and fetch the page as if it was fetched for USA or India, Geo Location?
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
Mayuresh Phadke
I am scraping the site https://www.primevideo.com/storefront/movie/ref=dv_web_nav_sf_movies
My spider works properly when I test it on my my local machine.
When I run my spider on Scrapy Cloud, the site returns a page meant for a different region. This could be due to the IP address presented by Scrapy Cloud.
How could I make the spider run on Scrapy Cloud and fetch the page as if it was fetched for USA or India, Geo Location?
Regards,
Mayuresh
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
To subscribe to Crawlera please see: https://support.scrapinghub.com/support/solutions/articles/22000188411-getting-started-with-crawlera
To restrict an account to a specific location please see: https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region
nestor
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
To subscribe to Crawlera please see: https://support.scrapinghub.com/support/solutions/articles/22000188411-getting-started-with-crawlera
To restrict an account to a specific location please see: https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 453 topics