My spider works properly when I test it on my my local machine.
When I run my spider on Scrapy Cloud, the site returns a page meant for a different region. This could be due to the IP address presented by Scrapy Cloud.
How could I make the spider run on Scrapy Cloud and fetch the page as if it was fetched for USA or India, Geo Location?
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
I am scraping the site https://www.primevideo.com/storefront/movie/ref=dv_web_nav_sf_movies
My spider works properly when I test it on my my local machine.
When I run my spider on Scrapy Cloud, the site returns a page meant for a different region. This could be due to the IP address presented by Scrapy Cloud.
How could I make the spider run on Scrapy Cloud and fetch the page as if it was fetched for USA or India, Geo Location?
Regards,
Mayuresh
Attachments (1)
primevideo.py
1.52 KB
0 Votes
nestor posted over 7 years ago Admin Best Answer
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
To subscribe to Crawlera please see: https://support.scrapinghub.com/support/solutions/articles/22000188411-getting-started-with-crawlera
To restrict an account to a specific location please see: https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region
0 Votes
1 Comments
nestor posted over 7 years ago Admin Answer
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
To subscribe to Crawlera please see: https://support.scrapinghub.com/support/solutions/articles/22000188411-getting-started-with-crawlera
To restrict an account to a specific location please see: https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region
0 Votes
Login to post a comment