My spider works properly when I test it on my my local machine.
When I run my spider on Scrapy Cloud, the site returns a page meant for a different region. This could be due to the IP address presented by Scrapy Cloud.
How could I make the spider run on Scrapy Cloud and fetch the page as if it was fetched for USA or India, Geo Location?
Nestor Toledo posted
almost 9 years ago
AdminBest Answer
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
Nestor Toledoposted
almost 9 years ago
AdminAnswer
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
I am scraping the site https://www.primevideo.com/storefront/movie/ref=dv_web_nav_sf_movies
My spider works properly when I test it on my my local machine.
When I run my spider on Scrapy Cloud, the site returns a page meant for a different region. This could be due to the IP address presented by Scrapy Cloud.
How could I make the spider run on Scrapy Cloud and fetch the page as if it was fetched for USA or India, Geo Location?
Regards,
Mayuresh
Attachments (1)
primevideo.py
1.52 KB
0 Votes
Nestor Toledo posted almost 9 years ago Admin Best Answer
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
To subscribe to Crawlera please see: https://support.scrapinghub.com/support/solutions/articles/22000188411-getting-started-with-crawlera
To restrict an account to a specific location please see: https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region
0 Votes
1 Comments
Nestor Toledo posted almost 9 years ago Admin Answer
If the website is IP dependent, you'll need to subscribe to Crawlera and setup a location restricted account, our Scrapy Cloud servers are based in Germany.
Crawlera plans start at $25,00/month for 150,000 monthly requests.
To subscribe to Crawlera please see: https://support.scrapinghub.com/support/solutions/articles/22000188411-getting-started-with-crawlera
To restrict an account to a specific location please see: https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region
0 Votes
Login to post a comment