Scenario: You have e.g. a MongoDB server that you would like your spiders to write to, and you would like to open access to that server only from Scrapy Cloud IPs.
Unfortunately, this is not possible. We cannot provide you a reliable range of IP addresses for our Scrapy Cloud crawling servers, because they're not static, they change frequently. So, even if we were to provide you the list we have now, it will soon change and your spider's connection with Mongo will break.
Here are a couple alternatives to consider:
- Write a script that pulls the data from Zyte (using the API) and writes it to your Mongo server. This script can run in your mongo server or any server (and you only need to whitelist that IP)
- Use authentication in Mongo
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article