3 things to evaluate to discover the best way for you to access web data.Register for the webinar
Start a new topic

How to use smart proxies only for the specific spiders, instead for the whole project at scrapy cloud?

Hi, my scrapy project at scrapy cloud has many spiders, I have recently bought smart proxies mainly for one spider, and I have added proxy-related settings in the spider's custom_settings variable so that it applied only to that spider and not the whole project, it is working as expected locally but in the cloud, it seems proxies are automatically applied to the whole project which is causing problems for my previous spiders. Please help me with how can I use smart proxies for the specific spiders at scrapy cloud. Thanks.

1 Comment

Hi Ali, Could you please share the settings.py and the custom_settings configuration for us to check?

Note: Please remove any sensitive information while sharing.

Login to post a comment