Hi, my scrapy project at scrapy cloud has many spiders, I have recently bought smart proxies mainly for one spider, and I have added proxy-related settings in the spider's custom_settings variable so that it applied only to that spider and not the whole project, it is working as expected locally but in the cloud, it seems proxies are automatically applied to the whole project which is causing problems for my previous spiders. Please help me with how can I use smart proxies for the specific spiders at scrapy cloud. Thanks.
0 Votes
1 Comments
Bala Ravichandranposted
about 2 years ago
Admin
Hi Ali, Could you please share the settings.py and the custom_settings configuration for us to check?
Note: Please remove any sensitive information while sharing.
Hi, my scrapy project at scrapy cloud has many spiders, I have recently bought smart proxies mainly for one spider, and I have added proxy-related settings in the spider's custom_settings variable so that it applied only to that spider and not the whole project, it is working as expected locally but in the cloud, it seems proxies are automatically applied to the whole project which is causing problems for my previous spiders. Please help me with how can I use smart proxies for the specific spiders at scrapy cloud. Thanks.
0 Votes
1 Comments
Bala Ravichandran posted about 2 years ago Admin
Hi Ali, Could you please share the settings.py and the custom_settings configuration for us to check?
Note: Please remove any sensitive information while sharing.
0 Votes
Login to post a comment