Deploy Project and run Spiders with settings of different environments

Modified on Wed, 3 Feb, 2021 at 7:59 AM


There are times when we want to test spiders with different configurations/environments but have to deploy spiders to different projects and then test the spiders in respective projects. But the recent SHUB release (v2.8.2) has made it possible to test multiple environments in single project. To make use of this feature, you can have different settings.py with required configurations and then while scheduling the spider provide the location of settings that has to be used. 


Please follow below steps to deploy project with different settings:


- Ensure that recent version of SHUB is installed. You can upgrade SHUB using "pip install --upgrade shub"
- Then schedule job with a flag  "-e SCRAPY_SETTINGS_MODULE=<settings-module-path>", like

shub schedule localinfo -e SCRAPY_SETTINGS_MODULE=testspiders.settings


For any further queries feel free to contact Support.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article