Easy Ways to Run Multiple Spiders?

Posted over 6 years ago by Grimmjam

Post a topic

Un Answered

Grimmjam

From what I can tell, you can't just check the box next to all of your spiders and hit a run button (unless I'm missing it). What are some easy ways to run all spiders for a project?

Hitting "Run" and making sure I manually pick each spider in the drop-down list is annoying when you have many spiders. Right now, I just view each spider in a new tab and hit Run on each one. Any better ideas with the dashboard? Or am I going to have to work with the API for this?

2 Votes

3 Comments

Grimmjam posted over 6 years ago

I would like to add that manually choosing spiders from the drop-down list stops working after about 9 spiders or so, so then you have to do it again and make sure you don't pick the same spiders as before. If it kept going until you actually had no spiders remaining, it would be a fair option.

0 Votes

Shadi Akiki posted about 6 years ago

Hello. Any update on this?

0 Votes

nestor posted about 6 years ago Admin

The UI uses the API to schedule the jobs, so why not use the API if you need to run multiple spiders?

Alternatively, you can use Periodic Jobs: https://support.scrapinghub.com/support/solutions/articles/22000200419-scheduling-periodic-jobs

0 Votes