Using async functionalities of Spiders in Script

Posted almost 7 years ago by xavierdurandsmet

Post a topic
Un Answered
x
xavierdurandsmet

Hi,


I am using Scrapy, it works great.

I am migrating to scrapingHub.

I have a script that runs spiders and waits asynchronously for the spiders to finish, to start doing other actions.

How can I do this in scrapingHub?

Example:

# do some stuff

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
# do other stuff after the the spiders are done running
Thank you

0 Votes


0 Comments

Login to post a comment