Queuing Jobs and providing scheduling, queuing in API
l
leonchughes
started a topic
almost 4 years ago
I am working on a service to provide a spider I have written and uploaded to the ScrapingHub service. At the moment I am on the free tier, but when the service launches I do plan to make use of the paid service.
As I am currently in development I am on the free tier. I therefore have 1 unit.
I am able to communicate with the service through the API.
When I run a spider, I get a 200 response from the service and a job id is generated. When I run the same spider again straight away, I receive a 400 response with 'spider already scheduled'.
I understand this is as there is a spider already running, but would it be better to implement a queuing system rather than giving the 400 response? Therefore when the currently running spider completes, the next in the queue is executed?
I also need to do this through the API, not through the admin interface as seen with the paid tier 'scheduling' option.
Is this an option or something that will be available?
leonchughes
I am working on a service to provide a spider I have written and uploaded to the ScrapingHub service. At the moment I am on the free tier, but when the service launches I do plan to make use of the paid service.
As I am currently in development I am on the free tier. I therefore have 1 unit.
I am able to communicate with the service through the API.
When I run a spider, I get a 200 response from the service and a job id is generated. When I run the same spider again straight away, I receive a 400 response with 'spider already scheduled'.
I understand this is as there is a spider already running, but would it be better to implement a queuing system rather than giving the 400 response? Therefore when the currently running spider completes, the next in the queue is executed?
I also need to do this through the API, not through the admin interface as seen with the paid tier 'scheduling' option.
Is this an option or something that will be available?
1 person has this question