Hi, below some ideas to improve the service (my opinion)
- Buy unit by demand and not by month (for example, I want to run 100 spiders for 2 hours once. I don't want to buy 99 units for a month, i will use just 2 hours)
- Request option to check if exists a free unit (a way to request to API if there is a free unit to run my spider - because if I have 4 units and I'm using all of them to run spiders and want to run another one, could be a option to ask if there is a free unit. If there isn't my code wait for a free unit before request to run a new job. Or, for codes that users wait the return from spider, my system can show something like "all slots are busy")
1 Votes
2 Comments
Sorted by
T
Timposted
almost 5 years ago
We also would like to be able to scale the amount of units up or down via an API instead of manually via the interface. The billing can be per month, that's not the issue for us. But we need to be able to automate the scaling and that's not possible now.
0 Votes
nestorposted
almost 6 years ago
Admin
Thanks for the feedback.
Regarding 1) You run jobs that last less than a minute each, do you need to run 100 jobs at the same time? Otherwise what's the point on having 100 units?
Regarding 2) If you schedule 20 jobs and you only have 10 units, the remaining 10 will be queued under "Next jobs" and then Scrapy Cloud will automatically schedule the next job as soon as another is completed and there's a free unit available. In other words, you can keep scheduling jobs without having to check if there's free units, they will be queued and run as successively as soon as there's a free unit.
Hi, below some ideas to improve the service (my opinion)
- Buy unit by demand and not by month (for example, I want to run 100 spiders for 2 hours once. I don't want to buy 99 units for a month, i will use just 2 hours)
- Request option to check if exists a free unit (a way to request to API if there is a free unit to run my spider - because if I have 4 units and I'm using all of them to run spiders and want to run another one, could be a option to ask if there is a free unit. If there isn't my code wait for a free unit before request to run a new job. Or, for codes that users wait the return from spider, my system can show something like "all slots are busy")
1 Votes
2 Comments
Tim posted almost 5 years ago
We also would like to be able to scale the amount of units up or down via an API instead of manually via the interface. The billing can be per month, that's not the issue for us. But we need to be able to automate the scaling and that's not possible now.
0 Votes
nestor posted almost 6 years ago Admin
Thanks for the feedback.
Regarding 1) You run jobs that last less than a minute each, do you need to run 100 jobs at the same time? Otherwise what's the point on having 100 units?
Regarding 2) If you schedule 20 jobs and you only have 10 units, the remaining 10 will be queued under "Next jobs" and then Scrapy Cloud will automatically schedule the next job as soon as another is completed and there's a free unit available. In other words, you can keep scheduling jobs without having to check if there's free units, they will be queued and run as successively as soon as there's a free unit.
0 Votes
Login to post a comment