Crawlera is working quite slow, it's processing a single request in average 6 to 8 seconds. I'm thiking of making concurrent sessions. I'm allowed 10 sessions in my c10 plan. My question is how can I make concurrent sessions in pythons and will it increase the speed because 8 seconds average is very time consuming. I need to parse thousands of result.
Best Answer
n
nestor
said
over 6 years ago
You could try using grequests, or better yet you can always try Scrapy. It's as simple as setting CONCURRENT_REQUESTS = ##.
1 Comment
nestor
said
over 6 years ago
Answer
You could try using grequests, or better yet you can always try Scrapy. It's as simple as setting CONCURRENT_REQUESTS = ##.
adanish
Crawlera is working quite slow, it's processing a single request in average 6 to 8 seconds. I'm thiking of making concurrent sessions. I'm allowed 10 sessions in my c10 plan. My question is how can I make concurrent sessions in pythons and will it increase the speed because 8 seconds average is very time consuming. I need to parse thousands of result.
You could try using grequests, or better yet you can always try Scrapy. It's as simple as setting CONCURRENT_REQUESTS = ##.
nestor
You could try using grequests, or better yet you can always try Scrapy. It's as simple as setting CONCURRENT_REQUESTS = ##.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics