videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Concurrent sessions in Python

Crawlera is working quite slow, it's processing a single request in average 6  to 8 seconds. I'm thiking of making concurrent sessions. I'm allowed 10 sessions in my c10 plan. My question is how can I make concurrent sessions in pythons and will it increase the speed because 8 seconds average is very time consuming. I need to parse thousands of result.


Best Answer

You could try using grequests, or better yet you can always try Scrapy. It's as simple as setting CONCURRENT_REQUESTS = ##.

1 Comment

Answer

You could try using grequests, or better yet you can always try Scrapy. It's as simple as setting CONCURRENT_REQUESTS = ##.

Login to post a comment