No recent searches
Popular Articles
Sorry! nothing found for
Posted almost 5 years ago by xiaobie
Hello. Could I ask a question about periodic jobs of Scrapinghub? Will the scraped duplicated data be automatically removed by Scrapinghub?
0 Votes
nestor posted almost 5 years ago Admin Best Answer
Scrapinghub doesn't remove duplicated data automatically. I would suggest you try out DeltaFetch to avoid crawling items that were crawled in previous jobs: https://support.scrapinghub.com/support/solutions/articles/22000221912-incremental-crawls-with-scrapy-and-deltafetch-in-scrapy-cloud
2 Comments
xiaobie posted almost 5 years ago
nestor posted almost 5 years ago Admin Answer
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
Hello. Could I ask a question about periodic jobs of Scrapinghub? Will the scraped duplicated data be automatically removed by Scrapinghub?
0 Votes
nestor posted almost 5 years ago Admin Best Answer
Scrapinghub doesn't remove duplicated data automatically. I would suggest you try out DeltaFetch to avoid crawling items that were crawled in previous jobs: https://support.scrapinghub.com/support/solutions/articles/22000221912-incremental-crawls-with-scrapy-and-deltafetch-in-scrapy-cloud
0 Votes
2 Comments
xiaobie posted almost 5 years ago
0 Votes
nestor posted almost 5 years ago Admin Answer
Scrapinghub doesn't remove duplicated data automatically. I would suggest you try out DeltaFetch to avoid crawling items that were crawled in previous jobs: https://support.scrapinghub.com/support/solutions/articles/22000221912-incremental-crawls-with-scrapy-and-deltafetch-in-scrapy-cloud
0 Votes
Login to post a comment