periodic jobs and duplicate data

x

started a topic almost 4 years ago

Hello. Could I ask a question about periodic jobs of Scrapinghub? Will the scraped duplicated data be automatically removed by Scrapinghub?

Best Answer

n

nestor said almost 4 years ago

Scrapinghub doesn't remove duplicated data automatically. I would suggest you try out DeltaFetch to avoid crawling items that were crawled in previous jobs: https://support.scrapinghub.com/support/solutions/articles/22000221912-incremental-crawls-with-scrapy-and-deltafetch-in-scrapy-cloud

said almost 4 years ago

Answer

Scrapinghub doesn't remove duplicated data automatically. I would suggest you try out DeltaFetch to avoid crawling items that were crawled in previous jobs: https://support.scrapinghub.com/support/solutions/articles/22000221912-incremental-crawls-with-scrapy-and-deltafetch-in-scrapy-cloud

x

said almost 4 years ago

Thanks so much.

Zyte Support Center