Start a new topic

discovery spider


I try to run a discovery spider. I have about 20k URLs in my database and I see after it is finished status of my job that is:

 "frontiers": {
        "seed-discovery": {
            "requests": 10000,
            "fingerprints": 19800

I also see that I have a problem with discovery process of these URLs (not all are scraped). What does that mean that requests have 10k? I think that my problem could be connected to that. What should I set and where? I couldn't find anything in the documentation.

Login to post a comment