discovery spider

Posted 12 months ago by Konrad K

Post a topic

Un Answered

Konrad K

Hello,

I try to run a discovery spider. I have about 20k URLs in my database and I see after it is finished status of my job that is:

"frontiers": {
        "seed-discovery": {
            "requests": 10000,
            "fingerprints": 19800
        },

I also see that I have a problem with discovery process of these URLs (not all are scraped). What does that mean that requests have 10k? I think that my problem could be connected to that. What should I set and where? I couldn't find anything in the documentation.

0 Votes

0 Comments