Zyte Support Center

I have a large file (150mb) that has 2 million start urls. What whould be the best way to pass them to scrapy spider hosted on scrapinghub?

What file do you have? I think the easiest would be to simple attach a generator to your `start_requests` function. Something like the following for your spider class. 

def start_requests(self):
    with open('file.csv', r) as fh:
        for line in fh.readlines():
            yield scrapy.Request(
                line, callback=self.parse_feed,
            )

Please note I did not test it. To handle parallel processing I believe you should change your settings. Especially if you are going to do two million requests rapidly they might block you. 

What do you mean? 2 million rows is big, but I think people crawl bigger sets of data. Try to add the CSV file as a requirement in your configuration and it should probably work. Maybe start with a subset of your start URLs.