Using FEEDS and writing scraped data to more than one location

Posted 10 months ago by Aaron McGarvey

Post a topic

Answered

Aaron McGarvey

I have a bunch of spiders running on Scrapy Cloud on a periodic basis. I need to be able to write the scraped data to multiple locations.

I am able to do so when I run the spiders on my local machine using the FEEDS variable that I set in custom settings:

custom_settings=
{
    "FEEDS": 
{f"s3://systems_data/systems_sample_results_page/FILE1.jsonl": {"format": "jsonlines"},
      f"s3://systems_data/systems_historical_results/FILE2.jsonl": {"format": "jsonlines"}
  }
}

In Scrapy cloud, there is a custom setting for FEED_URI but not FEEDS. As far as I can tell, this only allows writing to one location.

How do I write scraped data to multiple locations in Scrapy cloud?

Locally, I am using Mac OSx, Scrapy 2.9.0 Python 3.8.8

0 Votes

Adrian Chaves posted 6 months ago Admin Best Answer

You can use FEEDS in Scrapy Cloud as well, it allows arbitrary settings, not only the listed ones.

To enter an arbitrary setting name, select the first entry in the drop-down list, “Custom Name”.

You can also use the “Raw Settings” tab to edit all your settings as plain text, which is sometimes easier.

0 Votes

1 Comments

Adrian Chaves posted 6 months ago Admin Answer

You can use FEEDS in Scrapy Cloud as well, it allows arbitrary settings, not only the listed ones.

To enter an arbitrary setting name, select the first entry in the drop-down list, “Custom Name”.

You can also use the “Raw Settings” tab to edit all your settings as plain text, which is sometimes easier.

0 Votes