Gzip / Feed export error

Posted 3 months ago by Dave Zvenyach

Post a topic

Un Answered

Dave Zvenyach

I am getting a seek of closed file error (see below) when I try to export items into S3 using the Gzip postprocessing plugin. I am using the

scrapy:2.11 stack and the following settings:

"FEEDS": {
 "s3://bucket/key.jsonl.gz": {
  "format": "jsonlines",
  "encoding": "utf8",
  "store_empty": False,
  "indent": 2,
  "postprocessing": ["scrapy.extensions.postprocessing.GzipPlugin"],
 }
}

Any help would be welcome!

Thanks!

***

[scrapy.extensions.feedexport] Error storing jsonlines feed (17523 items) in: s3://bucket/key.jsonl.gz

Less

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 244, in _store_in_thread
    file.seek(0)
  File "/usr/local/lib/python3.11/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
ValueError: seek of closed file

0 Votes

0 Comments