Zyte Support Center

Getting this strange error message.

Assume it has smth to do with payload. Though all works fine locally on windows machine.

[scrapy.core.scraper] Spider error processing <POST http://oris.co.palm-beach.fl.us/or_web1/new_sch.asp> (referer: http://oris.co.palm-beach.fl.us/or_web1/)

 Less

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1276, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 http://oris.co.palm-beach.fl.us/or_web1/new_sch.asp>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/scrapy/core/spidermw.py", line 42, in process_spider_input
    result = method(response=response, spider=spider)
  File "/usr/local/lib/python3.6/site-packages/scrapy_pagestorage.py", line 68, in process_spider_input
    self.save_response(response, spider)
  File "/usr/local/lib/python3.6/site-packages/scrapy_pagestorage.py", line 102, in save_response
    self._writer.write(payload)
  File "/usr/local/lib/python3.6/site-packages/scrapinghub/hubstorage/batchuploader.py", line 224, in write
    data = jsonencode(item)
  File "/usr/local/lib/python3.6/site-packages/scrapinghub/hubstorage/serialization.py", line 38, in jsonencode
    return dumps(o, default=jsondefault)
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: keys must be a string

ok. the issue was with - Messagepack is not available, and page storage enabled for this project.

I have disabled pagestorage and it works fine now.

I wished Error messages were more readable.

Nope. Had nothing to do with init.

@aurish_hammad_hafeez

My local scrapy version is 1.5.1

So I updated the stack to 1.5

Getting same result.

I do not have init method in my spider and pass input variable directly. Can be this a problem in the cloud?

@aurish_hammad_hafeez

 From And To dates are passed as parameters when starting the spider.

def parse(self, response):
        # inspect_response(response, self)
        
        url = 'http://oris.co.palm-beach.fl.us/or_web1/new_sch.asp'
        headers = {
                        'upgrade-insecure-requests': "1",
                        'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36",
                        'origin': "http://oris.co.palm-beach.fl.us",
                        'content-type': "application/x-www-form-urlencoded",
                        'dnt': "1",
                        'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
                        'cache-control': "no-cache",
                                                }
        # Date range should be visin 90 days
        data = {'FromDate': self.FromDate,
                     'PageSize': '500',
                     'RecSetSize': '500',
                     'ToDate': self.ToDate,
                     'consideration': '',
                     'search_by': 'DocType',
                     'search_entry': 'LP'}                                            
        body = urlencode(data)                                         
        yield scrapy.Request(url, method="POST", headers = headers, body = body,  callback = self.parsed)