Zyte Support Center

An error occurs in Python 3.

How to solve it?

stacks:
    default: scrapy:1.4-py3

Traceback (most recent call last):
	  File "/usr/local/lib/python3.6/site-packages/scrapy/commands/crawl.py", line 58, in run
	    self.crawler_process.start()
	  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 280, in start
	    reactor.run(installSignalHandlers=False)  # blocking call
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1199, in run
	    self.mainLoop()
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1208, in mainLoop
	    self.runUntilCurrent()
	--- <exception caught here> ---
	  File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 828, in runUntilCurrent
	    call.func(*call.args, **call.kw)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/utils/reactor.py", line 41, in __call__
	    return self._func(*self._a, **self._kw)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/core/engine.py", line 135, in _next_request
	    self.crawl(request, spider)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/core/engine.py", line 210, in crawl
	    self.schedule(request, spider)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/core/engine.py", line 216, in schedule
	    if not self.slot.scheduler.enqueue_request(request):
	  File "/usr/local/lib/python3.6/site-packages/scrapy/core/scheduler.py", line 57, in enqueue_request
	    dqok = self._dqpush(request)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/core/scheduler.py", line 86, in _dqpush
	    self.dqs.push(reqd, -request.priority)
	  File "/usr/local/lib/python3.6/site-packages/queuelib/pqueue.py", line 35, in push
	    q.push(obj) # this may fail (eg. serialization error)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/squeues.py", line 15, in push
	    s = serialize(obj)
	  File "/usr/local/lib/python3.6/site-packages/scrapy/squeues.py", line 27, in _pickle_serialize
	    return pickle.dumps(obj, protocol=2)
	builtins.TypeError: can't pickle module objects

Hi,

Unfortunately we don't provide coding support. Please raise your questions through reddit or Stack overflow using "scrapy" tag.

Thanks for understanding,

Best,

Pablo