I just ran my first spider on scrapinghub cloud. Everything is fine except one field "description" is not being saved. The spider runs fine on my machine and can save everything in a csv file. The spider does not need javascript rendering, it simply fetches everything from a script tag. The only guess I have right now is that there might be a limit on the length of the string. Though the descriptions are not that big, around 400-500 characters each.
Has anybody encountered similar problem before or at least knows how to approach debugging it.
Thanks
Best Answer
k
khier996
said
almost 5 years ago
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
k
khier996
said
almost 5 years ago
oh yeah, forgot to mention that you can change scrapinghub python version in scrapinghub.yml:
khier996
Hi everybody,
I just ran my first spider on scrapinghub cloud. Everything is fine except one field "description" is not being saved. The spider runs fine on my machine and can save everything in a csv file. The spider does not need javascript rendering, it simply fetches everything from a script tag.
The only guess I have right now is that there might be a limit on the length of the string. Though the descriptions are not that big, around 400-500 characters each.
Has anybody encountered similar problem before or at least knows how to approach debugging it.
Thanks
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstkhier996
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
khier996
oh yeah, forgot to mention that you can change scrapinghub python version in scrapinghub.yml:
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 452 topics