I just ran my first spider on scrapinghub cloud. Everything is fine except one field "description" is not being saved. The spider runs fine on my machine and can save everything in a csv file. The spider does not need javascript rendering, it simply fetches everything from a script tag. The only guess I have right now is that there might be a limit on the length of the string. Though the descriptions are not that big, around 400-500 characters each.
Has anybody encountered similar problem before or at least knows how to approach debugging it.
Thanks
0 Votes
k
khier996 posted
over 5 years ago
Best Answer
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
0 Votes
2 Comments
Sorted by
k
khier996posted
over 5 years ago
oh yeah, forgot to mention that you can change scrapinghub python version in scrapinghub.yml:
stacks:
default: scrapy:1.3-py3
0 Votes
k
khier996posted
over 5 years ago
Answer
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
Hi everybody,
I just ran my first spider on scrapinghub cloud. Everything is fine except one field "description" is not being saved. The spider runs fine on my machine and can save everything in a csv file. The spider does not need javascript rendering, it simply fetches everything from a script tag.
The only guess I have right now is that there might be a limit on the length of the string. Though the descriptions are not that big, around 400-500 characters each.
Has anybody encountered similar problem before or at least knows how to approach debugging it.
Thanks
0 Votes
khier996 posted over 5 years ago Best Answer
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
0 Votes
2 Comments
khier996 posted over 5 years ago
oh yeah, forgot to mention that you can change scrapinghub python version in scrapinghub.yml:
0 Votes
khier996 posted over 5 years ago Answer
ok, I solved it. Apparently, there was a problem with using different python versions. On my machine I am using python 3, and by default scrapinghub is running 2.7. The exact problem was that I was using type(object) function and I guess in python 2.7 it gives some weird results( I am not exactly sure why. I am new to python)
0 Votes
Login to post a comment