Scrapy crawl yahoo sites redirect to guce.oath.com/collectConsent
Posted over 6 years ago by Leon Liang
Post a topicPeople who like this
Delete Comment
This post will be deleted permanently. Are you sure?
Code Snippet
Html
Html
Css
JavaScript
Sass
Xml
Ruby
PHP
Java
C#
C++
ObjectiveC
Perl
Python
VB
SQL
Generic Language
When I trying to crawl yahoo shop: https://hk.shop.yahoo.com/shop/CityLink-%E9%A0%98%E5%9F%9F-11756.
The code runs properly on my local machine. But when deployed to scrapinghub, the page will be redirected.
This might because when using local machine, there are cookies existing. But scrapinghub is using dynamic IP address. Then when I try to using splash to click the OK button on the redirected page, it seems not working. Here is what I have found https://stackoverflow.com/questions/51085067/using-scrapy-splash-clicking-a-button
And below is what I have tried to fix this issue.
Attachments (1)
yahoo.py
2.89 KB
0 Votes
0 Comments
Login to post a comment