When I trying to crawl yahoo shop: https://hk.shop.yahoo.com/shop/CityLink-%E9%A0%98%E5%9F%9F-11756.
The code runs properly on my local machine. But when deployed to scrapinghub, the page will be redirected.
This might because when using local machine, there are cookies existing. But scrapinghub is using dynamic IP address. Then when I try to using splash to click the OK button on the redirected page, it seems not working. Here is what I have found https://stackoverflow.com/questions/51085067/using-scrapy-splash-clicking-a-button
And below is what I have tried to fix this issue.
function main(splash) splash:wait(1) splash:runjs('document.querySelector("button.primary").click()') splash:wait(1) return { html = splash:html(), }end
Leon Liang
When I trying to crawl yahoo shop: https://hk.shop.yahoo.com/shop/CityLink-%E9%A0%98%E5%9F%9F-11756.
The code runs properly on my local machine. But when deployed to scrapinghub, the page will be redirected.
This might because when using local machine, there are cookies existing. But scrapinghub is using dynamic IP address. Then when I try to using splash to click the OK button on the redirected page, it seems not working. Here is what I have found https://stackoverflow.com/questions/51085067/using-scrapy-splash-clicking-a-button
And below is what I have tried to fix this issue.