I have a complex spider that uses Scrapy and Splash working on my server I have to migrate to a new server, but it seems to have problems with docker. So I was wondering if your splash service may be an option, but I'm not sure i understand well.
Some questions:
Can use the new server for the crawler and you Splash service as an alternative to a local installation of Splash inside Docker?
In case, in settings.py there is: SPLASH_URL = 'http://localhost:8050' I guess i should change that, how?
My most complicate SplashRequest is: SplashRequest( url = url, endpoint='execute', args={'lua_source': self.script_elenco_js, 'id_elenco': id_elenco, 'id_sessione': id_sessione, 'iframes':1}, callback=callback, )
'self.script_elenco_js', it is a complex Lua script (containing also javascript) Will it work with one of the 2 ways you show?
Thank you Roberto
0 Votes
vaz posted
over 7 years ago
Best Answer
Hi Roberto, to use docker images you can find this article useful:
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
Have a great day!
1 Votes
1 Comments
vazposted
over 7 years ago
Answer
Hi Roberto, to use docker images you can find this article useful:
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
I have a complex spider that uses Scrapy and Splash working on my server
I have to migrate to a new server, but it seems to have problems with docker.
So I was wondering if your splash service may be an option, but I'm not sure i understand well.
Some questions:
Can use the new server for the crawler and you Splash service as an alternative to a local installation of Splash inside Docker?
In case, in settings.py there is:
SPLASH_URL = 'http://localhost:8050'
I guess i should change that, how?
My most complicate SplashRequest is:
SplashRequest(
url = url,
endpoint='execute',
args={'lua_source': self.script_elenco_js, 'id_elenco': id_elenco, 'id_sessione': id_sessione, 'iframes':1},
callback=callback,
)
'self.script_elenco_js', it is a complex Lua script (containing also javascript)
Will it work with one of the 2 ways you show?
Thank you
Roberto
0 Votes
vaz posted over 7 years ago Best Answer
Hi Roberto, to use docker images you can find this article useful:
https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
Have a great day!
1 Votes
1 Comments
vaz posted over 7 years ago Answer
Hi Roberto, to use docker images you can find this article useful:
https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
Have a great day!
1 Votes
Login to post a comment