I have a complex spider that uses Scrapy and Splash working on my server I have to migrate to a new server, but it seems to have problems with docker. So I was wondering if your splash service may be an option, but I'm not sure i understand well.
Some questions:
Can use the new server for the crawler and you Splash service as an alternative to a local installation of Splash inside Docker?
In case, in settings.py there is: SPLASH_URL = 'http://localhost:8050' I guess i should change that, how?
My most complicate SplashRequest is: SplashRequest( url = url, endpoint='execute', args={'lua_source': self.script_elenco_js, 'id_elenco': id_elenco, 'id_sessione': id_sessione, 'iframes':1}, callback=callback, )
'self.script_elenco_js', it is a complex Lua script (containing also javascript) Will it work with one of the 2 ways you show?
Thank you Roberto
Best Answer
v
vaz
said
almost 7 years ago
Hi Roberto, to use docker images you can find this article useful:
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
Have a great day!
1 Comment
vaz
said
almost 7 years ago
Answer
Hi Roberto, to use docker images you can find this article useful:
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
chairam
I have a complex spider that uses Scrapy and Splash working on my server
I have to migrate to a new server, but it seems to have problems with docker.
So I was wondering if your splash service may be an option, but I'm not sure i understand well.
Some questions:
Can use the new server for the crawler and you Splash service as an alternative to a local installation of Splash inside Docker?
In case, in settings.py there is:
SPLASH_URL = 'http://localhost:8050'
I guess i should change that, how?
My most complicate SplashRequest is:
SplashRequest(
url = url,
endpoint='execute',
args={'lua_source': self.script_elenco_js, 'id_elenco': id_elenco, 'id_sessione': id_sessione, 'iframes':1},
callback=callback,
)
'self.script_elenco_js', it is a complex Lua script (containing also javascript)
Will it work with one of the 2 ways you show?
Thank you
Roberto
Hi Roberto, to use docker images you can find this article useful:
https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
Have a great day!
vaz
Hi Roberto, to use docker images you can find this article useful:
https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image
About Lua script, unfortunately we don't provide script assistance and you should ask in Stack Overflow or related channels. We offer developers assistance for advance crawlers through: https://scrapinghub.com/quote
Consider this solution if you want to speed up your project and develop advanced features saving time and resources.
Have a great day!
1 person likes this
-
How click an element and close a popup?
-
How many slots are the hosted splash servers configured with?
-
Scrapy Splash Scrapinghub deployment issue
-
Crawlspider and Splash
-
Interacting with Javascript Popup
-
Items API - RSS
-
Crawlera and MySQL connection
-
Scrapy and Splash times out for a specific site.
-
Can't make Splash works receiving HTTP 401
See all 36 topics