I have create a scrapy project and using pymongo, i can save data to my mongodb succeed.(server ip is localhost)
When i deploy my project to here and run it.
It shows 4 errors
I think the issue is
exceptions.ImportError: No module named pymongo.
What step should i miss it ? Or i should use a real ip connect my mongodb ?
Please tell me some advice. Thanks in advance.
0 Votes
nestor posted
almost 7 years ago
AdminBest Answer
There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:
project:
default: project id
It's supposed to be:
projects:
default: project id
OR
project: projectid
0 Votes
15 Comments
Sorted by
nestorposted
almost 7 years ago
Admin
You're welcome!
0 Votes
m
motogod19posted
almost 7 years ago
Thank you ,nestor . I haven't notice that my type error. Its stuck me for a month... There is no error now, thank you so much !
0 Votes
nestorposted
almost 7 years ago
AdminAnswer
There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:
project:
default: project id
It's supposed to be:
projects:
default: project id
OR
project: projectid
0 Votes
m
motogod19posted
almost 7 years ago
Thanks for reply . I have created scrapinghub.yml . Can i ask how to attach my scrapinghub.yml ? And i'm not so sure how to describe my structure . Here is my structure image, i run movies.py can crawl data to my mongodb , and pipelines.py is about mongodb setting.
0 Votes
nestorposted
almost 7 years ago
Admin
Please attach your scrapinghub.yml, and also what is your project structure?
0 Votes
m
motogod19posted
almost 7 years ago
Thanks for reply , i try type command shub deploy, its still show the error just like the image. Any idea ?
0 Votes
thriveniposted
almost 7 years ago
Admin
Can you try just SHUB Deploy . No need to give the project id again while deploying as you have already given the default project ID.
0 Votes
m
motogod19posted
almost 7 years ago
Hey , sorry for bothering again. I have try to solve this issue couple of days.
Here's my requirements.txt and scrapinghub.yml
When i type command shub deploy 250950
It shows error :
Error: Please define an ID for project "default"
I'm sure 250950 is my project ID.
What step should i miss it ?
Any help would be appreciated . Thank you.
0 Votes
m
motogod19posted
almost 7 years ago
I found a type error for my default number , i fix it but still get the same error
0 Votes
m
motogod19posted
almost 7 years ago
Hi pablo, Yes, I have add pymongo to requirements.txt. I have check my pymongo version too. My requirements.txt setting: My scrapinghub.yml setting:Still get the error from scrapinghub
exceptions.ImportError: No module named pymongo
1 pymongo==3.5.1 is my requirements.txt setting Is not right ? Thanks for your help again.
0 Votes
Pablo Hoffmanposted
almost 7 years ago
Admin
Did you add pymongo to requirements.txt?
0 Votes
m
motogod19posted
almost 7 years ago
By the way i still use localhost , although i think its just not this problem now.
0 Votes
m
motogod19posted
almost 7 years ago
Hi,
I try to add numbers for equirements.txt and configure scrapinghub.yml.
Just like your tutorial
1. Creating the requirements.txt file
This is a regular text file where you list the Python packages that your project depends on, one package per line. For example:
Python
js2xml==0.2.1
extruct==0.1.0
requests==2.6.0
You
should always pin down the specific versions for each of your
dependencies, as we did in the example above. By doing this, you avoid
the trouble of getting your spiders broken due to unexpected upgrades.
2. Configuring your dependencies on scrapinghub.yml
After creating the requirements file, add the requirements_file setting to scrapinghub.yml and point it to your project's requirements.txt path:
Can i ask how to fix it ? Or i type the wrong dependency ?
Any help would be appreciated. Thanks in advance.
0 Votes
m
motogod19posted
almost 7 years ago
Hi, Thanks for reply, i have try it but get another issue need some helps. I have created equirements.txt and configure scrapinghub.yml
I try to type command shub deploy 250950 or shub deploy under project. I get the error shows Error: Unable to parse configuration file /Users/motogod19/PythonTutorial/MyMovies/scrapinghub.yml. Maybe a missing colon? How to fix it ? Is any step i type wrong ? Thanks in advance.
0 Votes
Pablo Hoffmanposted
almost 7 years ago
Admin
You need to add pymongo as dependency to your project.
This article explains how to do it in Scrapy Cloud:
Hi,
I have create a scrapy project and using pymongo, i can save data to my mongodb succeed.(server ip is localhost)
When i deploy my project to here and run it.
It shows 4 errors
I think the issue is
exceptions.ImportError: No module named pymongo.
What step should i miss it ? Or i should use a real ip connect my mongodb ?
Please tell me some advice. Thanks in advance.
0 Votes
nestor posted almost 7 years ago Admin Best Answer
There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:
It's supposed to be:
OR
0 Votes
15 Comments
nestor posted almost 7 years ago Admin
You're welcome!
0 Votes
motogod19 posted almost 7 years ago
Thank you ,nestor . I haven't notice that my type error. Its stuck me for a month... There is no error now, thank you so much !
0 Votes
nestor posted almost 7 years ago Admin Answer
There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:
It's supposed to be:
OR
0 Votes
motogod19 posted almost 7 years ago
Thanks for reply .
I have created scrapinghub.yml . Can i ask how to attach my scrapinghub.yml ? And i'm not so sure how to describe my structure . Here is my structure image, i run movies.py can crawl data to my mongodb , and pipelines.py is about mongodb setting.
0 Votes
nestor posted almost 7 years ago Admin
Please attach your scrapinghub.yml, and also what is your project structure?
0 Votes
motogod19 posted almost 7 years ago
Thanks for reply , i try type command shub deploy, its still show the error just like the image. Any idea ?
0 Votes
thriveni posted almost 7 years ago Admin
Can you try just SHUB Deploy . No need to give the project id again while deploying as you have already given the default project ID.
0 Votes
motogod19 posted almost 7 years ago
Hey , sorry for bothering again. I have try to solve this issue couple of days.
Here's my requirements.txt and scrapinghub.yml
When i type command shub deploy 250950
It shows error :
Error: Please define an ID for project "default"
I'm sure 250950 is my project ID.
What step should i miss it ?
Any help would be appreciated . Thank you.
0 Votes
motogod19 posted almost 7 years ago
I found a type error for my default number , i fix it but still get the same error
0 Votes
motogod19 posted almost 7 years ago
Yes, I have add pymongo to requirements.txt.
I have check my pymongo version too.
My requirements.txt setting:
My scrapinghub.yml setting:Still get the error from scrapinghub
1 pymongo==3.5.1 is my requirements.txt setting
Is not right ?
Thanks for your help again.
0 Votes
Pablo Hoffman posted almost 7 years ago Admin
Did you add pymongo to requirements.txt?
0 Votes
motogod19 posted almost 7 years ago
0 Votes
motogod19 posted almost 7 years ago
Hi,
I try to add numbers for equirements.txt and configure scrapinghub.yml.
Just like your tutorial
1. Creating the requirements.txt file
This is a regular text file where you list the Python packages that your project depends on, one package per line. For example:
You should always pin down the specific versions for each of your dependencies, as we did in the example above. By doing this, you avoid the trouble of getting your spiders broken due to unexpected upgrades.
2. Configuring your dependencies on scrapinghub.yml
After creating the requirements file, add the requirements_file setting to scrapinghub.yml and point it to your project's requirements.txt path:
I can deploy my project now.
But i run my project still get the same error...
Can i ask how to fix it ? Or i type the wrong dependency ?
Any help would be appreciated. Thanks in advance.
0 Votes
motogod19 posted almost 7 years ago
Thanks for reply, i have try it but get another issue need some helps.
I have created equirements.txt and configure scrapinghub.yml
I try to type command shub deploy 250950 or shub deploy under project.
I get the error shows
Error: Unable to parse configuration file /Users/motogod19/PythonTutorial/MyMovies/scrapinghub.yml. Maybe a missing colon?
How to fix it ? Is any step i type wrong ? Thanks in advance.
0 Votes
Pablo Hoffman posted almost 7 years ago Admin
You need to add pymongo as dependency to your project.
This article explains how to do it in Scrapy Cloud:
https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud
And then, yes, you'll need to connect to your MongoDB entering its hostname, and authentication credentials.
0 Votes
Login to post a comment