Can i use scrapy project include pymongo ?

Posted about 7 years ago by motogod19

Post a topic

Answered

motogod19

Hi,

I have create a scrapy project and using pymongo, i can save data to my mongodb succeed.(server ip is localhost)

When i deploy my project to here and run it.

It shows 4 errors

I think the issue is

exceptions.ImportError: No module named pymongo.

What step should i miss it ? Or i should use a real ip connect my mongodb ?

Please tell me some advice. Thanks in advance.

0 Votes

nestor posted about 7 years ago Admin Best Answer

There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:

project:
   default: project id

It's supposed to be:

projects:
   default: project id

project: projectid

0 Votes

15 Comments

nestor posted about 7 years ago Admin

You're welcome!

0 Votes

motogod19 posted about 7 years ago

Thank you ,nestor . I haven't notice that my type error. Its stuck me for a month... There is no error now, thank you so much !

0 Votes

nestor posted about 7 years ago Admin Answer

There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:

project:
   default: project id

It's supposed to be:

projects:
   default: project id

project: projectid

0 Votes

motogod19 posted about 7 years ago

Thanks for reply .
I have created scrapinghub.yml . Can i ask how to attach my scrapinghub.yml ? And i'm not so sure how to describe my structure . Here is my structure image, i run movies.py can crawl data to my mongodb , and pipelines.py is about mongodb setting.

0 Votes

nestor posted about 7 years ago Admin

Please attach your scrapinghub.yml, and also what is your project structure?

0 Votes

motogod19 posted about 7 years ago

Thanks for reply , i try type command shub deploy, its still show the error just like the image. Any idea ?

0 Votes

thriveni posted about 7 years ago Admin

Can you try just SHUB Deploy . No need to give the project id again while deploying as you have already given the default project ID.

0 Votes

motogod19 posted about 7 years ago

Hey , sorry for bothering again. I have try to solve this issue couple of days.

Here's my requirements.txt and scrapinghub.yml

When i type command shub deploy 250950

It shows error :

Error: Please define an ID for project "default"

I'm sure 250950 is my project ID.

What step should i miss it ?

Any help would be appreciated . Thank you.

0 Votes

motogod19 posted about 7 years ago

I found a type error for my default number , i fix it but still get the same error

0 Votes

motogod19 posted about 7 years ago

Hi pablo,
Yes, I have add pymongo to requirements.txt.
I have check my pymongo version too.

My requirements.txt setting:

My scrapinghub.yml setting:

Still get the error from scrapinghub

exceptions.ImportError: No module named pymongo

1 pymongo==3.5.1 is my requirements.txt setting
Is not right ?
Thanks for your help again.

0 Votes

Pablo Hoffman posted about 7 years ago Admin

Did you add pymongo to requirements.txt?

0 Votes

motogod19 posted about 7 years ago

By the way i still use localhost , although i think its just not this problem now.

0 Votes

motogod19 posted about 7 years ago

Hi,

I try to add numbers for equirements.txt and configure scrapinghub.yml.

Just like your tutorial

1. Creating the requirements.txt file

This is a regular text file where you list the Python packages that your project depends on, one package per line. For example:

Python

js2xml==0.2.1
extruct==0.1.0
requests==2.6.0

You should always pin down the specific versions for each of your dependencies, as we did in the example above. By doing this, you avoid the trouble of getting your spiders broken due to unexpected upgrades.

2. Configuring your dependencies on scrapinghub.yml

After creating the requirements file, add the requirements_file setting to scrapinghub.yml and point it to your project's requirements.txt path:

Python

projects:
  default: 12345
requirements_file: requirements.txt

I can deploy my project now.

But i run my project still get the same error...

Can i ask how to fix it ? Or i type the wrong dependency ?

Any help would be appreciated. Thanks in advance.

0 Votes

motogod19 posted about 7 years ago

Hi,
Thanks for reply, i have try it but get another issue need some helps.
I have created equirements.txt and configure scrapinghub.yml

I try to type command shub deploy 250950 or shub deploy under project.
I get the error shows
Error: Unable to parse configuration file /Users/motogod19/PythonTutorial/MyMovies/scrapinghub.yml. Maybe a missing colon?

How to fix it ? Is any step i type wrong ? Thanks in advance.

0 Votes

Pablo Hoffman posted about 7 years ago Admin

You need to add pymongo as dependency to your project.

This article explains how to do it in Scrapy Cloud:

https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

And then, yes, you'll need to connect to your MongoDB entering its hostname, and authentication credentials.

0 Votes