Start a new topic
Answered

Can i use scrapy project include pymongo ?

Hi,

  I have create a scrapy project and using pymongo, i can save data to my mongodb succeed.(server ip is localhost)


When i deploy my project to here and run it.

It shows 4 errors

image


I think the issue is

exceptions.ImportError: No module named pymongo.


What step should i miss it ? Or i should use a real ip connect my mongodb  ?


Please tell me some advice. Thanks in advance.




Best Answer

There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:

 

project:
   default: project id

It's supposed to be:

projects:
   default: project id

OR

project: projectid

 


You need to add pymongo as dependency to your project.


This article explains how to do it in Scrapy Cloud:

https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud


And then, yes, you'll need to connect to your MongoDB entering its hostname, and authentication credentials.

Hi,
Thanks for reply, i have try it but get another issue need some helps.
I have created equirements.txt and configure scrapinghub.yml

image

image

I try to type command shub deploy 250950 or shub deploy under project.
I get the error shows
Error: Unable to parse configuration file /Users/motogod19/PythonTutorial/MyMovies/scrapinghub.yml. Maybe a missing colon?

image

How to fix it ?  Is any step i type wrong ? Thanks in advance.

 

Hi,

I try to add numbers for equirements.txt and configure scrapinghub.yml.


Just like your tutorial

1. Creating the requirements.txt file

This is a regular text file where you list the Python packages that your project depends on, one package per line. For example:

Python
js2xml==0.2.1
extruct==0.1.0
requests==2.6.0

You should always pin down the specific versions for each of your dependencies, as we did in the example above. By doing this, you avoid the trouble of getting your spiders broken due to unexpected upgrades.


2. Configuring your dependencies on scrapinghub.yml

After creating the requirements file, add the requirements_file setting to scrapinghub.yml and point it to your project's requirements.txt path:

Python
projects:
  default: 12345
requirements_file: requirements.txt


I can deploy my project now.

image


But i run my project still get the same error...

Can i ask how to fix it ? Or i type the wrong dependency ?

image


Any help would be appreciated. Thanks in advance.


By the way i still use localhost , although i think its just not this problem now.

 

Did you add pymongo to requirements.txt?

Hi pablo,
Yes, I have add pymongo to requirements.txt.
I have check my pymongo version too.

image

My requirements.txt setting:

image

My scrapinghub.yml setting:

image

Still get the error from scrapinghub
exceptions.ImportError: No module named pymongo

image

1 pymongo==3.5.1  is my requirements.txt setting
Is not right ?
Thanks for your help again.

 

image


 I found a type error for my default number , i fix it but still get the same error

 Hey , sorry for bothering again. I have try to solve this issue couple of days.


Here's my requirements.txt and scrapinghub.yml

image


image


When i type command shub deploy 250950

It shows error :

Error: Please define an ID for project "default"

image


I'm sure 250950 is my project ID.

image


What step should i miss it ?

Any help would be appreciated . Thank you.

Can you try just SHUB Deploy . No need to give the project id again while deploying as you have already given the default project ID.

Thanks for reply , i try type command shub deploy, its still show the error just like the image. Any idea ?

image


Please attach your scrapinghub.yml, and also what is your project structure?

Thanks for reply . 
I have created scrapinghub.yml . Can i ask how to attach my scrapinghub.yml ? And i'm not so sure how to describe my structure .  Here is my structure image, i run movies.py can crawl data to my mongodb , and pipelines.py is about mongodb setting.

image


Answer

There's an Attach File button at the bottom of the message box. Anyways it's not necessary, I looked at the screenshots of your previous messages. Your last screenshot reads:

 

project:
   default: project id

It's supposed to be:

projects:
   default: project id

OR

project: projectid

 

Thank you ,nestor .  I haven't notice that my type error. Its stuck me for a month... There is no error now, thank you so much !

image


Login to post a comment