How to deploy my spider into scrapinghub ?

Posted about 7 years ago by Length Power

Post a topic
Answered
L
Length Power

 Here is my first spider created on local pc.


scrapy startproject project
cd project
scrapy genspider quotes quotes.toscrape.com
vim project/items.py

import scrapy
class ProjectItem(scrapy.Item):
    quote = scrapy.Field()
    author = scrapy.Field()


vim project/spiders/quotes.py

# -*- coding: utf-8 -*-
import scrapy
from project.items import ProjectItem
 
 
class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']
 
    def parse(self, response):
        item = ProjectItem()
        for quote in response.css('div.quote'):
            item['quote'] = quote.css('span.text::text').extract_first()
            item['author'] = quote.xpath('span/small/text()').extract_first()
            yield item
 
        next_page = response.css('li.next a::attr("href")').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)


I can scrawl many quotes with command

scrapy crawl quotes -o /tmp/quotes.json -t json


Now i want to deploy it into  scrapinghub.


In my local pc,

sudo pip  install shub --upgrade

shub deploy
Error: Cannot find project: There is no scrapinghub.yml, scrapy.cfg, or Dockerfile in this directory or any of the parent directories.


How to deploy my spider into scrapinghub ?

0 Votes

L

Length Power posted about 7 years ago Best Answer

 Using shub deploy from the project folder ,solved.

0 Votes


3 Comments

Sorted by
V

Vincent Randal posted about 6 years ago

 I'm glad that worked, but please clear about the details. What is "the project" and where is it? I'm following the example on https://scrapy.org/ and I get this error after attempting to run shub deploy

Error: Cannot find project: There is no scrapinghub.yml, scrapy.cfg, or Dockerfile in this directory or any of the parent directories.



0 Votes

L

Length Power posted about 7 years ago Answer

 Using shub deploy from the project folder ,solved.

0 Votes

nestor

nestor posted about 7 years ago Admin

Are you using shub deploy from the project folder?

0 Votes

Login to post a comment