class QuotesSpider(scrapy.Spider): name = 'quotes' allowed_domains = ['quotes.toscrape.com'] start_urls = ['http://quotes.toscrape.com/']
def parse(self, response): item = ProjectItem() for quote in response.css('div.quote'): item['quote'] = quote.css('span.text::text').extract_first() item['author'] = quote.xpath('span/small/text()').extract_first() yield item
next_page = response.css('li.next a::attr("href")').extract_first() if next_page is not None: yield response.follow(next_page, self.parse)
I can scrawl many quotes with command
scrapy crawl quotes -o /tmp/quotes.json -t json
Now i want to deploy it into scrapinghub.
In my local pc,
sudo pip install shub --upgrade
shub deploy Error: Cannot find project: There is no scrapinghub.yml, scrapy.cfg, or Dockerfile in this directory or any of the parent directories.
How to deploy my spider into scrapinghub ?
0 Votes
L
Length Power posted
about 7 years ago
Best Answer
Using shub deploy from the project folder ,solved.
0 Votes
3 Comments
Sorted by
V
Vincent Randalposted
about 6 years ago
I'm glad that worked, but please clear about the details. What is "the project" and where is it? I'm following the example on https://scrapy.org/ and I get this error after attempting to run shub deploy
Error: Cannot find project: There is no scrapinghub.yml, scrapy.cfg, or Dockerfile in this directory or any of the parent directories.
0 Votes
L
Length Powerposted
about 7 years ago
Answer
Using shub deploy from the project folder ,solved.
0 Votes
nestorposted
about 7 years ago
Admin
Are you using shub deploy from the project folder?
Here is my first spider created on local pc.
scrapy startproject project
cd project
scrapy genspider quotes quotes.toscrape.com
vim project/items.py
import scrapy
class ProjectItem(scrapy.Item):
quote = scrapy.Field()
author = scrapy.Field()
vim project/spiders/quotes.py
# -*- coding: utf-8 -*-
import scrapy
from project.items import ProjectItem
class QuotesSpider(scrapy.Spider):
name = 'quotes'
allowed_domains = ['quotes.toscrape.com']
start_urls = ['http://quotes.toscrape.com/']
def parse(self, response):
item = ProjectItem()
for quote in response.css('div.quote'):
item['quote'] = quote.css('span.text::text').extract_first()
item['author'] = quote.xpath('span/small/text()').extract_first()
yield item
next_page = response.css('li.next a::attr("href")').extract_first()
if next_page is not None:
yield response.follow(next_page, self.parse)
I can scrawl many quotes with command
scrapy crawl quotes -o /tmp/quotes.json -t json
Now i want to deploy it into scrapinghub.
In my local pc,
sudo pip install shub --upgrade
shub deploy
Error: Cannot find project: There is no scrapinghub.yml, scrapy.cfg, or Dockerfile in this directory or any of the parent directories.
How to deploy my spider into scrapinghub ?
0 Votes
Length Power posted about 7 years ago Best Answer
Using shub deploy from the project folder ,solved.
0 Votes
3 Comments
Vincent Randal posted about 6 years ago
I'm glad that worked, but please clear about the details. What is "the project" and where is it? I'm following the example on https://scrapy.org/ and I get this error after attempting to run shub deploy
Error: Cannot find project: There is no scrapinghub.yml, scrapy.cfg, or Dockerfile in this directory or any of the parent directories.
0 Votes
Length Power posted about 7 years ago Answer
Using shub deploy from the project folder ,solved.
0 Votes
nestor posted about 7 years ago Admin
Are you using shub deploy from the project folder?
0 Votes
Login to post a comment