Zyte Smart Proxy Manager (formerly Crawlera) headers can be configured in Scrapy per project, per spider and per request.


Configuring the Headers Per Project


List the headers in the Scrapy project's settings.py file:


DEFAULT_REQUEST_HEADERS = {
    "X-Crawlera-Profile": "desktop",
    "X-Crawlera-Cookies": "disable",
}


Configuring the Headers Per Spider


If the project contains several spiders, but the headers should only be used by a select one, make use of custom_settings variable in the spider class file:


class SomeSpider(scrapy.Spider):
    name = 'somename'
    custom_settings = {
        "DEFAULT_REQUEST_HEADERS": {
            "X-Crawlera-Profile": "desktop",
            "X-Crawlera-Cookies": "disable",
        }
    }


Configuring the Headers Per Request


If the headers should only be used by a particular request in a particular method of the spider class (for instance, in start_requests, but not in parse_results), list the headers in the headers parameter of the scrapy.Request object:


def start_requests(self):
    headers = {
        "X-Crawlera-Profile": "desktop",
        "X-Crawlera-Cookies": "disable",
    }
    for start_url in self.start_urls:
        yield scrapy.Request(
            url=start_url,
            callback=self.parse_results,
            headers=headers)