X-Crawlera-Profile is a header that is meant to replace X-Crawlera-UA. It not only sets the User-Agent but also applies a set of headers which are actually used by the browser. For example, all modern browsers set Accept-Language and Accept-Encoding headers and some browsers set DNT and Upgrade-Insecure-Requests headers.
To use this header you should first note what your client sends by default. For example, CURL sends Accept header by default:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers"
{
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Forwarded": "for=172.245.146.176;proto=http",
"Host": "httpbin.scrapinghub.com",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"
}
}So when sending X-Crawlera-Profile:desktop in this case, you should send Accept with a blank value, e.g:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop" -H "Accept:"
{
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding": "identity",
"Accept-Language": "en-US,en;q=0.5",
"Content-Length": "0",
"Forwarded": "for=196.52.65.11;proto=http",
"Host": "httpbin.scrapinghub.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0"
}
}Otherwise your client's default request headers will override it:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop"
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "identity",
"Accept-Language": "en-us",
"Content-Length": "0",
"Dnt": "1",
"Forwarded": "for=172.245.145.161;proto=http",
"Host": "httpbin.scrapinghub.com",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
}
}Note that you can override any of the request headers set by X-Crawlera-Profile by sending the header manually, e.g:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop" -H "Accept-Language:en-GB"
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "identity",
"Accept-Language": "en-GB",
"Content-Length": "0",
"Forwarded": "for=191.101.233.31;proto=http",
"Host": "httpbin.scrapinghub.com",
"User-Agent": "Mozilla/5.0 (Windows NT 5.2; rv:45.0) Gecko/20100101 Firefox/45.0"
}
}You may also send the value mobile, for mobile-like requests:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:mobile" -H "Accept:"
{
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "identity",
"Accept-Language": "en-US,en;q=0.8",
"Content-Length": "0",
"Forwarded": "for=191.101.225.98;proto=http",
"Host": "httpbin.scrapinghub.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Linux; Android 7.1.1; SM-N950F Build/NMF26X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"
}
}or the value pass, which will pass all your HTTP request headers to the target website exactly as they come in your request:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:pass"
{
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Forwarded": "for=107.175.204.232;proto=http",
"Host": "httpbin.scrapinghub.com",
"User-Agent": "curl/7.58.0"
}
}