X-Crawlera-Profile is a header that is meant to replace X-Crawlera-UA. It not only sets the User-Agent but also applies a set of headers which are actually used by the browser. For example, all modern browsers set Accept-Language and Accept-Encoding headers and some browsers set DNT and Upgrade-Insecure-Requests headers.
To use this header you should first note what your client sends by default. For example, CURL sends Accept header by default:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" { "headers": { "Accept": "*/*", "Content-Length": "0", "Forwarded": "for=172.245.146.176;proto=http", "Host": "httpbin.scrapinghub.com", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" } }
So when sending X-Crawlera-Profile:desktop in this case, you should send Accept with a blank value, e.g:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop" -H "Accept:" { "headers": { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "identity", "Accept-Language": "en-US,en;q=0.5", "Content-Length": "0", "Forwarded": "for=196.52.65.11;proto=http", "Host": "httpbin.scrapinghub.com", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0" } }
Otherwise your client's default request headers will override it:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop" { "headers": { "Accept": "*/*", "Accept-Encoding": "identity", "Accept-Language": "en-us", "Content-Length": "0", "Dnt": "1", "Forwarded": "for=172.245.145.161;proto=http", "Host": "httpbin.scrapinghub.com", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6" } }
Note that you can override any of the request headers set by X-Crawlera-Profile by sending the header manually, e.g:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:desktop" -H "Accept-Language:en-GB" { "headers": { "Accept": "*/*", "Accept-Encoding": "identity", "Accept-Language": "en-GB", "Content-Length": "0", "Forwarded": "for=191.101.233.31;proto=http", "Host": "httpbin.scrapinghub.com", "User-Agent": "Mozilla/5.0 (Windows NT 5.2; rv:45.0) Gecko/20100101 Firefox/45.0" } }
You may also send the value mobile, for mobile-like requests:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:mobile" -H "Accept:" { "headers": { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", "Accept-Encoding": "identity", "Accept-Language": "en-US,en;q=0.8", "Content-Length": "0", "Forwarded": "for=191.101.225.98;proto=http", "Host": "httpbin.scrapinghub.com", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Linux; Android 7.1.1; SM-N950F Build/NMF26X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36" } }
or the value pass, which will pass all your HTTP request headers to the target website exactly as they come in your request:
curl -U APIKEY: -x proxy.zyte.com:8011 "http://httpbin.scrapinghub.com/headers" -H "X-Crawlera-Profile:pass" { "headers": { "Accept": "*/*", "Content-Length": "0", "Forwarded": "for=107.175.204.232;proto=http", "Host": "httpbin.scrapinghub.com", "User-Agent": "curl/7.58.0" } }