We are changing the behavior of a custom header (X-Crawlera-Profilethat can be sent along with your request to Smart Proxy Manager. This change is expected to go out on Feb 5th 2019 between 12:00-14:00 UTC


Current Behaviour


When X-Crawlera-Profile is sent (say, with values desktop or mobile) along with the proxy request, Smart Proxy Manager sends a default set of headers that match the desktop or mobile user in the outgoing request to the target website. However, when X-Crawlera-Profile is sent along with other HTTP headers such as Accept or Accept-Language, those headers end up overriding Smart Proxy Manager’s behaviour of choosing the best set of browser profile headers. As a result, Smart Proxy Manager will pass through the other HTTP headers to the target website.


New Behaviour


When X-Crawlera-Profile is sent along with other HTTP headers such as Accept or Accept-Language, those headers will not override Smart Proxy Manager’s behaviour of what set of browser profile headers will be sent to the target website. Smart Proxy Manager will not pass through the other HTTP headers to the target website unless it receives an additional header X-Crawlera-Profile-Pass listing the headers that need to be overridden. 


For example, in the following request, Smart Proxy Manager will send Accept-Language:de_DE to the outgoing request to the target website because the header X-Crawlera-Profile-Pass is present in the request with the name of the header (Accept-Language) that needs to be overridden. If X-Crawlera-Profile-Pass is not included in the request, then Smart Proxy Manager will send what it usually sends for the browser profile and ignore the Accept-Language:de_DE header. 


curl -x proxy.zyte.com:8011 -U $MYAPIKEY: \
-H X-Crawlera-Profile:desktop -H Accept-Language:de_DE \
-H X-Crawlera-Profile-Pass:Accept-Language https://httpbin.org/headers


Please note that multiple header names can be passed for X-Crawlera-Profile-Pass header with comma separated values. 


curl -x proxy.zyte.com:8011 -U $MYAPIKEY: \
-H X-Crawlera-Profile:desktop -H Accept-Language:de_DE \
-H Accept:text/html -H X-Crawlera-Profile-Pass:Accept-Language,Accept \
https://httpbin.org/headers


Those header names will be overridden only if they are not empty in the request. If you ask Smart Proxy Manager to override Accept header but do not send any Accept header, then Smart Proxy Manager will include the default Accept header for the browser profile.