Start a new topic

Use a Crawlera API KEY per request

Hi,


Currently, the Crawlera API KEY is assigned to a Spider and therefore to all its subsequent requests. Is it possible to select the Crawlera API KEY on a per request basis inside a Spider? This comes in handy for websites that display different content based on the IP address without having to create different Spiders for each location from where we would like to get the content.

 

Thanks a lot for your help


Not by default, you would need to modify or create your own middleware with multiple API Keys as variables and have it use a specific API Key based on the domain. So if you have the domain example.com it would use CRAWLERA_APIKEY_US and if it is the domain example.fr it would use CRAWLERA_APIKEY_FR from the settings.

Thank you for your answer, I implemented my own middleware from your crawlera middleware and this works very well.


Now, I have another issue. The response of the targeted website contains the country from where it sees the request and it happens that some of the requests appear to come from another country that the one that is associated with the corresponding CRAWLERA_APIKEY_*. For example, requests using a CRAWLERA_APIKEY associated with a FR region may be viewed by the targeted website as coming from NL, IE, SE and other geographical european countries around FR.


My current workaround is to loop until the expected country is recognized by the targeted website. As a consequence, a bit more than 10% of my requests are "lost" or wasted. How is this possible ? Does this come from Crawlera that uses inappropriate IP addresses every now and then, or is it the targeted website to blame. Is there a way to ensure 0% loss ? Because when disabling Crawlera and using top VPN providers I get 0% loss for that website and region.


Again thank you for your time and help.

Login to post a comment