Crawlera PHP cURL post example

Posted over 5 years ago by Erik Efland

Post a topic

Answered

Erik Efland

The example showed for Crawlera PHP cURL is only for website GET is it possible to have a example where POST is used?

and also how to set user-agent like "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0".

<?php

$ch = curl_init();

$url = 'https://twitter.com/';
$proxy = 'proxy.crawlera.com:8010';
$proxy_auth = '<API KEY>:';

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxy_auth);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 180);
curl_setopt($ch, CURLOPT_CAINFO, '/path/to/crawlera-ca.crt'); //required for HTTPS
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE); //required for HTTPS

$scraped_page = curl_exec($ch);

if($scraped_page === false)
{
echo 'cURL error: ' . curl_error($ch);
}
else
{
echo $scraped_page;
}

curl_close($ch);

?>

0 Votes

nestor posted over 5 years ago Admin Best Answer

Should be the same way you would send POST without Crawlera:

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

0 Votes

3 Comments

nestor posted over 5 years ago Admin

There's no reason for Crawlera to reject a POST request, if it is malformed or unsupported type, it is rejected by the target website.

Do you have a real example?

0 Votes

Erik Efland posted over 5 years ago

thanks, found that Crawlera only accept string and not array for CURLOPT_POSTFIELDS

0 Votes

nestor posted over 5 years ago Admin Answer

Should be the same way you would send POST without Crawlera:

curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

0 Votes