No recent searches
Popular Articles
Sorry! nothing found for
Posted over 5 years ago by Erik Efland
The example showed for Crawlera PHP cURL is only for website GET is it possible to have a example where POST is used?
and also how to set user-agent like "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0".
<?php$ch = curl_init();$url = 'https://twitter.com/';$proxy = 'proxy.crawlera.com:8010';$proxy_auth = '<API KEY>:';curl_setopt($ch, CURLOPT_URL, $url);curl_setopt($ch, CURLOPT_PROXY, $proxy);curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxy_auth);curl_setopt($ch, CURLOPT_HEADER, 1);curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);curl_setopt($ch, CURLOPT_TIMEOUT, 180);curl_setopt($ch, CURLOPT_CAINFO, '/path/to/crawlera-ca.crt'); //required for HTTPScurl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE); //required for HTTPS$scraped_page = curl_exec($ch);if($scraped_page === false){ echo 'cURL error: ' . curl_error($ch);}else{ echo $scraped_page;}curl_close($ch);?>
0 Votes
nestor posted over 5 years ago Admin Best Answer
Should be the same way you would send POST without Crawlera:
curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
3 Comments
nestor posted over 5 years ago Admin Answer
Erik Efland posted over 5 years ago
thanks, found that Crawlera only accept string and not array for CURLOPT_POSTFIELDS
nestor posted over 5 years ago Admin
There's no reason for Crawlera to reject a POST request, if it is malformed or unsupported type, it is rejected by the target website.
Do you have a real example?
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
The example showed for Crawlera PHP cURL is only for website GET is it possible to have a example where POST is used?
and also how to set user-agent like "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0".
<?php
$ch = curl_init();
$url = 'https://twitter.com/';
$proxy = 'proxy.crawlera.com:8010';
$proxy_auth = '<API KEY>:';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxy_auth);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 180);
curl_setopt($ch, CURLOPT_CAINFO, '/path/to/crawlera-ca.crt'); //required for HTTPS
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE); //required for HTTPS
$scraped_page = curl_exec($ch);
if($scraped_page === false)
{
echo 'cURL error: ' . curl_error($ch);
}
else
{
echo $scraped_page;
}
curl_close($ch);
?>
0 Votes
nestor posted over 5 years ago Admin Best Answer
Should be the same way you would send POST without Crawlera:
0 Votes
3 Comments
nestor posted over 5 years ago Admin Answer
Should be the same way you would send POST without Crawlera:
0 Votes
Erik Efland posted over 5 years ago
thanks, found that Crawlera only accept string and not array for CURLOPT_POSTFIELDS
0 Votes
nestor posted over 5 years ago Admin
There's no reason for Crawlera to reject a POST request, if it is malformed or unsupported type, it is rejected by the target website.
Do you have a real example?
0 Votes
Login to post a comment