The example showed for Crawlera PHP cURL is only for website GET is it possible to have a example where POST is used?
and also how to set user-agent like "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0".
<?php$ch = curl_init();$url = 'https://twitter.com/';$proxy = 'proxy.crawlera.com:8010';$proxy_auth = '<API KEY>:';curl_setopt($ch, CURLOPT_URL, $url);curl_setopt($ch, CURLOPT_PROXY, $proxy);curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxy_auth);curl_setopt($ch, CURLOPT_HEADER, 1);curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);curl_setopt($ch, CURLOPT_TIMEOUT, 180);curl_setopt($ch, CURLOPT_CAINFO, '/path/to/crawlera-ca.crt'); //required for HTTPScurl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE); //required for HTTPS$scraped_page = curl_exec($ch);if($scraped_page === false){ echo 'cURL error: ' . curl_error($ch);}else{ echo $scraped_page;}curl_close($ch);?>
Should be the same way you would send POST without Crawlera:
curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
thanks, found that Crawlera only accept string and not array for CURLOPT_POSTFIELDS
There's no reason for Crawlera to reject a POST request, if it is malformed or unsupported type, it is rejected by the target website.
Do you have a real example?
Erik Efland
The example showed for Crawlera PHP cURL is only for website GET is it possible to have a example where POST is used?
and also how to set user-agent like "User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0".
<?php
$ch = curl_init();
$url = 'https://twitter.com/';
$proxy = 'proxy.crawlera.com:8010';
$proxy_auth = '<API KEY>:';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxy_auth);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 180);
curl_setopt($ch, CURLOPT_CAINFO, '/path/to/crawlera-ca.crt'); //required for HTTPS
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE); //required for HTTPS
$scraped_page = curl_exec($ch);
if($scraped_page === false)
{
echo 'cURL error: ' . curl_error($ch);
}
else
{
echo $scraped_page;
}
curl_close($ch);
?>
Should be the same way you would send POST without Crawlera:
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
Should be the same way you would send POST without Crawlera:
Erik Efland
thanks, found that Crawlera only accept string and not array for CURLOPT_POSTFIELDS
nestor
There's no reason for Crawlera to reject a POST request, if it is malformed or unsupported type, it is rejected by the target website.
Do you have a real example?
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics