Crawlera is a rotating proxy network that behaves like a regular proxy server, so all you will need to do to fetch any given URL through Crawlera is configure it at your proxy server.
In that context, crawling those REST calls is perfectly possible:
curl -iSL -k -U $CRAWLERA_API_KEY: -x proxy.crawlera.com:8010 -X POST "$URL"
This should get your post request through. Let me know if have problems.
Hi,
I'm exploring scrapinghub to scrap data from REST API calls with dynamic input for parameters, but didn't found any reference doc or video.
So, please let me know if its possible or not with scrapinghub.
Sample API call:
POST https://www.xxxx.com/search/SearchResults?fromCity=<from_city_id>&toCity=<to_city_id>&meta=true&DOJ=<DD-MM-YYYY>
HEADERS: Content-Type = application/json
0 Votes
1 Comments
peixoto posted over 4 years ago Admin
HI,
I assume you are referring the Crawlera.
Crawlera is a rotating proxy network that behaves like a regular proxy server, so all you will need to do to fetch any given URL through Crawlera is configure it at your proxy server.
In that context, crawling those REST calls is perfectly possible:
curl -iSL -k -U $CRAWLERA_API_KEY: -x proxy.crawlera.com:8010 -X POST "$URL"
This should get your post request through. Let me know if have problems.
0 Votes
Login to post a comment