Scrap data from REST API calls with dynamic input for parameters - Zyte

How can we help you today?

Start a new topic

Scrap data from REST API calls with dynamic input for parameters

A

Abhishek Mittal

started a topic almost 4 years ago

Hi,

I'm exploring scrapinghub to scrap data from REST API calls with dynamic input for parameters, but didn't found any reference doc or video.

So, please let me know if its possible or not with scrapinghub.

Sample API call:

POST https://www.xxxx.com/search/SearchResults?fromCity=<from_city_id>&toCity=<to_city_id>&meta=true&DOJ=<DD-MM-YYYY>

HEADERS: Content-Type = application/json

1 Comment

peixoto

said almost 4 years ago

HI,

I assume you are referring the Crawlera.

Crawlera is a rotating proxy network that behaves like a regular proxy server, so all you will need to do to fetch any given URL through Crawlera is configure it at your proxy server.

In that context, crawling those REST calls is perfectly possible:

curl -iSL -k -U $CRAWLERA_API_KEY: -x proxy.crawlera.com:8010 -X POST "$URL"

This should get your post request through. Let me know if have problems.

Login to post a comment

More topics in Technical Help

See all 453 topics