Problem with type response

Posted about 5 years ago by Igor Drokin

Post a topic

Un Answered

Igor Drokin

Hello!

When i make a post request in Scrapy by scrapy.http.FormRequest without using crawler i get correct response. Response has type json. But when i use crawlera i get html response. My url

"https://www.wayfair.co.uk/a/product/shipping/get_liteship_and_inventory_data", headers

{
    "Accept": "application/json",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "q=0.8,en-US;q=0.5,en;q=0.3",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
    "Pragma": "no-cache",
    "TE": "Trailers",

    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
     "X-Requested-With": "XMLHttpRequest",
}

and payload for example:

"product_data[0][sku]": "DNOR5398",
"product_data[0][qty]": "1",
"product_data[0][is_fully_configured]": "true",
"should_calculate_all_kit_items": "true",
"source": "main_pdp_initial_load",
"event_id": "0",
"postal_code": "",
"clearance": "false",
"should_load_schedule_delivery_calendar": "false",
"out_of_stock_ordering_mode": "0",

So, I indicate the header "Accept" by value "application/json", but with crawlera i get html body response. I think, crawlera change my request. How can i fix it?

Thanks!

1 Votes

0 Comments