scraping a URL address for reviews

So I need to extract the reviews from the URL of a product on this site, more specifically the username, date, text, and score. However, I have some issues with it because I keep getting an error: failed to retrieve reviews for page 1. Error: "Connection broken: InvalidChunkLength(got length b'', 0 bytes read)"; "InvalidChunkLength(got length b'', 0 bytes read)"; I tried adding a time delay but it still doesn't work. How can I modify this?

import json
import requests
from bs4 import BeautifulSoup

url = "https://www.emag.ro/covor-antiderapant-negru-poliester-80-x-300-cm-c027-80x300/pd/DBY5YJMBM/?ref=sponsored_products_fill_a_b_5_3&provider=rec&recid=rec_73_c449bb3e50b63cc8f6da4a42a31af359f6cbfb3c547bc5748cb6d45501a29685_1684315709&scenario_ID=73&aid=034a897a-956c-11ed-9004-0ab644dfda7c&oid=89847310"

review_url = "https://www.emag.ro/review/get-review-listing-page?id={product_id}&page={page}"
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0'}


product_id = url.split("/pd/")[1].split("/")[0]

reviews = []

page = 1
while True:
    r_url = review_url.format(product_id=product_id, page=page)
    try:
        response = requests.get(r_url, headers=headers)
        response.raise_for_status()  
        data = response.json()
    except (requests.RequestException, json.JSONDecodeError) as e:
        print(f"Failed to retrieve reviews for page {page}. Error: {str(e)}")
        break

    if not data['reviews']:
        break

    for r in data['reviews']:
        review_text = r['content']
        author = r['author']['name']
        date = r['date']
        score = r['rating']
        reviews.append({"author": author, "date": date, "review_text": review_text, "score": score})

    page += 1

with open('reviews.json', 'w') as f:
    json.dump(reviews, f, indent=4)

You got the review URL all wrong.

To get the reviews, you need the following parts, for example:

1. https://www.emag.ro/product-feedback/
2. covor-pufos-moale-compatibil-multiple-spatii-si-stiluri-grosime-4cm-120cm-x-160cm-gri-ronyes18/pd/DBSKZPMBM
3. /reviews/list

This gives you a JSON with everything you need. Check it out yourself:

https://www.emag.ro/product-feedback/covor-kring-meknes-1200-gsm-100-poliester-160x230-cm-maro-e2020-8b/pd/D605NYMBM/reviews/list

Here's a full working example:

import json
import re

import requests

SAMPLE_URLS = [
    "https://www.emag.ro/covor-pufos-moale-compatibil-multiple-spatii-si-stiluri-grosime-4cm-120cm-x-160cm-gri-ronyes18/pd/DBSKZPMBM/?ref=profiled_categories_home_1_3&provider=rec&recid=rec_50_c4fe1107f88ac6bacec1c30b98de4480bfbe39d2b440960a4182a588f76c40f9_1684357366&scenario_ID=50",
    "https://www.emag.ro/covor-kring-meknes-1200-gsm-100-poliester-160x230-cm-maro-e2020-8b/pd/D605NYMBM/?ref=profiled_categories_home_1_2&provider=rec&recid=rec_50_c4fe1107f88ac6bacec1c30b98de4480bfbe39d2b440960a4182a588f76c40f9_1684357366&scenario_ID=50",
    "https://www.emag.ro/antiderapant-pentru-covor-tip-plasa-cali-poliester-80-x-180-cm-j119c9-99st4951/pd/DZDZXTMBM/?ref=profiled_categories_home_1_1&provider=rec&recid=rec_50_c4fe1107f88ac6bacec1c30b98de4480bfbe39d2b440960a4182a588f76c40f9_1684357366&scenario_ID=50"
]

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42",
}

PRODUCT_ID = re.compile(r"pd/(.*?)/reviews")


def splitter(url: str) -> str:
    return url.rsplit('?')[0].split('ro/')[-1]


def save_json(data: dict, filename: str) -> None:
    with open(filename, 'w') as file:
        json.dump(data, file, indent=4)


review_urls = [
    f"https://www.emag.ro/product-feedback/{splitter(url)}reviews/list"
    for url in SAMPLE_URLS
]

with requests.Session() as s:
    s.headers.update(HEADERS)
    for url in review_urls:
        review_data = s.get(url).json()
        save_json(review_data, f"{re.search(PRODUCT_ID, url).group(1)}.json")
        print(url)
        for review in review_data["reviews"]["items"]:
            print(f"{review['rating']}: {review['title']}")
            print(f"{review['content']}")

This should print (shortened for brevity):

https://www.emag.ro/product-feedback/covor-pufos-moale-compatibil-multiple-spatii-si-stiluri-grosime-4cm-120cm-x-160cm-gri-ronyes18/pd/DBSKZPMBM/reviews/list
2: Dezamagitor
A ajuns foarte repede, bine ambalat. Impaturit, f subtire, l-as numi paturica( voi anexa poze) se vede prin el asa este de subtire.
1: Nu recomand
Când am deschis easy boxul, pentru că acolo a sosit, am rămas surprins sa vad cat e se mic pachetul. Că o pătură. Nici pe departe grosimea de 4 cm. Numai aspect de covor nu are. Nu recomand.
1: Nemultumit
Cel mai prost produs comandat vreodată, nu se poate numi covor pt ca nu este, este ca o pătura! Oribil! O mare reclamație din partea mea merita acest produs! Foarte nemulțumita! Un mare retur! Timp pierdut degeaba!
1: Nu recomand
Nu este deloc nici pe departe cu ce este în poza... Asta nu e covor ci pătură de picnic
1: Nu recomand
The worst material ever
1: Nu recomand
Nu-i ce am văzut în poză îmi pare rău că l-am comandat
1: Nu recomand
Dezamăgitor, nu se poate numii covor ci mai degrabă o pătură, jalnic
1: Nu recomand
Nu se poate numi covor.
1: Nu recomand
Covorul este mai degrabă o paturica... Nu este deloc ca în poza și în descriere. Regret ca l-am comandat
1: Nu recomand
E efectiv o patura (pufoasa ce-i drept)

...

This also dumps the JSON response to a file as it iterates thru the URLs.

A sample JSON is too big to show it all, but running this, for example:

$ jq -r '.reviews.items[0].product.image' < D605NYMBM.json

Should output:

{
  "original": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg",
  "resized_images": [
    {
      "size": "80x80",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=80&height=80&hash=F675BCF7E208EEA9E59AAC0EFB10F96D"
    },
    {
      "size": "130x130",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=100&height=100&hash=2EED3E66E2550EE01608A7A622C72BBD"
    },
    {
      "size": "300x300",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=300&height=300&hash=AAD99FCED7B7A35CC54810994A4EA9B8"
    },
    {
      "size": "450x450",
      "url": "https://s13emagst.akamaized.net/products/30770/30769634/images/res_835280d02e4d164887cce1917e98a10e.jpg?width=450&height=450&hash=21D2141F4007A320F28E893E5E299114"
    }
  ]
}

@baduker I didn't know about this URL. Is there similar URL to get alternative offers? For example, for product id DQQ60WBBM (https://www.emag.ro/-/pd/DQQ60WBBM) I want to get main offer data (seller, price) + 8 (at this time) alternative offers data (same).