Unable to scrape a hidden phone number from a webpage using the requests module

I'm trying to use the requests module to scrape a phone number located in the middle of a webpage, revealed upon clicking on the button right next to it, visible as To Miss in English or Arata in Romanian.

Normally, when I run the script as is, it throws a "KeyError" error on this line: params['context'] = resp.json()['context'].

However, when I open that link in Chrome Browser, manually click on the button next to the hidden phone number to reveal it, and run the script afterwards, it fetches the phone number flawlessly.

How can I use the script to scrape the phone number without manual intervention?

import requests
from bs4 import BeautifulSoup

link = 'https://www.olx.ro/d/oferta/trepte-glafuri-plinte-IDh12rV.html'
phone_link_base = 'https://www.olx.ro/api/v1/offers/{}/limited-phones/'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
    'Accept': '*/*',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9,bn;q=0.8',
    'Origin': 'https://www.olx.ro',
    'Referer': 'https://www.olx.ro/',
    'Authorization': 'Bearer d4305f7b7375e182187d3d4bc5226b688c9ea131',
}

payload = {"action":"reveal_phone_number","aud":"atlas","actor":{"username":"700663656"},"scene":{"origin":"www.olx.ro","sitecode":"olxro","ad_id":""}}
params = {"context":"","response":""}


ad_id = '251445460'

with requests.Session() as s:
    s.headers.update(headers)
    payload['scene']['ad_id'] = ad_id
    resp = s.post('https://friction.olxgroup.com/challenge',json=payload)
    params['context'] = resp.json()['context']
    res = s.post('https://friction.olxgroup.com/exchange',json=params)
    headers['Friction-Token'] = res.json()['token']
    s.headers.update(headers)
    response = s.get(phone_link_base.format(ad_id))
    print(response.json()['data']['phones'])

Here is another link with ad id number for your test:

another_link = 'https://www.olx.ro/d/oferta/echipa-meseriasi-constructi-execut-case-la-rosu-finisaje-interioare-IDdT0s8.html'
ad_id = '205202152'

To expand on the comments, it is clear that requests is not a browser and will not execute JavaScript, as mentioned in psf/requests issue 6040.
When you click on the "<kbd>Arata</kbd>" button manually, the server sends an AJAX request to retrieve the phone number, and that is not something you are replicating with your current requests approach.

It is also possible OLX utilizes complex measures to prevent web scraping, including dynamic content loading and anti-bot mechanisms.
For instance, the URL you are trying to scrape (https://www.olx.ro/api/v1/offers/{}/limited-phones/) is under the "Disallow: /api/" path, which according to the robots.txt file, should not be accessed by robots (which includes web scrapers).

Back to your code: using static headers and payload, especially using a hard-coded authorization token, is not good practice and will not work most of the time, as they are subject to change.

So, as commented, Selenium would automate a browser and allow you to interact with JavaScript-based websites, clicking on buttons, and waiting for responses, provided you are logged on first.

For that, you need to understand the structure of the login page. In general, you should find the elements for the username and password inputs, fill them with your credentials, and then click the login button. After login, you can extract the token from the cookies (or just continue using Selenium, which should use it automatically in subsequent URL queries).

Here is a general example on how to login using Selenium:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# replace with the path to your own chromedriver
driver = webdriver.Chrome('/path/to/chromedriver')

# Go to the login page
driver.get('https://www.olx.ro/account/?origin=header&backUrl=%2F')

# Enter username and password
username = driver.find_element(By.ID, 'userEmail')
password = driver.find_element(By.ID, 'userPass')
username.send_keys('your_username')
password.send_keys('your_password')

# Click login button
login_button = driver.find_element(By.ID, 'se_userLogin')
login_button.click()

# wait for login to complete
time.sleep(5)

# assuming the cookie name is 'access_token'
cookies = driver.get_cookies()
for cookie in cookies:
    if cookie['name'] == 'access_token':
        print(cookie['value'])

driver.quit()

Replace 'your-username' and 'your-password' with your actual username and password. You also need to replace 'username', 'password' and 'login-button' with the actual IDs of those elements on the login page, which you can find by inspecting the webpage. The same goes for 'access_token', you need to replace it with the actual name of the cookie containing the access token.

However, make sure that automating the process of logging into a user account may violate the website's terms of service.
If a website is using such tokens to prevent scraping, it is clear they do not want their site to be scraped. Not only may it be against their terms of service, but it could also be illegal, depending on your jurisdiction.

OLX's ToS (Term of Services) does include:

> Any aggregation and processing of data and other information available on the Website for the purpose of further distributing them to third parties on other websites and outside the Internet is prohibited. Also, the use of the Website and OLX signs, including graphic elements characteristic, without the express and prior consent of OLX is prohibited."

(Or so I believe: I am not from Romania, and "I Am Not A Lawyer")

But if you made sure you can automate the login part, then you can add to the script:

# Go to the specific ad page
driver.get('https://www.olx.ro/d/oferta/trepte-glafuri-plinte-IDh12rV.html')

# wait for page to load
time.sleep(5)

# Click the "Arata" button
show_phone_button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-action='show-phone']")))
show_phone_button.click()

# wait for phone number to load
time.sleep(5)

# Extract the phone number
phone_number = driver.find_element(By.CSS_SELECTOR, "span[class='xxxx-large margintop7']")

print('Phone number:', phone_number.text)

Make sure again this is for a punctual, local and limited use.