Selenium, the problem of not being able to obtain all price and date information of a product

the code that follows the price of the cheapest price product searched on the site at a certain time at a certain time interval for the same seller:

import pandas as pd
import undetected_chromedriver as uc
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import matplotlib.pyplot as plt
from time import sleep
import inspect
import os
from bs4 import BeautifulSoup
import requests


# Get the search term and tracking period from the user
search_term = input("Please enter the name of the product you want to search: ")
months =input("Please enter the number of months you want to track the product: ")

# To ensure that the user enters a non-string value 
while not months.isdigit():
    print("Warning: Please enter a valid integer value for the number of months.")
    months = input("Please enter the number of months you want to track the product: ")
months = int(months)


# Start the web driver and go to the Hepsiburada homepage
options = uc.ChromeOptions()
options.add_argument('--blink-settings=imagesEnabled=false') # disable images for loading of page faster
options.add_argument('--disable-notifications')
prefs = {"profile.default_content_setting_values.notifications" : 2}
options.add_experimental_option("prefs",prefs)
driver = uc.Chrome(options=options)

url = 'https://www.hepsiburada.com/'
driver.get(url)
wait = WebDriverWait(driver, 15)

# close cookies bar
wait.until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()

# Enter the search term in the search box and press Enter
search_box = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'theme-IYtZzqYPto8PhOx3ku3c')))
search_box.send_keys(search_term + Keys.RETURN)



# load all products
number_of_products = int(wait.until(EC.visibility_of_all_elements_located((By.CLASS_NAME, 'searchResultSummaryBar-AVnHBWRNB0_veFy34hco')))[1].text)
### visibility_of_all_elements_located is a wait strategy in Selenium that checks if all elements of a certain type are visible on the page and waits until they become visible before continuing.


number_of_loaded_products = 0
while number_of_loaded_products < number_of_products:
    loaded_products = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'li[class*=productListContent][id]')))
    number_of_loaded_products = len(loaded_products)
    driver.execute_script('arguments[0].scrollIntoView({block: "center", behavior: "smooth"});', loaded_products[-1])

# Get the link, name, price and seller of all the products
product = {key:[] for key in ['name','price','seller','url']}
product['name']  = [h3.text for h3 in driver.find_elements(By.CSS_SELECTOR, 'h3[data-test-id=product-card-name]')]
product['url']   = [a.get_attribute('href') for a in driver.find_elements(By.CSS_SELECTOR, 'a[class*=ProductCard]')]
product['price'] = [float(div.text.replace('TL','').replace(',','.')) for div in driver.find_elements(By.CSS_SELECTOR, 'div[data-test-id=price-current-price]')]
for i,url in enumerate(product['url']):
    print(f'Search seller names {i+1}/{number_of_loaded_products}', end='\r')
    driver.get(url)
    product['seller'] += [wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.seller a'))).text]
    product['url'][i] = driver.current_url # useful to replace some long urls
    
# Sort by price in ascending order
import pandas as pd
product_list = pd.DataFrame(product).sort_values(by='price').to_dict('list')

print(f"\nThe product selected from the search results is:"+
      f"\nname:   {product_list['name'][0]}"+
      f"\nprice:  {product_list['price'][0]}"+
      f"\nseller: {product_list['seller'][0]}"+
      f"\nurl:    {product_list['url'][0]}")
    
# Go to the page of the selected product
driver.get(product_list['url'][0])

# Get the prices
prices = []
dates = []
while len(prices) < months:
    price_elems = driver.find_elements(By.XPATH, "//div[@class='price-area']//strong[@itemprop='price']")
    print(price_elems)
    date_elems = driver.find_elements(By.XPATH, "//div[@class='product-info']//span[@class='product-info-date']")
    print(date_elems)
    for price_elem, date_elem in zip(price_elems, date_elems):
        price = float(price_elem.text.replace('.', '').replace(',', '.'))
        date = pd.to_datetime(date_elem.text, format='%d %B %Y, %H:%M')
        prices.append(price)
        dates.append(date)
        
    next_button = driver.find_element(By.XPATH, "//a[@class='page-next']")
    if 'disabled' in next_button.get_attribute('class'):
        break
    else:
        driver.execute_script("arguments[0].click();", next_button)
        

# Create a DataFrame and select the data for the last X months
df = pd.DataFrame({'Date': dates, 'Price': prices})
df['Hour'] = df['Date'].dt.hour
df = df.groupby(['Date', 'Hour']).mean().reset_index()
start_date = pd.Timestamp.today() - pd.DateOffset(months=months)
end_date = pd.Timestamp.today()
df = df.loc[(df['Date'] >= start_date) & (df['Date'] <= end_date)]

# Create the plot
plt.plot(df['Date'], df['Price'])
plt.title('Price Changes of {} in the Last {} Months'.format(product_list['name'][0], months))
plt.xlabel('Date')
plt.ylabel('Price (TL)')
plt.show()

I am trying to create a graph of the price of a product searched on the website Hepsiburada.com.tr with the cheapest price, for the same seller during a certain month, at the same time (For instance, let's say the product whose price we follow is "pınar süt 1lt"). However, I could not draw the graph because I could not obtain the "prices" and "dates" information.This list is empty. How can I obtain this graph?

Focusing point:The piece of code under the '# Get the prices' comment is working incorrectly. The code up to this part is working properly.

I'll try to direct you towards a solution. As I understand you need to track product price changes and process them somehow. You can do it by periodically running a script that collects product prices and stores data somewhere for future analysis.

I see currently you use WebDriver to grab price data. My first suggestion is to try to use Web API instead. Communication with a website via HTTP is much faster and more stable than via UI. Automating UI requires you to deal with different issues related to element inaccessibility, tricky waits, and unexpected overlapping controls. Web API gives you a clear interface to request and receive needed data without the overhead to handle UI. Speaking shortly - UI is for humans, API is for machines. Use API if possible.

In case you need to stay with WebDriver, check the following to address data extraction issues:

• the required data is displayed on the page before extraction

Make sure the steps leading to the target data page are completed successfully. It may happen some automated interactions fail silently and the needed data is not shown.
Button clicks may be skipped, element state waits may be wrong thus letting the script extract data when it's not displayed.

Watch your script execution in real-time and make sure it successfully passes all steps before data extraction. If some steps fail, put sleeps initially just to verify it is a page state issue, then replace on custom waits if so. Try to use different click methods if some fail.

• make sure the required data is in view, not outside the visible page area

To extract data from some controls they need to be in view. Scroll the page if needed

• parent iframe is selected before interaction with a child element

Some UI controls may be put inside iframes. If interaction with a control fails, check if it is inside iframe. WebDriver needs to be switched to the parent iframe before interaction with controls inside.
Use WebInspector in a browser to find if a given UI control is inside iframe

• try to execute a script in another browser

Ideally, WebDriver script should work the same for all browsers but in fact issues happen and a button click that fails in one browser may works in another one.
If your script looks perfect but element interaction still fails, try another browser.