how to do a web scraping getting all the data

I was doing a web scrapping to get reviews from the play store in order to study NLP, but my script only brought the first reviews.

I noticed that when we clicked on "Show all reviews" the classes of the elements of the comments on the html page remain the same and the URL too, so theoretically all the data should come.

Can anyone help me?

import requests
from bs4 import BeautifulSoup

def scrape_playstore_simpler_reviews(url):
    reviews_list = []
    response = requests.get(url)

    if response.status_code == 200:
        soup = BeautifulSoup(response.content, 'html.parser')
        app_title1 = soup.find_all('h1', class_='Fd93Bb')

        for app_title in app_title1:
            app_title1 = app_title.find('span')


        reviews = soup.find_all('div', class_='EGFGHd')

        for review in reviews:
            user_name = review.find('div', class_='X5PpBb')
            comment_text = review.find('div', class_='h3YV2d')
            helpful_count = review.find('div', class_='AJTPZc')
            post_date = review.find('span', class_='bp9Aid')
            star_rating = review.find('div', class_='iXRFPc').attrs['aria-label']
            
            review_data = {
                'Título do aplicativo': app_title,
                'Nome do usuário': user_name,
                'Texto do comentário': comment_text,
                'Qtd apoio ao comentário': helpful_count,
                'Data da postagem': post_date,
                'Qtd estrelas': star_rating
            }

            reviews_list.append(review_data)
            print(review_data)
    else:
        print('Falha ao fazer requisição HTTP')

url = 'https://play.google.com/store/apps/details?id=ru.zengalt.simpler&hl=pt_BR'
reviews = scrape_playstore_simpler_reviews(url)

scrape_playstore_simpler_reviews(url)

I expected to see all the app rating comments I looked for

Most websites now use AJAX to dynamically generate content. To get the same content that you see when viewing the page in a browser you'll need to use something like Selenium WebDriver. Sadly, the days of easily scraped static content are long past.

It seems that the issue with your web scraping script lies in how you're trying to extract the reviews from the Google Play Store page. The reason you're only getting the first reviews is that the page uses dynamic loading, and the reviews are loaded as you scroll down.

To scrape all the reviews, you'll need to simulate scrolling or use the Play Store API to get all the reviews. Directly scraping the page won't work since the initial response doesn't contain all the reviews.

Here's a general outline of how you can scrape all reviews using the Play Store API:

1. Use the Play Store API to fetch the reviews data.
2. Parse the JSON response to extract the necessary information.

As you may see when using the console window, and the "ctrl + shift + c" and hovering the mouse over the popup comments, they have a different than the comments after the popup:

I think you should learn some css selectors like the one below to try out the ids/classes of the elements:

try using this in the console after opening the popup:

document.querySelectorAll('.RHo1pe')

It should bring all the 40 comments.