python selenium: wait for popup window

I am having difficulty coding python selenium to wait for a popup window. If I write:

from selenium.webdriver.support import expected_conditions as ec
parent_driver = webdriver.Chrome()
parent_driver.get(url)
element = 'ignore this part of the code'
actions = ActionChains(parent_driver)
actions.move_to_element(element).click(element).perform()
##need code here to instruct selenium to wait
popup_driver = parent_driver.switch_to.active_element
##the following code doesn't solve the problem
WebDriverWait(popup_driver, 30).until(
            ec.presence_of_element_located((By.CLASS_NAME,'next_page'))
        )

This doesn't work because the code has not yet detected the popup window, but I cannot instruct selenium to wait for an element in the popup driver to appear without switching to the active element which does not exist yet. None of the elements in the popup_driver appear in the parent_driver. The code that I have seen for detecting popup windows does not work. So I have tried:

    handles = parent_driver.window_handles
    WebDriverWait.until(ec.new_window_is_opened(parent_driver.window_handles))

This does not work, because the number of window_handles does not change when the popup window appears.

    parent_driver.until(ec.visibility_of_element_located((arbitrary, "arbitrary")))

This does not work because I see nothing in the parent_driver that hints at the existence of a popup window. The actual website that I'm working on is goodreads

And the pop-up appears by pressing the text where it says 'x people voted'.

It seems to me that the code for the popup window does not appear in the parent code. So I cannot program selenium to wait until something changes in the parent code because the parent code will never change. Instead what I have to do is:

 child_driver = driver.switch_to.active_element

But selenium will only replace the active driver with the child driver if the child driver actually is present. More precisely, suppose there is an element that exists on the popup window that does not exist on the parent window, let's say it is

The 'window' that opens when you click a '<n> people voted' is not an actual popup, it's just another element in the DOM. You can tell because you can't grab it and move it away from the page. It also doesn't have the chrome that a normal window would (like a close button, a title, etc.)

So, the following works just fine:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# setting up the browser
s = Service(executable_path='bin/chromedriver.exe')
driver = webdriver.Chrome(service=s)
driver.get('https://www.goodreads.com/list/show/1.Best_Books_Ever')

# just grabbing the first one
result = driver.find_element(By.XPATH, "//*[contains(text(), 'people voted')]")
result.click()

# wait for the 'box' element to appear
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "box")))

# `element` is what you want, but wait for it to complete loading, which is when this appears
wait.until(EC.presence_of_element_located((By.ID, "loadingPage")))

# now you can access its contents
print(element.text)

# close the browser, you're done
driver.close()

Having said that, these sites tend to not like you scraping them heavily, so either choose a focused approach, or don't be too surprised if your requests end up getting blocked. Regardless, using an API would be a better idea, if one is available.

The ID for the 'popup' you're waiting for just happens to be the non-descript box. And just waiting for the box to load will get you an empty element if your code runs fast enough, so you'd want to wait for the loadingPage element to have loaded as well, as it only loads after the interesting content has been added.

Note that I'm purposefully using double quotes around the argument passed to find_element and presence_of_element_located, since the values for that argument often contain ' themselves.