How to find an existing HTML element with python-selenium in a jupyterhub page?

You were close enough. Factually your entire program had only a single issue as follows:

• The xpath_runall = "//li[@data-command='notebook:run-all-below']" doesn't identify the visible element with text as Run Selected Cell and All Below uniquely as the first matched element is a hidden element.

---

Additional considerations

Some more optimizations:

• The element identified as xpath = "//button[@title='Save the notebook contents and create checkpoint']" is a clickable element. So instead of EC as presence_of_element_located() you can use element_to_be_clickable()

• Once the element is returned through EC as element_to_be_clickable() you can invoke the click() on the same line.

• The xpath to identify the element with text as Run Selected Cell and All Below would be:

  //li[@data-command='notebook:run-all-below']//div[@class='lm-Menu-itemLabel p-Menu-itemLabel' and text()='Run Selected Cell and All Below']
  

• As the application is built through JavaScript you need to use ActionChains.

---

Solution

Your optimized solution will be:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.action_chains import ActionChains
  
  driver = webdriver.Firefox(executable_path=r'C:\WebDrivers\geckodriver.exe')
  driver.get("https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/master?urlpath=lab/tree/demo")
  WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.XPATH, "//button[@title='Save the notebook contents and create checkpoint']")))
  print("Page loaded")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='Run']"))).click()
  print("Clicked on Run")
  element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[@data-command='notebook:run-all-below']//div[@class='lm-Menu-itemLabel p-Menu-itemLabel' and text()='Run Selected Cell and All Below']")))
  ActionChains(driver).move_to_element(element).click(element).perform()
  print("Clicked on Run Selected Cell and All Below")
  

• Console Output:

  Page loaded
  Clicked on Run
  Clicked on Run Selected Cell and All Below
  

This worked for me. I find the top-level menu item using full xpath and then click on it. I wait a small amount of time to ensure that the popup menu has appeared and then using an offset from the original menu item I have pre-determined, I move the mouse to that offset and click on what I know to be the correct sub-menu item. In the code below, I first give myself a chance to select a cell:

driver.implicitly_wait(300) # wait up to 300 seconds before calls to find elements time out
driver.get('https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/master?urlpath=lab/tree/demo')
driver.execute_script("scroll(0, 0);")
elem = driver.find_element_by_xpath('//div[text()="Run"]')
elem.click() # click on top-level menu item
time.sleep(.2) # wait for sub-menu to appear
action = webdriver.common.action_chains.ActionChains(driver)
action.move_to_element_with_offset(elem, 224, 182)
# click on sub-menu item:
action.click()
action.perform()

Update: A More Optimal Solution

driver.implicitly_wait(300) # wait up to 300 seconds before calls to find elements time out
driver.get('https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/master?urlpath=lab/tree/demo')
driver.execute_script("scroll(0, 0);")
elem = driver.find_element_by_xpath('//div[text()="Run"]')
elem.click()
driver.implicitly_wait(.2)
elem2 = driver.find_element_by_xpath('//*[contains(text(),"Run Selected Cell and All Below")]')
driver.execute_script("arguments[0].click();", elem2) # sub-menu, however, stays open
# to close the sub-menu menu:
elem.click()