How to extract information from a complex web page using Selenium via Ruby

As an experiment I wanted to test how to use Ruby, Selenium and the web-driver to access a complex web site. For this test I thought I could take flights.google.com and see how to find something on the page having taken a quick look at the dynamic nature of what visually looks simple but is a complex generated view.

Seems easy enough. So for example the site is flights.google.com and entering two destination points say SFO to LAX the URL would be https://www.google.com/travel/flights/search?tfs=CBwQAhogEgoyMDIzLTA2LTE0KABqBwgBEgNTRk9yBwgBEgNMQVgaIBIKMjAyMy0wNi0xOCgAagcIARIDTEFYcgcIARIDU0ZPQAFIAXABggELCP___________wGYAQE

Now when the page is generated you get some nice displayed lists of the flights and if I want to find a result it's not a readable named set of items.

<div class="yR1fYc" jsaction="click:O1htCb;gP4E0b:O1htCb;DIjhEc:YmNhJf" jsname="BXUrOb">
<div class="mxvQLc ceis6c uj4xv uVdL1c A8qKrc" jsname="HSrbLb">…</div>
</div>

Typically with Webdriver and Selenium I would use

require 'selenium-webdriver'
require 'capybara'

driver = Selenium::WebDriver.for :chrome
driver.get 'https://www.google.com/flights/'

and then use the find element approach with some named element

flights = driver.find_elements_by_class_name('flight')

In this case its a more cryptic name so not sure how to tackle the problem if I'm assuming the names of classes may be dynamically generated.

Any suggestions or approaches ?

TL;DR

You may not be able to permanently solve your problem if the obfuscation is deliberate and continuously evolving. However, there are some best practices you can follow with web sites that aren't deliberately breaking standards. You may just need to reframe your solution to be less brittle in the face of dynamic content and find out what doesn't change on the page.

Analysis and Recommendations

If you're dealing with dynamically-generated content, you need one or more of the following:

1. A driver that can render JavaScript, which not all of them do, for JavaScript-dependent sites.
   • Your Chrome driver does, but may not render the same way it would on other JavaScript drivers; it may be worth trying some others.
   • If Google is deliberately using features of Chrome to obfuscate pages, using a different JavaScript driver or engine may help. YMMV with this one.
2. Some sort of parent-child, n^th element, or container-based search to find what you want if you can't rely on a given class or ID name.
3. Be willing to change your approach to a full-page regexp or fixed-string search if you know that there's a reliable prefix, suffix, or other string-based logic to identify text near the data or HTML elements you want.
4. Consider testing against the HTML-only version of sites when available.
   • You could test with felinks or similar to see what renders without JavaScript.
   • You can search for their screen-reader or accessibility-enabled pages for the content you're trying to find.
   • You can see if they support Web Accessibility Initiative Accessible Rich Internet Applications WAI-ARIA or similar gracefully degrading interfaces that you can parse properly.
5. Consider API access instead.
   • Sometimes web-scraping is the wrong way to solve a data retrieval problem when there are RESTful or GraphQL options available.
   • For example, Google used to offer Google Flight Search (GFS) but seems to have discontinued it. I didn't find an alternative, but I only spent about 30 seconds trying to find one.

For example, if you can rely on something like "First Name" being somewhere in a given div container, you could use an XPath expression to find the parent container and then do a more structured or permissive search from there.

Capybara is pretty powerful, but sometimes you have to do your own parsing. If that happens, take a look at what Nokogiri has to offer, and see if you can do what you want even if you have to perform multiple extraction/search steps to get it done.

Not everything has a one-liner as a solution. Dynamic sites that are designed to break standard tools like mechanize or CSS-based parsers are generally doing that for a reason, I suppose, even though it often breaks compatibility with standards. Assuming they aren't also deliberately breaking accessibility features like ALT text or other accessibility attributes, you might think about leveraging those or similar accessibility features since commercial sites are often required to meet ADA or Section 508 standards, no matter how much they abuse the DOM.

You can still rely on the structure, since the structure is not random; a CSS selector like this should be able to give you the list elements of the results without relying on class names:

body > c-wiz:nth-of-type(2) > div:nth-of-type(1) > div:nth-of-type(2) >
c-wiz:nth-of-type(1) > div:nth-of-type(1) >
c-wiz:nth-of-type(1) > div:nth-of-type(2) > div:nth-of-type(2) > div:nth-of-type(3) >
ul:nth-of-type(1) > li