Scraping USA Today Using the Wayback Machine

I am trying to scrape the news title and its url from the USA today website on:
https://web.archive.org/web/20220101001435/https://www.usatoday.com/

Excerpt of the soup it returned that i need:

<a class="section-helper-flex section-helper-row ten-column spacer-small p1-container" data-index="2" data-module-name="promo-story-thumb-small" href="https://www.usatoday.com/story/sports/nfl/2021/12/31/nfl-predictions-wrong-worst-preseason-mvp-super-bowl/9061437002/" onclick="firePromoAnalytics(event)"><div class="section-helper-flex section-helper-column p1-text-wrap"><div class="promo-premium-content-label-wrap"></div><div class="p1-title"><div class="p1-title-spacer">Revisiting our worst NFL predictions of 2021: What went wrong?</div></div><div class="p1-info-wrap"><span class="p1-label">NFL</span>

I don't know how to take specific excerpts like this out of the whole soup as it should with as i assume:

links=soup.find_all('a',href=True)

code:

import re
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import requests

url = 'https://web.archive.org/web/20220101001435/https://www.usatoday.com/'
options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)
driver.get(url)
soup = BeautifulSoup(driver.page_source, "html.parser")

links=soup.find_all('a',href=True)
print (links)

I expect an output of a 2-D array with the first element being the title and the second element being the link, something like:

[...[Revisiting our worst NFL predictions of 2021: What went wrong?,https://www.usatoday.com/story/sports/nfl/2021/12/31/nfl-predictions-wrong-worst-preseason-mvp-super-bowl/9061437002/],...]

soup.find_all returns a list of bs4.element.Tag, not a list of title and href. The href=True specified merely makes sure that the a elements selected have the attribute href with them.

So here is the complete code that will do what you want:

import requests
from bs4 import BeautifulSoup

# 1.
url = "https://web.archive.org/web/20220101001435/https://www.usatoday.com/"
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser")
# 2.
links = soup.find_all("a", class_="section-helper-flex", href=True)

results = []
for link in links:
    # 3.
    title = link.find("div", class_="p1-title-spacer")
    href = link["href"]
    results.append([title.text, href])

print(results)

The process is as follows:

1. Get the HTML source, which you don't need selenium but can use the requests library. After that, parse that with bs4.
2. Find all the <a> elements. Since your original filter is way to generous that would get every single link on the page, I modified it so it will only yield elements that have articles.
3. Finally, extract the title inside a and the URL.

This does exactly what you want:

titles = [title.text for title in soup.select("div.p1_title > div.p1-title-spacer")]
urls = 
 for link in soup.select("a.section-helper-flex.section-helper-row.ten-column.spacer-small.p1-container")]
output = list(zip(titles, urls))
print(output)

What you were doing wrong is that soup.findall returns a list of bs4.element.Tags, not lists containing the text and href.

I used a list comprehension which sought out all divs of the class p1-title-spacer which were children of a div with class p1_title and extracted the text from each of them. Similarly, I used CSS selectors to extract the urls too.

I recommend that you read the documentation for findall.

Alternatively, you could do this without using beautifulsoup at all, purely using selenium:

from selenium.webdriver.common.by import By

titles = [title.text for title in driver.find_elements(By.CSS_SELECTOR, ""div.p1_title > div.p1-title-spacer"")]
urls = 

output = list(zip(titles, urls))
print(output)

You could also look into WebDriverWait to ensure that you don't look for elements before they've loaded.

Here's the whole thing:

import re
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import requests

url = 'https://web.archive.org/web/20220101001435/https://www.usatoday.com/'
options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)
driver.get(url)
soup = BeautifulSoup(driver.page_source, "html.parser")
titles = [title.text for title in soup.select("div.p1_title > div.p1-title-spacer")]
urls = 
 for link in soup.select("a.section-helper-flex.section-helper-row.ten-column.spacer-small.p1-container")]
output = list(zip(titles, urls))
print(output)