Website giving wrong HTML code when scraping

While trying to scrape nutritional data from a website (https://www.realcanadiansuperstore.ca) I noticed that whatever I tried I couldn't get the proper html code from the site, so I couldn't get the nutritional information. I tried almost everything I've seen online but I still can't get it to work, the only way was to manually download the code from every page.

fp = Request( 
url='https://www.realcanadiansuperstore.ca/honey-nut-cheerios-breakfast-cereal-family-size-wh/p/21103495_EA', 
headers={'User-Agent': 'Mozilla/5.0'} 
) 

mybytes = urlopen(fp).read()

mystr = mybytes.decode("utf8") urlopen(fp).close()

print(mystr)

This is the code I tried but it always returns the wrong HTML code. I've already figured out how to find nutritional value from the proper code I just need to figure out how I can get the code in an efficient way. Please don't judge if the answer is obvious I just started learning this a few days ago haha.

I'm going to write this as an answer, in the hopes that we can use it as a duplicate closer in the future, since this has become one of the most frequently asked Python questions.

Web Pages

There are two parts to displaying a web page. First is the HTML that gets sent from the web server to the browser. This is what you see if you do "View Source" within your browser, it's what you get if you use curl or wget to fetch the page, and it's what you get if you read the page with requests, as you are doing. The browser interprets this HTML to produce a tree of user interface objects, which is called the "DOM" or "Domain Object Model", and the DOM is rendered on the screen.

However, in many cases, this HTML is only partially responsible for what you see on the screen. Most web sites use Javascript to build parts of the screen image on the fly, and some pages use Javascript to build everything. The Javascript code modifies the DOM by adding, editing, or deleting the components. The DOM exists only in the browser's memory as a tree of objects. You see this DOM tree when you use the Developer tools in your browser.

If you are seeing components in your Developer Tools that are not present in the "View Source" HTML, then the only way you can get those components is to use a real browser, using a package like requests-html or Selenium. Those packages run a browser like Chrome or Firefox, which interprets the Javascript code, and provides APIs through which you can scroll through the objects in the DOM.

Some people try to use request-html and just read the "text" attribute, but that only gives you the original HTML. The browser does not actually maintain a modified set of HTML, so you can't simply download the final page you see and search it with BeautifulSoup. As a rule, BeautifulSoup is not useful if you need to query the live DOM, because there's no HTML for it to parse.