将JS变量传递给Selenium的execute_script [Python]

huangapple go评论88阅读模式
英文:

Pass JS variable inside Selenium execute_script [Python]

问题

I'm sorry for literally spamming Selenium threads.

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)

这是我用来在动态生成的列表上向下滚动的代码(Instagram关注者)。这是一个有限的列表,所以你滚动得越低,滚动条就会越小,因为在顶部列表只有12个元素,而在底部则包括所有的关注者。我的问题是我有一些数据丢失:当我扫描了分别有406和280个关注者的2个帐户时,爬虫仅返回了392和264个用户名。
我检查了页面的HTML源代码,以查看数据丢失是否是由于数据生成/抓取/处理引起的。看起来“完全”生成的HTML列表中有392个和264个元素。因此,页面上加载元素的方式有问题。我不知道是什么导致了这些损失,我想是因为互联网断断续续的问题,再加上粗糙的滚动(每次滚动前后都有大约2-3秒的暂停,所以加载时间不应该是个问题)。
回到

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)

我想让滚动更加精细。
随着滚动逐渐变小,滚动条保持相同的大小,我不能使用预设的像素数。显然,也不能使用固定的因子来乘以滚动高度,比如1/2或3/4,因为滚动不会到达列表的末尾。我认为我需要一个随着每次滚动循环而增加的动态因子,类似于:

for i in range(max_followers):
  q = i / max_followers
  driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight * q", element)

所以每次滚动将下降约1个元素/迭代。
但是当我运行这样的代码时,我会得到一个错误:

selenium.common.exceptions.JavascriptException: Message: javascript error: q is not defined

我想这是可以理解的,我没有在js内定义一个变量。是否有办法在execute_script内定义q,并将在Python中生成的值(i,max_followers)传递给JavaScript?

英文:

Im sorry for literally spamming Selenium threads.

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)

This is the code I use to scroll down on a dynamically generated list ( Instagram followers). It's a finite list, so the lower you go with the scroll, the smaller that scroll gets, because at the top the list has only 12 elements inside, while at the bottom it consists of all your followers. My issue is that I have some data loss: when I scanned 2 accounts with 406, respectively 280 followers, the scraper returned only 392 and 264 usernames.
I checked the HTML sauce of the page to see if the loss was due to data gen/scraping/data processing. And it seems that the "fully" generated HTML lists had 392 , respectively 264 elements inside. So the way elements were loaded on the page was problematic. I don't know what is causing these losses, I suppose internet hiccups combined with chunky scrolling ( around 2-3 secs of pause before and after each scroll with time.sleep(), so time to load shout not be a problem ).
Returning to

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)

I want to make the scroll a little finer.
As the scroll gets progressively smaller and smaller and the scrollbar keeps the same size, I can't go for a preset pixel amount . Obviously neither with a fixed factor to multiply the scrollheight, like 1/2 or 3/4 bcs the scroll won't reach the end of the list. The way I see it I need a dynamic factor that increases with every iteration of the scroll loop, something like

for i in range(max_followers):
  q=i/max_followers
  driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight*q", element)

so each scroll will go down ~1 element/iteration.
But when I run a code like this I get an error:

selenium.common.exceptions.JavascriptException: Message: javascript error: q is not defined

Pretty understandable I guess, I have not defined a variable inside js.Is there any way to define q inside execute_script and pass values generated in python ( i, max_followers ) to jscript?

答案1

得分: 1

你可以将多个参数传递给 execute_script

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight * arguments[1]", element, q)
英文:

You can pass multiple arguments to execute_script.

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight * arguments[1]", element, q)

答案2

得分: 1

你可以在将字符串传递给JavaScript之前对其进行格式化。在Python3中标准的做法是使用f-字符串,它使用{}来引用变量。

for i in range(max_followers):
  q=i/max_followers
  driver.execute_script(f"arguments[0].scrollTop = arguments[0].scrollHeight*{q}", element)

这个代码将会用计算出的q替换掉q。你可以在这里阅读更多关于f-字符串语法的信息:https://realpython.com/python-f-strings/。

英文:

You can format the string before passing it to Javascript. The standard way to do this in Python3 is f-strings, which uses {} to reference a variable.

for i in range(max_followers):
  q=i/max_followers
  driver.execute_script(f"arguments[0].scrollTop = arguments[0].scrollHeight*{q}", element)

What this does is replace q with your calculated q. You can read more about f-string syntax here: https://realpython.com/python-f-strings/.

huangapple
  • 本文由 发表于 2023年5月11日 09:30:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/76223556.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定