Playwright在有头模式下工作，但在无头模式下失败。

huangapple
go评论119阅读模式
英文:

Playwright works in headful mode but fails in headless

问题

我试图使用这个示例来获取NFT在opensea上的报价数量：

  1. import { test, expect } from '@playwright/test';
  3. test('test', async ({ page }) => {
  4.     await page.goto('https://opensea.io/assets/ethereum/0x63217dbb73e7a02c1d30f486e899ee66d0aa5e0b/6341');
  5.     await page.waitForLoadState('networkidle');
  7.     let selector = page.locator("[id='Body offers-panel'] li");
  8.     const offers = await selector.count();
  10.     console.log('Num of offers:', offers);
  11. });

然后我运行 "npx playwright tests"，它总是打印 "Num of offers: 0"。

但是，如果我在 --headed 模式下运行它，它完美地运行并输出 "Num of offers: 5"。

有人能解释/帮助我理解吗？

我尝试过使用：

  1. let selector = page.locator("[id='Body offers-panel'] li").waitFor();

尝试等待直到所有请求完成：

  1. await page.waitForLoadState('networkidle');

尝试等待选择器：

  1. let selector = page.locator("[id='Body offers-panel'] li").first().waitFor();

但都没有起作用，除非我在 --headed 模式下运行测试，无论我尝试哪个NFT地址。

我想解决这个问题或理解为什么会发生这种情况。

英文:

im trying this sample to obtain the number of offers a NFT has in opensea:

  1. import { test, expect } from '@playwright/test';
  3. test('test', async ({ page }) => {
  4.     await page.goto('https://opensea.io/assets/ethereum/0x63217dbb73e7a02c1d30f486e899ee66d0aa5e0b/6341');
  5.     await page.waitForLoadState('networkidle');
  7.     let selector = page.locator("[id='Body offers-panel'] li");
  8.     const offers = await selector.count();
  10.     console.log('Num of offers:', offers);
  11. });

and then I run "npx playwright tests" what always print "Num of offers: 0"

But if I run it in --headed mode, it works perfectly and outputs "Num of offers: 5"

Can anyone explain/help me to understand it?

I tried using:

  1. let selector = page.locator("[id='Body offers-panel'] li").waitFor();

Tried to wait until all requests are done

  1. await page.waitForLoadState('networkidle');

tried to wait for the selector:

  1. let selector = page.locator("[id='Body offers-panel'] li").first().waitFor();

But none worked, I always have 0 count unless I run the test in --headed mode, no matter of which NFT address I try.

I would like to solve it or understand why this happen

答案1

得分: 2

一些网站如果检测到无界面客户端，可能会阻止加载页面。这是为了防止数据抓取等行为。我猜这可能是发生的情况。

请参见：
你是无界面吗？
检测无界面

英文:

Some websites will not load the page if they detect a headless client. This is to prevent scraping and such. My guess is this is what's happening here

See: <br/>
Are you headless? <br/>
Detect Headless

答案2

得分: 1

Headless 模式会使服务器更容易识别你的脚本是一个机器人。当以无界面方式运行时，你会被检测到并被屏蔽，但在有界面方式下可以绕过检测。

由于你看不到任何内容，因此在无界面模式下进行调试比有界面模式更加困难。使用 console.log(await page.content()) 和 await page.screenshot({path: "test.png"}) 是找出为什么你期望在页面上出现的元素没有出现的好策略。

在这种情况下，在 goto 后添加以下内容来获取页面的全部文本内容：

  1. const text = (await page.textContent("body"))
  2.   .replace(/ +/g, " ")
  3.   .replace(/(\n ?)+/g, "\n")
  4.   .trim();
  5. console.log(text);

输出将是：

  1. 拒绝访问
  2. 错误代码 1020
  3. 您无法访问 <Your URL>。站点所有者可能设置了限制，防止您访问该站点。
  4. 错误详情
  5. 请向站点所有者提供此信息。
  6. 我在访问 <Your URL> 时发生了错误。
  7. 错误代码：1020
  8. Ray ID: **************
  9. 国家：US
  10. 数据中心：*****
  11. IP：*****************
  12. 时间戳：2023-02-17 22:39:13 UTC
  13. 点击以复制
  14. 此页面有用吗？
  15. 是
  16. 否
  17. 感谢您的反馈！
  18. 由 Cloudflare 提供性能和安全性支持

这并不是完美的保证，但添加用户代理头部是一个简单的选项，似乎足够避免在当前时间点在该特定站点上被检测为无界面模式：

  1. import {expect, test} from "@playwright/test"; // ^1.30.0
  3. const url = "<Your URL>";
  4. const userAgent =
  5.   "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36";
  7. test.describe("with user agent", () => {
  8.   test.use({userAgent});
  10.   test("is able to retrieve offers", async ({page}) => {
  11.     await page.goto(url);
  12.     const selector = page.locator('[id="Body offers-panel"] li');
  13.     const offers = await selector.count();
  14.     console.log("Num of offers:", offers); // => Num of offers: 11
  15.   });
  16. });
英文:

Headless mode makes it more obvious to servers that your script is a bot. You're being detected and blocked headlessly, but bypassing detection when running headfully.

Since you can't see anything, headless is a bit harder to debug than headful. Using console.log(await page.content()) and await page.screenshot({path: &quot;test.png&quot;}) are good strategies for figuring out why elements you expect to be on the page aren't.

In this case, adding

  1. const text = (await page.textContent(&quot;body&quot;))
  2.   .replace(/ +/g, &quot; &quot;)
  3.   .replace(/(\n ?)+/g, &quot;\n&quot;)
  4.   .trim();
  5. console.log(text);

after goto to get the full text content of the page gives:

  1. Access denied
  2. Error code 1020
  3. You do not have access to &lt;Your URL&gt;.The site owner may have set restrictions that prevent you from accessing the site.
  4. Error details
  5. Provide the site owner this information.
  6. I got an error when visiting &lt;Your URL&gt;.
  7. Error code: 1020
  8. Ray ID: **************
  9. Country: US
  10. Data center: *****
  11. IP: *****************
  12. Timestamp: 2023-02-17 22:39:13 UTC
  13. Click to copy
  14. Was this page helpful?
  15. Yes
  16. No
  17. Thank you for your feedback!
  18. Performance &amp; security by Cloudflare

It's not a perfect guarantee, but adding a user agent header is an easy option that seems to be enough to avoid headless detection on this particular site at this point in time:

  1. import {expect, test} from &quot;@playwright/test&quot;; // ^1.30.0
  3. const url = &quot;&lt;Your URL&gt;&quot;;
  4. const userAgent =
  5.   &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36&quot;;
  7. test.describe(&quot;with user agent&quot;, () =&gt; {
  8.   test.use({userAgent});
  10.   test(&quot;is able to retrieve offers&quot;, async ({page}) =&gt; {
  11.     await page.goto(url);
  12.     const selector = page.locator(&#39;[id=&quot;Body offers-panel&quot;] li&#39;);
  13.     const offers = await selector.count();
  14.     console.log(&quot;Num of offers:&quot;, offers); // =&gt; Num of offers: 11
  15.   });
  16. });

答案3

得分: 1

你好，我正在使用Java与Playwright，遇到了类似的问题 - 除了无头检测之外，行为差异可能还有其他来源吗？我相当确定该网站没有实施无头检测。

英文:

Hello I am using Playwright with Java and have similar problems - is it possible that there are other sources of differences in behavior apart from headless detection? I am pretty sure the site does not have headless detection implemented.

huangapple
  • 本文由 发表于 2023年2月18日 03:55:56
  • 转载请务必保留本文链接：https://go.coder-hub.com/75488727.html
  • opensea
  • playwright
  • typescript
  • web-scraping
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定