Python Selenium Undetected Chromedriver不适用于带有身份验证代理。

huangapple go评论122阅读模式
英文:

Python Selenium Undetected Chromedriver doesn't work with auth proxy

问题

我使用Undetected Chromedriver来绕过Cloudflare的检测。

我尝试强制它与代理一起工作,但没有成功。
第一次尝试(IP没有更改):

import undetected_chromedriver as uc
options = uc.ChromeOptions()
...
ip, port, username, password = proxy.split(':')   # 代理格式为ip:port:username:password
options.add_argument(f'--proxy-server=http://{username}:{password}@{ip}:{port}')
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)
driver.get('https://2ip.ru/')

第二次尝试(使用Selenium-Wire):

from seleniumwire import undetected_chromedriver as uc
...
ip, port, username, password = proxy.split(':')   # 代理格式为ip:port:username:password
wire_options = {
    'proxy': {
        'http': f'http://{username}:{password}@{ip}:{port}', 
        'https': f'http://{username}:{password}@{ip}:{port}',
        'no_proxy': 'localhost,127.0.0.1'  # 排除的地址
    }  
}
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options, seleniumwire_options=wire_options)
driver.get('https://2ip.ru/')

浏览器尝试使用代理连接,但我需要手动输入密码和用户名。
如果您能指出我的错误或提供其他解决方案,我将不胜感激。

英文:

I use Undetected Chromedriver to bypass Cloudflare detection.

I trying force it to work with proxy, but unsuccessful.
First attempt (IP just doesn't change):

import undetected_chromedriver as uc
options = uc.ChromeOptions()
...
ip,port,username,password = proxy.split(':')   #proxy in format ip:port:username:password
options.add_argument(f'--proxy-server=http://{username}:{password}@{ip}:{port}')
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)
driver.get('https://2ip.ru/')

Second attempt (with Selenium-Wire):

from seleniumwire import undetected_chromedriver as uc
...
ip,port,username,password = proxy.split(':')   #proxy in format ip:port:username:password
wire_options = {
        'proxy': {
            'http': f'http://{username}:{password}@{ip}:{port}', 
            'https': f'http://{username}:{password}@{ip}:{port}',
            'no_proxy': 'localhost,127.0.0.1' # excludes
        }  
    }
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options,seleniumwire_options=wire_options)
driver.get('https://2ip.ru/')

Browser trying to connect with proxy, but I need to manually enter password and username.
I will be thanks, if you show me my mistakes or offer another decision of problem.

答案1

得分: 0

SeleniumBase 具有一个特殊的未检测到-chromedriver模式,可让您通过方法参数(或命令行)设置代理设置。

pip install seleniumbase,然后在填写SB()调用中的代理设置后运行python

from seleniumbase import SB

with SB(uc=True, pls="none", proxy="USER:PASS@HOST:PORT") as sb:
    sb.open("https://nowsecure.nl/#relax")
    sb.assert_text("OH YEAH, you passed!", "h1")
    sb.post_message("Selenium wasn't detected!")

它将uc设置为True(启用未检测到-chromedriver模式),将pls设置为"none"(在该模式下使用不同的pageLoadStrategy可能会冻结代理测试),并设置代理设置(在您填写所需设置后)。

英文:

SeleniumBase has a special undetected-chromedriver mode that lets you set the proxy settings via method arg (or on the command-line).

pip install seleniumbase, then run with python after you fill in the proxy settings on the SB() call:

from seleniumbase import SB

with SB(uc=True, pls="none", proxy="USER:PASS@HOST:PORT") as sb:
    sb.open("https://nowsecure.nl/#relax")
    sb.assert_text("OH YEAH, you passed!", "h1")
    sb.post_message("Selenium wasn't detected!")

It sets uc to True (to enabled undetected-chromedriver mode), it sets pls to "none" (a different pageLoadStrategy might freeze up proxy tests in that mode), and it sets the proxy settings (after you fill in the one you want).

答案2

得分: 0

在互联网上搜索了一整天后,我发现Chrome不支持基本的代理身份验证。感谢[AbdeLhalimSB][1]指引我找到了[正确的解决方案][2]。但是这个解决方案以及其他类似的解决方案对我不起作用,因为我的浏览器拒绝从zip存档加载扩展程序。因此,我需要更改这段代码,以便浏览器可以从文件夹加载扩展程序。

extension.py:

import os

def proxies(username, password, endpoint, port, path):
    try:
        os.mkdir(path)
    except FileExistsError:
        pass
    manifest_json = """
    {
        "version": "1.0.0",
        "manifest_version": 2,
        "name": "Proxies",
        "permissions": [
            "proxy",
            "tabs",
            "unlimitedStorage",
            "storage",
            "<all_urls>",
            "webRequest",
            "webRequestBlocking"
        ],
        "background": {
            "scripts": ["background.js"]
        },
        "minimum_chrome_version": "22.0.0"
    }
    """

    background_js = """
    var config = {
        mode: "fixed_servers",
        rules: {
            singleProxy: {
                scheme: "http",
                host: "%s",
                port: parseInt(%s)
            },
            bypassList: ["localhost"]
        }
    };

    chrome.proxy.settings.set({ value: config, scope: "regular" }, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
                username: "%s",
                password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
        callbackFn,
        { urls: ["<all_urls>"] },
        ['blocking']
    );
    """ % (endpoint, port, username, password)

    with open(path + "\\manifest.json", 'w') as m_file:
        m_file.write(manifest_json)
    with open(path + "\\background.js", 'w') as b_file:
        b_file.write(background_js)

main.py:

from webdriver_manager.chrome import ChromeDriverManager
import undetected_chromedriver as uc
import os
from extension import proxies

options = uc.ChromeOptions()
script_address = os.getcwd()
...
ip, port, username, password = proxy.split(':')  # 代理的格式是ip:端口:用户名:密码
proxies(username, password, ip, int(port), script_address + '\\Proxies\\' + ip)
options.add_argument('--load-extension=' + script_address + '\\Proxies\\' + ip)
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)

请注意,此解决方案在无头模式下可以工作,但在隐身模式下不可以。
[1]: https://stackoverflow.com/users/20112601/abdelhalimsb
[2]: https://stackoverflow.com/questions/76396403/how-can-i-integrate-proxies-into-my-selenium-web-scraper-for-true-people-search/76400335#76400335

英文:

After day of searches on the Internet, I found out that Chrome doesn't support basic proxy authentification. Thank [AbdeLhalimSB][1] for leading me to the [right solution][2]. But this and other similar solutions don't work for me because my browser refuses to load extensions from zip archive. So, I needed to change this code so that browser can load extension from folder.

extension.py:

import os
def proxies(username, password, endpoint, port,path):
    try:
        os.mkdir(path)
    except FileExistsError:
        pass
    manifest_json = &quot;&quot;&quot;
    {
        &quot;version&quot;: &quot;1.0.0&quot;,
        &quot;manifest_version&quot;: 2,
        &quot;name&quot;: &quot;Proxies&quot;,
        &quot;permissions&quot;: [
            &quot;proxy&quot;,
            &quot;tabs&quot;,
            &quot;unlimitedStorage&quot;,
            &quot;storage&quot;,
            &quot;&lt;all_urls&gt;&quot;,
            &quot;webRequest&quot;,
            &quot;webRequestBlocking&quot;
        ],
        &quot;background&quot;: {
            &quot;scripts&quot;: [&quot;background.js&quot;]
        },
        &quot;minimum_chrome_version&quot;:&quot;22.0.0&quot;
    }
    &quot;&quot;&quot;

    background_js = &quot;&quot;&quot;
    var config = {
            mode: &quot;fixed_servers&quot;,
            rules: {
            singleProxy: {
                scheme: &quot;http&quot;,
                host: &quot;%s&quot;,
                port: parseInt(%s)
            },
            bypassList: [&quot;localhost&quot;]
            }
        };

    chrome.proxy.settings.set({value: config, scope: &quot;regular&quot;}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
                username: &quot;%s&quot;,
                password: &quot;%s&quot;
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
                callbackFn,
                {urls: [&quot;&lt;all_urls&gt;&quot;]},
                [&#39;blocking&#39;]
    );
    &quot;&quot;&quot; % (endpoint, port, username, password)

    with open(path+&quot;\\manifest.json&quot;, &#39;w&#39;) as m_file:
        m_file.write(manifest_json)
    with open(path+&quot;\\background.js&quot;, &#39;w&#39;) as b_file:
        b_file.write(background_js)

main.py:

from webdriver_manager.chrome import ChromeDriverManager
import undetected_chromedriver as uc
import os
from extension import proxies
options = uc.ChromeOptions()
script_address = os.getcwd()
...
ip,port,username,password = proxy.split(&#39;:&#39;)   #proxy in format ip:port:username:password
proxies(username, password, ip, int(port),script_address+&#39;\\Proxies\\&#39;+ip)
options.add_argument(&#39;--load-extension=&#39;+script_address+&#39;\\Proxies\\&#39;+ip)
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)

Note, this solution will work in headless mode, but not in incognito mode.
[1]: https://stackoverflow.com/users/20112601/abdelhalimsb
[2]: https://stackoverflow.com/questions/76396403/how-can-i-integrate-proxies-into-my-selenium-web-scraper-for-true-people-search/76400335#76400335

huangapple
  • 本文由 发表于 2023年6月25日 21:44:06
  • 转载请务必保留本文链接:https://go.coder-hub.com/76550705.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定