英文:
Why am I unable to retrieve the full list of cookies when using Selenium with Python3?
问题
I am trying to use Python3 & selenium to login to a site and extract the cookies so I can use them for a later call. However, I'm unable to retrieve the complete set of cookies.
For example, when I go to the post-authentication page manually, I can see the full list of cookie tokens from the developer tools browser GUI. These are:
csn, cpn, _ga, didomi_token, euconsent-v2, _ga_T0L2RVVYNH, _gid, _gat_gtag_UA_114622452_16, .lpac, XSRF-TOKEN, and Extranet_AL.
If I copy this complete block and use it in a subsequent request (something like)
cookies = {
"cookies" : "_csn_: accepted, _cpn_: accepted....... etc.,"
}
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0",
}
response = requests.get(api_url, headers=headers, cookies=cookies, timeout=2)
This works fine.
However, if I pull the cookies programmatically from selenium,
pickle.dump(driver.get_cookies(), open("session.pkl", "wb"))
then I only get the following tokens:
.lpac, XSRF-TOKEN, Extranet_AL, csn, and cpn
.. and they are insufficient to authenticate with the above requests.get() command.
I have double-checked that selenium is switching to the correct page and I can read the HTML elements, so I'm pretty sure it isn't a hang-up from changing tabs.
Note: I've tried Chrome and Firefox webdrivers... I'm not tied to either but both show the same problem.
Does anyone know how I can get around this? Any suggestions are really appreciated!
英文:
I am trying to use Python3 & selenium to login to a site and extract the cookies so I can use them for a later call. However, I'm unable to retrieve the complete set of cookies.
For example, when I go to the post-authentication page manually, I can see the full list of cookie tokens from the developer tools browser GUI. These are:
csn, cpn, _ga, didomi_token, euconsent-v2, _ga_T0L2RVVYNH, _gid, _gat_gtag_UA_114622452_16, .lpac, XSRF-TOKEN, and Extranet_AL.
If I copy this complete block and use it in a subsequent request (something like)
cookies = {
"cookies" : "_csn_: accepted, _cpn_: accepted....... etc.,"
}
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0",
}
response = requests.get(api_url, headers=headers, cookies=cookies, timeout=2)
This works fine.
However, if I pull the cookies programatically from selenium,
pickle.dump(driver.get_cookies(), open("session.pkl", "wb"))
then I only get the following tokens:
.lpac, XSRF-TOKEN, Extranet_AL, csn, and cpn
.. and they are insufficient to authenticate with the above requests.get() command.
I have double checked that selenium is switching to the correct page and I can read the html elements, so I'm pretty sure it isn't a hang up from changing tabs.
Note: I've tried Chrome and Firefox webdrivers... I'm not tied to either but both show the same problem.
Does anyone know how I can get around this? Any suggestions are really appreciated!
答案1
得分: 1
很可能是基于收集到的 cookie 创建 cookie 字符串时出现了问题。请检查这一点：
cookies_json = self.driver.get_cookies()
cookies_str = ''.join(map(lambda c: '%s=%s; ' % (c['name'], c['value']), cookies_json))
response = requests.get(url, headers={'cookie': cookies_str, 'User-Agent': userAgent})
英文:
Most probably there is an issue creating a cookie string based on the collected cookies. Check this:
cookies_json = self.driver.get_cookies()
cookies_str = ''.join(map(lambda c: '%s=%s; ' % (c['name'], c['value']), cookies_json))
response = requests.get( url, headers={'cookie': cookies_str, 'User-Agent': userAgent})
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论