未能在发出带参数的GET请求后生成200状态。

huangapple
go评论107阅读模式
英文:

Failed to produce a 200 status after issuing a get requests with parameters

问题

I'm trying to send GET HTTP requests to a webpage and receive a 200 status in response. It is necessary to send appropriate parameters with the link while issuing requests. However, when I send a requests using the link and payload, I always receive a 410 status code.

When I use the URL copied from the dev tools directly in the following code, I get a 200 status.

Please note that I've modified the values of AnchorProdSetId and ProdSetId before sending payload with the link.

How can I get a 200 (403 when detected) status after issuing requests using the link with payload?

  1. import json
  2. import requests
  4. complete_url_from_d_tools = 'https://www.mcmaster.com/mv1687879695/WebParts/Content/ContentWebPart.aspx?cntnrIDtxt=ProdPageContent&srchidtxt=5875341805879&cntnrwdth=769&srchrslttxt=screws&GrpUsrInps=[{%22AnchorProdSetId%22:%22-2674186034712253%22,%22AnchorStateIsSet%22:true,%22ClearAllProdSetId%22:-1,%22ClearAllStateIsSet%22:true,%22GrpEID%22:%22%22,%22ProdSetId%22:%22-2674186034712253%22}]&viewporthgt=192&cssAlias=undefined&usedynamicimageswapping=true&conditionalclientlandingpage=true&forcemdmfiltering=true&workstationdata=true&fwdSsToNav=true&fwdSsWpNav=true&fwdCrcWpNav=true&sortRowsOnServer=true&showbroadlandingpagecopy=true&filtSsAttrEarly=true&legImgContainsHash=true&envrmgrcharsetind=3&useEs6=true'
  6. link = 'https://www.mcmaster.com/mv1687538569/WebParts/Content/ContentWebPart.aspx'
  8. headers = {
  9.     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
  10.     'Accept': '*/*',
  11.     'Accept-Encoding': 'gzip, deflate, br',
  12.     'Accept-Language': 'en-US,en;q=0.9,bn;q=0.8',
  13.     'Referer': 'https://www.mcmaster.com/products/screws/',
  14.     'X-Requested-With': 'XMLHttpRequest',
  15. }
  16. payload = {
  17.     'cntnrIDtxt': 'ProdPageContent',
  18.     'srchidtxt': '5875341805879',
  19.     'cntnrwdth': '769',
  20.     'srchrslttxt': 'screws',
  21.     'GrpUsrInps': '[{"AnchorProdSetId":"","AnchorStateIsSet":"true","ClearAllProdSetId":-1,"ClearAllStateIsSet":"true","GrpEID":"","ProdSetId":""}]',
  22.     'viewporthgt': '192',
  23.     'cssAlias': 'undefined',
  24.     'usedynamicimageswapping': 'true',
  25.     'conditionalclientlandingpage': 'true',
  26.     'forcemdmfiltering': 'true',
  27.     'workstationdata': 'true',
  28.     'fwdSsToNav': 'true',
  29.     'fwdSsWpNav': 'true',
  30.     'fwdCrcWpNav': 'true',
  31.     'sortRowsOnServer': 'true',
  32.     'showbroadlandingpagecopy': 'true',
  33.     'filtSsAttrEarly': 'true',
  34.     'legImgContainsHash': 'true',
  35.     'envrmgrcharsetind': '3',
  36.     'useEs6': 'true',
  37. }
  39. payload['GrpUsrInps'] = json.loads(payload['GrpUsrInps'])
  40. payload['GrpUsrInps'][0]['AnchorProdSetId'] = '-2674186034712253'
  41. payload['GrpUsrInps'][0]['ProdSetId'] = '-2674186034712253'
  42. payload['GrpUsrInps'] = json.dumps(payload['GrpUsrInps'])
  44. with requests.Session() as s:
  45.     s.headers.update(headers)
  46.     # res = s.get(complete_url_from_d_tools)
  47.     res = s.get(link, params=payload)
  48.     print(res.status_code)

(Note: The code provided is in English. If you need a translation into Chinese, please let me know.)

英文:

I'm trying to send GET HTTP requests to a webpage and receive a 200 status in response. It is necessary to send appropriate parameters with the link while issuing requests. However, when I send a requests using the link and payload, I always receive a 410 status code.

When I use the URL copied from the dev tools directly in the following code, I get a 200 status.

Please note that I've modified the values of AnchorProdSetId and ProdSetId before sending payload with the link.

How can I get a 200 (403 when detected) status after issuing requests using the link with payload?

  1. import json
  2. import requests
  3. complete_url_from_d_tools = 'https://www.mcmaster.com/mv1687879695/WebParts/Content/ContentWebPart.aspx?cntnrIDtxt=ProdPageContent&srchidtxt=5875341805879&cntnrwdth=769&srchrslttxt=screws&GrpUsrInps=[{%22AnchorProdSetId%22:%22-2674186034712253%22,%22AnchorStateIsSet%22:true,%22ClearAllProdSetId%22:-1,%22ClearAllStateIsSet%22:true,%22GrpEID%22:%22%22,%22ProdSetId%22:%22-2674186034712253%22}]&viewporthgt=192&cssAlias=undefined&usedynamicimageswapping=true&conditionalclientlandingpage=true&forcemdmfiltering=true&workstationdata=true&fwdSsToNav=true&fwdSsWpNav=true&fwdCrcWpNav=true&sortRowsOnServer=true&showbroadlandingpagecopy=true&filtSsAttrEarly=true&legImgContainsHash=true&envrmgrcharsetind=3&useEs6=true'
  4. link = 'https://www.mcmaster.com/mv1687538569/WebParts/Content/ContentWebPart.aspx'
  5. headers = {
  6. 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
  7. 'Accept': '*/*',
  8. 'Accept-Encoding': 'gzip, deflate, br',
  9. 'Accept-Language': 'en-US,en;q=0.9,bn;q=0.8',
  10. 'Referer': 'https://www.mcmaster.com/products/screws/',
  11. 'X-Requested-With': 'XMLHttpRequest',
  12. }
  13. payload = {
  14. 'cntnrIDtxt': 'ProdPageContent',
  15. 'srchidtxt': '5875341805879',
  16. 'cntnrwdth': '769',
  17. 'srchrslttxt': 'screws',
  18. 'GrpUsrInps': '[{"AnchorProdSetId":"","AnchorStateIsSet":"true","ClearAllProdSetId":-1,"ClearAllStateIsSet":"true","GrpEID":"","ProdSetId":""}]',
  19. 'viewporthgt': '192',
  20. 'cssAlias': 'undefined',
  21. 'usedynamicimageswapping': 'true',
  22. 'conditionalclientlandingpage': 'true',
  23. 'forcemdmfiltering': 'true',
  24. 'workstationdata': 'true',
  25. 'fwdSsToNav': 'true',
  26. 'fwdSsWpNav': 'true',
  27. 'fwdCrcWpNav': 'true',
  28. 'sortRowsOnServer': 'true',
  29. 'showbroadlandingpagecopy': 'true',
  30. 'filtSsAttrEarly': 'true',
  31. 'legImgContainsHash': 'true',
  32. 'envrmgrcharsetind': '3',
  33. 'useEs6': 'true',
  34. }
  35. payload['GrpUsrInps'] = json.loads(payload['GrpUsrInps'])
  36. payload['GrpUsrInps'][0]['AnchorProdSetId'] = '-2674186034712253'
  37. payload['GrpUsrInps'][0]['ProdSetId'] = '-2674186034712253'
  38. payload['GrpUsrInps'] = json.dumps(payload['GrpUsrInps'])
  39. with requests.Session() as s:
  40. s.headers.update(headers)
  41. # res = s.get(complete_url_from_d_tools)
  42. res = s.get(link,params=payload)
  43. print(res.status_code)

答案1

得分: 1

看起来这个服务器返回的是brotli压缩响应。要从服务器获取200响应，请安装brotli包：

  1. pip install brotli

然后：

  1. import requests
  3. url = "https://www.mcmaster.com/mv1687879695/WebParts/Content/ContentWebPart.aspx?cntnrIDtxt=ProdPageContent&srchidtxt=5875341805879&cntnrwdth=1001&srchrslttxt=screws&GrpUsrInps=%5B%7B%22AnchorProdSetId%22%3A%22-3383254440195390%22%2C%22AnchorStateIsSet%22%3Atrue%2C%22ClearAllProdSetId%22%3A-1%2C%22ClearAllStateIsSet%22%3Atrue%2C%22GrpEID%22%3A%22%22%2C%22ProdSetId%22%3A%22-3383254440195390%22%7D%5D&viewporthgt=296&cssAlias=undefined&usedynamicimageswapping=true&conditionalclientlandingpage=true&forcemdmfiltering=true&workstationdata=true&fwdSsToNav=true&fwdSsWpNav=true&fwdCrcWpNav=true&sortRowsOnServer=true&showbroadlandingpagecopy=true&filtSsAttrEarly=true&legImgContainsHash=true&clientNavigationEvents=%5B%5D&envrmgrcharsetind=5&useEs6=true"
  4. r = requests.get(url)
  5. print(r.status_code)

打印：

  1. 200

编辑：要获取“Socket Head Screws”的内容：

  1. import requests
  3. AnchorProdSetId = '-3383254440195390'
  4. ProdSetId = '-3383254440195390'
  6. params = {
  7.     "cntnrIDtxt": "ProdPageContent",
  8.     "srchidtxt": "5875341805879",
  9.     "cntnrwdth": "1001",
  10.     "srchrslttxt": "screws",
  11.     "GrpUsrInps": '[{"AnchorProdSetId":"' + AnchorProdSetId + '","AnchorStateIsSet":true,"ClearAllProdSetId":-1,"ClearAllStateIsSet":true,"GrpEID":"","ProdSetId":"' + ProdSetId + '"}]',
  12.     "viewporthgt": "296",
  13.     "cssAlias": "undefined",
  14.     "usedynamicimageswapping": "true",
  15.     "conditionalclientlandingpage": "true",
  16.     "forcemdmfiltering": "true",
  17.     "workstationdata": "true",
  18.     "fwdSsToNav": "true",
  19.     "fwdSsWpNav": "true",
  20.     "fwdCrcWpNav": "true",
  21.     "sortRowsOnServer": "true",
  22.     "showbroadlandingpagecopy": "true",
  23.     "filtSsAttrEarly": "true",
  24.     "legImgContainsHash": "true",
  25.     "clientNavigationEvents": "[]",
  26.     "envrmgrcharsetind": "5",
  27.     "useEs6": "true",
  28. }
  30. url = 'https://www.mcmaster.com/mv1687879695/WebParts/Content/ContentWebPart.aspx'
  31. r = requests.get(url, params=params)
  33. print(r.text)
英文:

It seems that this server returns brotli compressed response. To get 200 response from the server, install brotli package:

  1. pip install brotli

Then:

  1. import requests
  2. url = "https://www.mcmaster.com/mv1687879695/WebParts/Content/ContentWebPart.aspx?cntnrIDtxt=ProdPageContent&srchidtxt=5875341805879&cntnrwdth=1001&srchrslttxt=screws&GrpUsrInps=%5B%7B%22AnchorProdSetId%22%3A%22-3383254440195390%22%2C%22AnchorStateIsSet%22%3Atrue%2C%22ClearAllProdSetId%22%3A-1%2C%22ClearAllStateIsSet%22%3Atrue%2C%22GrpEID%22%3A%22%22%2C%22ProdSetId%22%3A%22-3383254440195390%22%7D%5D&viewporthgt=296&cssAlias=undefined&usedynamicimageswapping=true&conditionalclientlandingpage=true&forcemdmfiltering=true&workstationdata=true&fwdSsToNav=true&fwdSsWpNav=true&fwdCrcWpNav=true&sortRowsOnServer=true&showbroadlandingpagecopy=true&filtSsAttrEarly=true&legImgContainsHash=true&clientNavigationEvents=%5B%5D&envrmgrcharsetind=5&useEs6=true"
  3. r = requests.get(url)
  4. print(r.status_code)

Prints:

  1. 200

EDIT: To get content for "Socket Head Screws":

  1. import requests
  3. AnchorProdSetId = '-3383254440195390'
  4. ProdSetId = '-3383254440195390'
  6. params = {
  7.     "cntnrIDtxt": "ProdPageContent",
  8.     "srchidtxt": "5875341805879",
  9.     "cntnrwdth": "1001",
  10.     "srchrslttxt": "screws",
  11.     "GrpUsrInps": '[{"AnchorProdSetId":"' + AnchorProdSetId + '","AnchorStateIsSet":true,"ClearAllProdSetId":-1,"ClearAllStateIsSet":true,"GrpEID":"","ProdSetId":"' + ProdSetId + '"}]',
  12.     "viewporthgt": "296",
  13.     "cssAlias": "undefined",
  14.     "usedynamicimageswapping": "true",
  15.     "conditionalclientlandingpage": "true",
  16.     "forcemdmfiltering": "true",
  17.     "workstationdata": "true",
  18.     "fwdSsToNav": "true",
  19.     "fwdSsWpNav": "true",
  20.     "fwdCrcWpNav": "true",
  21.     "sortRowsOnServer": "true",
  22.     "showbroadlandingpagecopy": "true",
  23.     "filtSsAttrEarly": "true",
  24.     "legImgContainsHash": "true",
  25.     "clientNavigationEvents": "[]",
  26.     "envrmgrcharsetind": "5",
  27.     "useEs6": "true",
  28. }
  30. url = 'https://www.mcmaster.com/mv1687879695/WebParts/Content/ContentWebPart.aspx'
  31. r = requests.get(url, params=params)
  33. print(r.text)

huangapple
  • 本文由 发表于 2023年6月29日 04:47:05
  • 转载请务必保留本文链接：https://go.coder-hub.com/76576603.html
  • python
  • python-3.x
  • python-requests
  • web-scraping
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定