英文:
Different results between Go's http library and Postman -- Postman works but Go doesn't
问题
你好。我正在尝试使用Go来爬取这个URL:http://us2.m2web.talk2m.com/valleycarriers/Gorman%20Bros/usr/viewon/Overview.shtm
。
在这个过程中出现了问题。我可以像Chrome一样,通过发送带有必要参数的POST请求来解决这个问题,但是即使Chrome这样做了之后,它也会在地址栏上弹出一个浏览器窗口,要求输入用户名和密码。这是因为响应头中的Cache-Clear
是must-revalidate
。
所以我启动了Postman。然后发送了一个类似这样的POST请求:
https`://username:password@url
它返回了三个cookies。我将这三个cookies提供给了我想要获取数据的端点:
https://u:p@us2.m2web.talk2m.com/valleycarriers/Gorman Bros/rcgi.bin/vows/readVars
然后得到了结果!
在发送第一个请求之后,Postman发送了两个连续的请求。
我试图在Go中替换这个过程:
loginString := "account=valleycarriers&username=u&password=p&connect=connect&attempt=0"
requestLogin, err := http.NewRequest("POST", "http://u:p@us2.m2web.talk2m.com/valleycarriers/Gorman%20Bros/usr/viewon/Overview.shtm", strings.NewReader(loginString))
if err != nil {
log.Fatal(err)
}
requestLogin.Header.Add("Content-Type", "application/x-www-form-urlencoded")
requestLogin.Header.Add("Content-Length", "87")
requestLogin.Header.Add("Host", "us2.m2web.talk2m.com")
requestLogin.Header.Add("Authorization", "Basic b3BlcmF0b3I6b3BlcmF0b3IxMjM=")
但是这个请求不起作用。它返回了两个cookies,但是Postman返回了三个。Postman没有返回正文,但是它返回了登陆页面。在Postman中,如果我将第一个请求的cookies传递给第二个请求,它会返回我想要的数据。但是在Go中,它只是再次返回登陆页面。
我试图使用Fiddler检查Go发送的出站流量,但是与Postman不同,Fiddler没有显示来自这个端点的任何流量。
谢谢。
英文:
Good day. I'm trying to scrape this URL with Go:
http://us2.m2web.talk2m.com/valleycarriers/Gorman%20Bros/usr/viewon/Overview.shtm
There's a chink in the chain. I could simply send a POST request to this endopoint with the necessary parameters, just like Chrome does, but even after Chrome does that, it pops up a window on the address bar, a browser window, asking for username and password. Because Cache-Clear
in response header is must-revalidate
.
So I fired up Postman. And sent a POST the same url, but like this:
https`://username:password@url
And it returned three cookies. I fed these three cookies to the endpoint that serves the data I want:
https://u:p@us2.m2web.talk2m.com/valleycarriers/Gorman Bros/rcgi.bin/vows/readVars
And got results!
Postman sends two consecutive requests after sending the first request.
I tried to replace that in Go:
loginString := "account=valleycarriers&username=u&password=p&connect=connect&attempt=0"
requestLogin, err := http.NewRequest("POST", "http://u:p@us2.m2web.talk2m.com/valleycarriers/Gorman%20Bros/usr/viewon/Overview.shtm", strings.NewReader(loginString))
if err != nil {
log.Fatal(err)
}
requestLogin.Header.Add("Content-Type", "application/x-www-form-urlencoded")
requestLogin.Header.Add("Content-Length", "87")
requestLogin.Header.Add("Host", "us2.m2web.talk2m.com")
requestLogin.Header.Add("Authorization", "Basic b3BlcmF0b3I6b3BlcmF0b3IxMjM=")
But this request doesn't work. It returns two cookies, but Postman retursn three. Postman returns no body, but it returns the landing page. In Postman, if I make the pass the cookies from the first request to the second requests, it returns the data I want. But in Go it just returns the landing page, again.
I tried to inspect the outgoing traffic from Go using Fiddler, but unlike Postman, Fiddler doesn't display any traffic from this endpoint.
Thanks.
答案1
得分: 1
如果我能评论的话,我会这样说,我发现Postman中的Code-> {language}选项非常有用。如果你在Postman桌面版中点击“保存”下面的“Code”文本,它会将你的请求的每一部分转换为你选择的语言中通常准确的代码。
在没有看到你的Postman示例、完整的Go代码或错误/状态代码的情况下,很难确定问题出在哪里。这可能是你的问题被投票降低的原因。如果你能发布这些内容,并且Postman自动生成的代码不能解决你的问题,我可以查看具体情况并尝试调试问题,但请先尝试以下步骤。
英文:
If I could comment I would, but I will just say as an answer, I have found the Code->{language} option in Postman very useful. If you click on the "Code" text under "Save" in Postman desktop, it will convert every bit of your request into usually accurate code in your language of choice.
Without seeing your Postman example, full Go code, or error/status codes, it is hard for me to figure out what is wrong. This is likely the reason for downvotes on your question. If you can post these things and the autogenerated code from Postman doesn't solve your problem, I can look at the specific case and try to debug the issue, but try these steps first.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论