Python requests works fine, when trying same request in golang is not working as expected

huangapple go评论76阅读模式
英文:

Python requests works fine, when trying same request in golang is not working as expected

问题

以下是翻译好的内容:

Python代码,记住要使__cf_bm cookie有效才能获得JSON响应。__cf_bm只在大约30分钟内有效,所以你必须自己从https://kick.com获取一个cookie来复制这个示例。

import requests

headers = {
'Accept':'application/json',
'Cookie':'__cf_bm=yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy',
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

resp = requests.request('GET', 'https://kick.com/api/v1/channels/xqc', headers=headers)
print (resp.text)

用户代理与cookie相关联,请确保使用获取cookie时使用的用户代理,而不是示例中的用户代理。accept: application/json并不是很重要,可以省略。

问题是当我使用相同的标头、cookie和用户代理在golang中进行相同的请求时,我得到了一个cloudflare响应(当__cf_bm cookie已过期时,也会得到相同的响应),但我知道cookie还没有过期。

Golang代码,go版本go1.20.5 linux/amd64

package main

import (
	"compress/gzip"
	"fmt"
	"io"
	"io/ioutil"
	"net/http"
)

func MakeRequest(url string) (*string, error) {
	req, _ := http.NewRequest("GET", url, nil)
	req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
	req.Header.Set("Accept", "application/json")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Accept-Encoding", "gzip, deflate")
	req.AddCookie(&http.Cookie{Name: "__cf_bm", Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy"})

	client := &http.Client{}
	resp, err := client.Do(req)
	var body string

	if err != nil {
		return &body, err
	}

	defer resp.Body.Close()

	var reader io.ReadCloser
	reader, _ = gzip.NewReader(resp.Body)
	bytes, _ := ioutil.ReadAll(reader)
	body = string(bytes)
	return &body, nil
}

func main() {
	data, _ := MakeRequest("https://kick.com/api/v1/channels/xqc")
	fmt.Println(*data)
}

我添加了Connection和Accept-Encoding标头,这是Python请求默认添加的,但是Golang没有添加。

Python和Golang中的标头是相同的,但响应不同,我做错了什么,还是Golang发送请求的方式不同。我看不出我做错了什么...

英文:

Python code, keep in mind that the __cf_bm cookie has to be valid in order to get a json response. The __cf_bm is only valid for around 30min, so you have to get a cookie from https://kick.com on your own to replicate this example.

import requests

headers = {
'Accept':'application/json',
'Cookie':'__cf_bm=yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy',
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}


resp = requests.request('GET', 'https://kick.com/api/v1/channels/xqc', headers=headers)
print (resp.text)

The user-agent is tied to the cookie, make sure to use your user agent with which you got your cookie, not the one in the example. The accept: application/json is not that important, can be omitted.

The problem is when i make the SAME request in golang with the same headers and cookies and user-agent, that i get a cloudflare response, (which is the same response, when the __cf_bm cookie has expired) but i know that the cookie has not expired yet.

Golang Code, go version go1.20.5 linux/amd64

package main

import (
	"compress/gzip"
	"fmt"
	"io"
	"io/ioutil"
	"net/http"
)

func MakeRequest(url string) (*string, error) {
	req, _ := http.NewRequest("GET", url, nil)
	req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
	req.Header.Set("Accept", "application/json")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Accept-Encoding", "gzip, deflate")
	req.AddCookie(&http.Cookie{Name: "__cf_bm", Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy"})

	client := &http.Client{}
	resp, err := client.Do(req)
	var body string

	if err != nil {
		return &body, err
	}

	defer resp.Body.Close()

	var reader io.ReadCloser
	reader, _ = gzip.NewReader(resp.Body)
	bytes, _ := ioutil.ReadAll(reader)
	body = string(bytes)
	return &body, nil
}

func main() {
	data, _ := MakeRequest("https://kick.com/api/v1/channels/xqc")
	fmt.Println(*data)
}

I added Connection and Accept-Encoding header which python requests adds by default but golang doesn't.

The headers are the same in python and golang but the response is not, i'm doing something wrong or is the way golang sends the request. I can't see what im doing wrong...

答案1

得分: 1

gzip模块是否需要?

bytes, _ := ioutil.ReadAll(resp.Body)
body = string(bytes)
return &body, nil
英文:

Is the gzip mod needed?

bytes, _ := ioutil.ReadAll(resp.Body)
body = string(bytes)
return &body, nil

答案2

得分: -1

首先,上面的评论者是正确的,这里不需要使用Gzip模块。
接下来的错误是:如果你创建一个未初始化的bytes.Buffer类型的变量,并将其赋值给io.Reader类型的字段,那么在检查io.Reader是否为nil之后,会出现错误:无效的内存地址或空指针解引用。更多信息请参考如何正确检查io.Reader是否为nil?

正确的代码:

func main() {
    request, err := http.Get("https://kick.com/api/v1/channels/xqc")
    if err != nil {
        log.Fatal(err)
    }
    myCookie := &http.Cookie{
        Name:  "__cf_bm",
        Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy",
    }
    request.Request.AddCookie(myCookie)
    request.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
    request.Header.Set("Accept", "application/json")
    request.Header.Set("Connection", "keep-alive")

    body, err := io.ReadAll(request.Body)
    if err != nil {
        fmt.Println(err)
    }
    defer request.Body.Close()
    st := string(body)
    fmt.Println(st)
}
英文:

Firstly, the commentator above was right.Gzip mod not needed here.
Next the mistake was : if you create a variable of type bytes.Buffer (without initialization) and assign it to a field of type io.Reader, then after checking io.Reader for nil there will be an error: invalid memory address or nil pointer dereference. More about that How to correctly check io.Reader for nil?

The right code:

func main() {
request, err := http.Get("https://kick.com/api/v1/channels/xqc")
if err != nil {
	log.Fatal(err)
}
myCookie := &http.Cookie{
	Name:  "__cf_bm",
	Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy",
}
request.Request.AddCookie(myCookie)
request.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
request.Header.Set("Accept", "application/json")
request.Header.Set("Connection", "keep-alive")

body, err := io.ReadAll(request.Body)
if err != nil {
	fmt.Println(err)
}
defer request.Body.Close()
st := string(body)
fmt.Println(st)
}

huangapple
  • 本文由 发表于 2023年6月19日 10:11:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/76503246.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定