从API中流式传输数据

huangapple go评论106阅读模式
英文:

Stream data from API

问题

我正在尝试从一个名为Mailhog的电子邮件测试工具的API中获取邮件数据。如果我使用以下调用来获取邮件列表:
GET /api/v1/messages
我可以将这些数据加载到一个结构体中,并打印出我需要的值。

然而,如果我使用一个不同的端点来获取新邮件的流,我会遇到不同的行为。当我运行我的Go应用程序时,我没有任何输出。

我需要像使用while循环一样不断监听端点以获取输出吗?我的最终目标是在邮件到达时提取一些信息,然后将它们传递给另一个函数。

这是我尝试访问流端点的代码:

res, err := http.Get("http://localhost:8084/api/v1/events")

if err != nil {
	panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)

if err != nil {
	panic(err.Error())
}

var data Email
json.Unmarshal(body, &data)
fmt.Printf("Email: %v\n", data)

如果我在Mailhog服务上使用相同的端点进行curl请求,我会在邮件到达时得到输出。然而,我无法弄清楚为什么我的Go应用程序没有输出。应用程序确实保持运行,只是我没有得到任何输出。

我是Go的新手,如果这是一个非常简单的问题,请原谅我。

英文:

I am trying to pull data on mails coming into an API from an email testing tool mailhog.
If I use a call to get a list of emails e.g
GET /api/v1/messages
I can load this data into a struct with no issues and print out values I need.

However if I use a different endpoint that is essentially a stream of new emails coming in, I have different behavior. When I run my go application I get no output whatsoever.

Do I need to do like a while loop to constantly listen to the endpoint to get the output?
My end goal is to pull some information from emails as they come in and then pass them into a different function.

Here is me trying to access the streaming endpoint
https://github.com/mailhog/MailHog/blob/master/docs/APIv1.md

res, err := http.Get("http://localhost:8084/api/v1/events")

if err != nil {
	panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)

if err != nil {
	panic(err.Error())
}

var data Email
json.Unmarshal(body, &data)
fmt.Printf("Email: %v\n", data)

If I do a curl request at the mailhog service with the same endpoint, I do get output as mails come in. However I cant seem to figure out why I am getting no output via my go app. The app does stay running just I dont get any output.

I am new to Go so apologies if this is a really simple question

答案1

得分: 1

来自ioutil.ReadAll文档:

> ReadAll 从 r 中读取直到出现错误或EOF,并返回它读取的数据。

当你用它来读取常规端点的主体时,它能正常工作,因为有效载荷有一个EOF:服务器使用头部的Content-Length来告诉主体响应有多少字节,一旦客户端读取了这么多字节,它就知道已经读取了整个主体并可以停止。

但是,你的“流式”端点不使用Content-Length,因为主体的大小是未知的,它应该在事件到达时写入,所以在这种情况下不能使用ReadAll。通常,在这种情况下,你应该逐行读取,其中每一行表示一个事件。bufio.Scanner正是这样做的:

res, err := http.Get("http://localhost:8084/api/v1/events")
if err != nil {
    panic(err.Error())
}
scanner := bufio.NewScanner(res.Body)
for scanner.Scan() {
	if err := scanner.Err(); err != nil {
		panic(err.Error())
	}
	event := scanner.Bytes()
    var data Email
    json.Unmarshal(event, &data)
    fmt.Printf("Email: %v\n", data)
}

curl可以按照你的期望处理响应,因为它检查端点是否会流式传输数据,所以它会相应地做出反应。将curl获取的响应添加到问题中可能会有帮助。

英文:

From ioutil.ReadAll documentation:

> ReadAll reads from r until an error or EOF and returns the data it read.

When you use to read the body of a regular endpoint, it works because the payload has an EOF: the server uses the header Content-Length to tell how many bytes the body response has, and once the client read that many bytes, it understands that it has read all of the body and can stop.

Your "streaming" endpoint doesn't use Content-Length though, because the body has an unknown size, it's supposed to write events as they come, so you can't use ReadAll in this case. Usually, in this case, you are supposed to read line-by-line, where each line represents an event. bufio.Scanner does exactly that:

res, err := http.Get("http://localhost:8084/api/v1/events")
if err != nil {
    panic(err.Error())
}
scanner := bufio.NewScanner(res.Body)
for e.scanner.Scan() {
	if err := e.scanner.Err(); err != nil {
		panic(err.Error())
	}
	event := e.scanner.Bytes()
    var data Email
    json.Unmarshal(event, &data)
    fmt.Printf("Email: %v\n", data)
}

curl can process the response as you expect because it checks that the endpoint will stream data, so it reacts accordinly. It may be helpful to add the response curl gets to the question.

huangapple
  • 本文由 发表于 2022年7月4日 21:44:33
  • 转载请务必保留本文链接:https://go.coder-hub.com/72857740.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定