英文:
Inserting into firehose with Go
问题
我有以下的JSON文件:
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose."
}
我有以下的Go函数:
func insertIntoFireHose(sess *session.Session, hoseName string) {
svc := firehose.New(sess, aws.NewConfig().WithRegion("us-east-1"))
//firehoseData := getObjectFromS3Bucket(sess)
firehoseData, _ := os.ReadFile("/temp/test.json")
var rec firehose.Record
var recInput firehose.PutRecordInput
dataJson, _ := json.Marshal(firehoseData)
rec.SetData(dataJson)
recInput.SetDeliveryStreamName(hoseName)
recInput.SetRecord(&rec)
res, err1 := svc.PutRecord(&recInput)
if err1 != nil {
log.Fatal(err1)
}
fmt.Println(res)
}
我想要做的是获取一个文件并将其插入到firehose中,但是我得到了以下错误信息:
{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":
{"type":"not_x_content_exception",
"reason":"not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}
我不太确定我做错了什么。
将记录更改为直接从文件获取数据会返回以下错误:
One or more records are malformed. Please ensure that each record is single valid JSON object and that it does not contain newlines.
英文:
I have the following JSON file
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose."
}
And I have the following Go function
func insertIntoFireHose(sess *session.Session, hoseName string) {
svc := firehose.New(sess, aws.NewConfig().WithRegion("us-east-1"))
//firehoseData := getObjectFromS3Bucket(sess)
firehoseData, _ := os.ReadFile("/temp/test.json")
var rec firehose.Record
var recInput firehose.PutRecordInput
dataJson, _ := json.Marshal(firehoseData)
rec.SetData(dataJson)
recInput.SetDeliveryStreamName(hoseName)
recInput.SetRecord(&rec)
res, err1 := svc.PutRecord(&recInput)
if err1 != nil {
log.Fatal(err1)
}
fmt.Println(res)
}
What I want to do is get a file and insert it into the firehose, but I get this error message:
{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":
{"type":"not_x_content_exception",
"reason":"not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}
And I'm not quite sure what I'm doing wrong.
Changing the record to get data directly from the file returns this error:
One or more records are malformed. Please ensure that each record is single valid JSON object and that it does not contain newlines.
答案1
得分: 2
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose."
}
我认为这不是一个有效的JSON,第3行有一个尾随逗号。
这是有效的JSON:
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose."
}
而且firehoseData已经是[]byte
类型,所以我认为你不需要再次使用json.Marshal
。
这是marshal的结果:
代码:
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
firehoseData, _ := os.ReadFile("./file.json") // 相同的值
fmt.Printf("%+v\n", string(firehoseData))
test, err := json.Marshal(firehoseData)
fmt.Printf("%+v\n", string(test))
fmt.Printf("%+v\n", err)
}
输出:
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose.",
}
"ew0KICAgICJAdGltZXN0YW1wIjogIjIwMjEtMTEtMTlUMjE6MzI6NTUuMTk2WiIsDQogICAgIkB2ZXJzaW9uIjogIjEiLA0KICAgICJtZXNzYWdlIjogIk1hbnVhbCB0ZXN0IHRvIGZpcmVob3NlLiIsDQp9"
英文:
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose.",
}
I don't think this is a valid JSON, it has a trailing comma in the 3rd line.
this is the valid one
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose."
}
and firehoseData is already []byte
, so I think you don't need to json.Marshal
it again.
This is the result of the marshal
code:
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
firehoseData, _ := os.ReadFile("./file.json") // same value
fmt.Printf("%+v\n", string(firehoseData))
test, err := json.Marshal(firehoseData)
fmt.Printf("%+v\n", string(test))
fmt.Printf("%+v\n", err)
}
output:
{
"@timestamp": "2021-11-19T21:32:55.196Z",
"@version": "1",
"message": "Manual test to firehose.",
}
"ew0KICAgICJAdGltZXN0YW1wIjogIjIwMjEtMTEtMTlUMjE6MzI6NTUuMTk2WiIsDQogICAgIkB2ZXJzaW9uIjogIjEiLA0KICAgICJtZXNzYWdlIjogIk1hbnVhbCB0ZXN0IHRvIGZpcmVob3NlLiIsDQp9"
<nil>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论