使用Golang Beats获取Elasticsearch原始JSON数据。

huangapple go评论94阅读模式
英文:

Elasticsearch return raw JSON with Golang Beats

问题

我正在处理一个与我们的 Elasticsearch 进行交互的节拍,目标是获取 Elasticsearch 中的元素并对其进行修改,以避免重复。

我已经编写了一个搜索功能,它可以正常工作:

client, err := elasticsearch.NewClient(elasticsearch.ClientSettings{
	URL:      host,
	Index:    indexSel,
	Username: username,
	Password: password,
	Timeout:  60 * time.Second,
}, nil)
if err != nil {
	log.Fatal(err)
}

params := map[string]string{
	"q": "_id:" + maref,
}
_, resp, err := client.SearchURI(index, "", params)

if err != nil {
	log.Fatal(err)
	return
} else {
	fmt.Println(string(resp))
}

这是来自 Elasticsearch api.go 的函数:

func (es *Connection) SearchURI(index string, docType string, params map[string]string) (int, *SearchResults, error) {
	status, resp, err := es.apiCall("GET", index, docType, "_search", "", params, nil)
	if err != nil {
		return status, nil, err
	}
	result, err := readSearchResult(resp)
	return status, result, err
}

func readSearchResult(obj []byte) (*SearchResults, error) {
	var result SearchResults
	if obj == nil {
		return nil, nil
	}

	err := json.Unmarshal(obj, &result)
	if err != nil {
		return nil, err
	}
	return &result, err
}

以下是相关类型的定义:

type SearchResults struct {
	Took   int                        `json:"took"`
	Shards json.RawMessage            `json:"_shards"`
	Hits   Hits                       `json:"hits"`
	Aggs   map[string]json.RawMessage `json:"aggregations"`
}

type Hits struct {
	Total int
	Hits  []json.RawMessage `json:"hits"`
}

目前,答案是一个原始的 JSON:

&{11 [123 34 116 111 116 97 108 34 58 53 44 34 115 117 99 99 101 115 115 102 117 108 34 58 53 44 34 102 97 105 108 101 100 34 58 48 125] {1 [[123 34 95 105 110 100 101 120 34 58 34 99 111 112 105 108 98 101 97 116 45 50 48 49 54 46 49 48 46 50 53 34 44 34 95 116 121 112 101 34 58 34 73 110 99 105 100 101 110 116 34 44 34 95 105 100 34 58 34 65 86 102 54 55 69 103 109 74 66 45 119 85 116 103 82 99 90 113 97 34 44 34 95 115 99 111 114 101 34 58 49 46 48 44 34 95 115 111 117 114 99 101 34 58 123 34 64 116 105 109 101 115 116 97 109 112 34 58 34 50 48 49 54 45 49 48 45 50 53 84 48 56 58 49 54 58 53 55 46 53 53 57 90 34 44 34 97 103 101 110 116 95 105 100 34 58 49 53 44 34 98 101 97 116 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 53 55 99 53 99 49 57 49 101 48 100 57 34 44 34 110 97 109 101 34 58 34 53 55 99 53 99 49 57 49 101 48 100 57 34 125 44 34 100 101 115 99 114 105 112 116 105 111 110 34 58 34 101 108 97 115 116 105 99 34 44 34 102 105 110 97 108 99 108 97 115 115 34 58 34 34 44 34 105 100 34 58 34 73 45 48 48 53 56 50 54 34 44 34 111 114 103 95 105 100 34 58 49 55 44 34 111 114 103 95 110 97 109 101 34 58 34 83 104 97 109 34 44 34 112 114 105 111 114 105 116 121 34 58 52 44 34 114 101 102 101 114 101 110 99 101 34 58 34 73 45 48 48 53 56 50 54 34 44 34 115 116 97 114 116 95 100 97 116 101 34 58 34 50 48 49 54 45 49 48 45 50 53 84 49 48 58 49 54 58 53 54 46 56 56 49 55 55 52 49 43 48 50 58 48 48 34 44 34 115 116 97 116 117 115 34 58 34 97 115 115 105 103 110 101 100 34 44 34 116 116 111 34 58 48 44 34 116 116 111 95 49 48 48 95 116 114 105 103 103 101 114 101 100 34 58 48 44 34 116 116 114 34 58 48 44 34 116 116 114 95 49 48 48 95 116 114 105 103 103 101 114 101 100 34 58 48 44 34 116 121 112 101 34 58 34 73 110 99 105 100 101 110 116 34 125 125]]} map[]}

所以我的问题是:

如何将结果转换为 ASCII?
我读到可能是由于 json.RawMessage 类型不会转换 JSON,但我找不到如何进行转换。

提前感谢,

Plosxh。

编辑:解决方案:

感谢 icza,这是我的解决方案,我将 fmt.Println(resp) 更改为:

for _, v := range resp.Hits.Hits {
	fmt.Println(string(v))
}

它完美地工作!

编辑2:
我不得不重新创建一个完整的类型来解析 resp.Hits.Hits,以便像普通的 JSON 一样使用它。

英文:

I'm working on a beat that interrogate our elastic search, the goal is to get elements in elastics to modify them to avoid duplication.

I've written a search and it works:

client,err :=elasticsearch.NewClient(elasticsearch.ClientSettings{URL:host,Index:indexSel,Username: username,Password: password,Timeout:  60 * time.Second,}, nil)
if err != nil {
log.Fatal(err)
}
params := map[string]string{
"q": "_id:"+maref,
}
_, resp, err := client.SearchURI(index, "", params)
if err != nil {
log.Fatal(err)
return
} else {
fmt.Println(string(resp))
}

And here is the function from the elastic api.go:

func (es *Connection) SearchURI(index string, docType string, params map[string]string) (int, *SearchResults, error) {
status, resp, err := es.apiCall("GET", index, docType, "_search", "", params, nil)
if err != nil {
return status, nil, err
}
result, err := readSearchResult(resp)
return status, result, err
}
func readSearchResult(obj []byte) (*SearchResults, error) {
var result SearchResults
if obj == nil {
return nil, nil
}
err := json.Unmarshal(obj, &result)
if err != nil {
return nil, err
}
return &result, err
}

With the following types:

type SearchResults struct {
Took   int                        `json:"took"`
Shards json.RawMessage            `json:"_shards"`
Hits   Hits                       `json:"hits"`
Aggs   map[string]json.RawMessage `json:"aggregations"`
}
type Hits struct {
Total int
Hits  []json.RawMessage `json:"hits"`
}

For now the answer is a raw JSON:

&{11 [123 34 116 111 116 97 108 34 58 53 44 34 115 117 99 99 101 115 115 102 117 108 34 58 53 44 34 102 97 105 108 101 100 34 58 48 125] {1 [[123 34 95 105 110 100 101 120 34 58 34 99 111 112 105 108 98 101 97 116 45 50 48 49 54 46 49 48 46 50 53 34 44 34 95 116 121 112 101 34 58 34 73 110 99 105 100 101 110 116 34 44 34 95 105 100 34 58 34 65 86 102 54 55 69 103 109 74 66 45 119 85 116 103 82 99 90 113 97 34 44 34 95 115 99 111 114 101 34 58 49 46 48 44 34 95 115 111 117 114 99 101 34 58 123 34 64 116 105 109 101 115 116 97 109 112 34 58 34 50 48 49 54 45 49 48 45 50 53 84 48 56 58 49 54 58 53 55 46 53 53 57 90 34 44 34 97 103 101 110 116 95 105 100 34 58 49 53 44 34 98 101 97 116 34 58 123 34 104 111 115 116 110 97 109 101 34 58 34 53 55 99 53 99 49 57 49 101 48 100 57 34 44 34 110 97 109 101 34 58 34 53 55 99 53 99 49 57 49 101 48 100 57 34 125 44 34 100 101 115 99 114 105 112 116 105 111 110 34 58 34 101 108 97 115 116 105 99 34 44 34 102 105 110 97 108 99 108 97 115 115 34 58 34 34 44 34 105 100 34 58 34 73 45 48 48 53 56 50 54 34 44 34 111 114 103 95 105 100 34 58 49 55 44 34 111 114 103 95 110 97 109 101 34 58 34 83 104 97 109 34 44 34 112 114 105 111 114 105 116 121 34 58 52 44 34 114 101 102 101 114 101 110 99 101 34 58 34 73 45 48 48 53 56 50 54 34 44 34 115 116 97 114 116 95 100 97 116 101 34 58 34 50 48 49 54 45 49 48 45 50 53 84 49 48 58 49 54 58 53 54 46 56 56 49 55 55 52 49 43 48 50 58 48 48 34 44 34 115 116 97 116 117 115 34 58 34 97 115 115 105 103 110 101 100 34 44 34 116 116 111 34 58 48 44 34 116 116 111 95 49 48 48 95 116 114 105 103 103 101 114 101 100 34 58 48 44 34 116 116 114 34 58 48 44 34 116 116 114 95 49 48 48 95 116 114 105 103 103 101 114 101 100 34 58 48 44 34 116 121 112 101 34 58 34 73 110 99 105 100 101 110 116 34 125 125]]} map[]}

So my question is:

How can I convert the result to ascii?
I've read it's probably due to the json.RawMessage type that doesn't convert JSON but I can't find how to convert it.

Thanks by advance,

Plosxh.

EDIT : Solution:

Thx to icza, here's my solution I change my "fmt.Println(resp)" into:

for _, v := range resp.Hits.Hits {
fmt.Println(string(v))
}

It works like a charm!

Edit 2 :
I had to re-create a whole type en json.Unmarshal "resp.Hits.Hits" in order to use it like a regular json.

答案1

得分: 1

json.RawMessage 是一个字节切片:

type RawMessage []byte

你可以通过简单的类型转换将字节切片转换为 string

var result SearchResults
// ...
for _, v := range result.Hits {
fmt.Println("Total:", v.Total, "Hits:", string(v.Hits))
}

或者:

fmt.Printf("Total: %d, Hits: %s\n", v.Total, v.Hits)

参考这个例子:

var raw json.RawMessage
raw = json.RawMessage(`something raw`)
fmt.Println(raw)
fmt.Println(string(raw))
fmt.Printf("%s\n", raw)

输出结果(在 Go Playground 上尝试):

[115 111 109 101 116 104 105 110 103 32 114 97 119]
something raw
something raw
英文:

json.RawMessage is a byte slice:

type RawMessage []byte

You can convert a byte slice to a string with a simple type conversion:

var result SearchResults
// ...
for _, v := range result.Hits {
fmt.Println("Total:", v.Total, "Hits:", string(v.Hits))
}

Or:

fmt.Printf("Total: %d, Hits: %s\n", v.Total, v.Hits)

See this example:

var raw json.RawMessage
raw = json.RawMessage(`something raw`)
fmt.Println(raw)
fmt.Println(string(raw))
fmt.Printf("%s\n", raw)

Output (try it on the Go Playground):

[115 111 109 101 116 104 105 110 103 32 114 97 119]
something raw
something raw

huangapple
  • 本文由 发表于 2016年10月25日 17:49:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/40236700.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定