英文:
How to create structure for nested data set in golang?
问题
新手学习golang，尝试编写一个脚本来批量上传到Elasticsearch服务器。我的JSON数据集如下所示...
{
"product_displayname": "LG Stylus 2 Plus K535D (16 GB, Brown)",
"product_price": "24000.00",
"popularity": "0.00",
"barcode": "",
"exclusive_flag": "0",
"product_id": "176982",
"product_name": "Stylus 2 Plus K535D (Brown)",
"brand_name": "LG",
"brand_id": "1",
"product_spec": {
"display_spec": [
{
"spec_id": "103",
"sdv": "24000",
"snv": "24000.0000"
},
{
"spec_id": "104",
"sdv": "GSM",
"snv": "0.0000"
}
],
"filter_spec": [
{
"spec_id": "103",
"sdv": "24000",
"snv": "24000.0000"
},
{
"spec_id": "105",
"sdv": "Touch Screen",
"snv": "0.0000"
}
]
}
}
我根据谷歌和其他在线信息创建了以下的Golang结构体来表示上述数据集：
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductPrice string `json:"product_price"`
Popularity string `json:"popularity"`
Barcode string `json:"barcode"`
ExclusiveFlag string `json:"exclusive_flag"`
ProductID string `json:"product_id"`
ProductName string `json:"product_name"`
BrandName string `json:"brand_name"`
BrandID string `json:"brand_id"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
但是，当我尝试在我的批量上传脚本中使用上述结构体和示例数据时，我遇到了以下错误：
github.com/crazyheart/elastic-bulk-upload/main.go:70: syntax error: missing operand
github.com/crazyheart/elastic-bulk-upload/main.go:70: unknown escape sequence
github.com/crazyheart/elastic-bulk-upload/main.go:71: syntax error: non-declaration statement outside function body
我觉得我在映射Golang结构体中的嵌套字段display_spec和filter_spec时犯了一些错误，但是我无法找出问题所在。
你可以参考以下的main.go代码：
package main
import (
"fmt"
"golang.org/x/net/context"
"gopkg.in/olivere/elastic.v5"
"strconv"
)
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductPrice string `json:"product_price"`
Popularity string `json:"popularity"`
Barcode string `json:"barcode"`
ExclusiveFlag string `json:"exclusive_flag"`
ProductID string `json:"product_id"`
ProductName string `json:"product_name"`
BrandName string `json:"brand_name"`
BrandID string `json:"brand_id"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
func main() {
// 创建一个上下文
ctx := context.Background()
client, err := elastic.NewClient()
if err != nil {
fmt.Println("%v", err)
}
// 批量上传代码
n := 0
for i := 0; i < 1000; i++ {
bulkRequest := client.Bulk()
for j := 0; j < 10000; j++ {
n++
product_data := Product{
ProductDisplayname: "LG Stylus 2 Plus K535D (16 GB, Brown)",
ProductPrice: "24000.00",
Popularity: "0.00",
Barcode: "",
ExclusiveFlag: "0",
ProductID: "17698276",
ProductName: "Stylus 2 Plus K535D (Brown)",
BrandName: "LG",
BrandID: "1",
ProductSpec: struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
}{
DisplaySpec: []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
}{
{
SpecID: "103",
Sdv: "24000",
Snv: "24000.0000",
},
{
SpecID: "104",
Sdv: "GSM",
Snv: "0.0000",
},
},
FilterSpec: []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
}{
{
SpecID: "103",
Sdv: "24000",
Snv: "24000.0000",
},
{
SpecID: "105",
Sdv: "Touch Screen",
Snv: "0.0000",
},
},
},
}
req := elastic.NewBulkIndexRequest().Index("shopfront").Type("products").Id(strconv.Itoa(n)).Doc(product_data)
bulkRequest = bulkRequest.Add(req)
}
bulkResponse, err := bulkRequest.Do(ctx)
if err != nil {
fmt.Println(err)
}
if bulkResponse != nil {
fmt.Println(bulkResponse)
}
fmt.Println(i)
}
}
希望这可以帮助到你解决问题。
英文:
New to golang & trying to make a script for making bulk upload to Elasticsearch server. My json data set is something like this...
{
product_displayname: "LG Stylus 2 Plus K535D (16 GB, Brown)",
product_price: "24000.00",
popularity: "0.00",
barcode: "",
exclusive_flag: "0",
product_id: "176982",
product_name: "Stylus 2 Plus K535D (Brown)",
brand_name: "LG",
brand_id: "1",
product_spec : {
display_spec: [{
spec_id: "103",
sdv: "24000",
snv: "24000.0000"
}, {
spec_id: "104",
sdv: "GSM",
snv: "0.0000"
}],
filter_spec: [{
spec_id: "103",
sdv: "24000",
snv: "24000.0000"
}, {
spec_id: "105",
sdv: "Touch Screen",
snv: "0.0000"
}]
}
}
Golang Structure I made(by refering google & other online info) for the above dataset is like this...
type Product struct {
product_displayname string `json:"product_displayname"`
product_price string `json:"product_price"`
popularity string `json:"popularity"`
barcode string `json:"barcode"`
exclusive_flag string `json:"exclusive_flag"`
product_id string `json:"product_id"`
product_name string `json:"product_name"`
brand_name string `json:"brand_name"`
brand_id string `json:"brand_id"`
product_spec
}
type product_spec struct {
display_spec []display_speclist
filter_spec []filter_speclist
}
type display_speclist struct {
spec_id string `json:"spec_id"`
sdv string `json:"sdv"`
snv string `json:"snv"`
}
type filter_speclist struct {
spec_id string `json:"spec_id"`
sdv string `json:"sdv"`
snv string `json:"snv"`
}
But, whenever I'm trying to use above Structure with sample data in my bulk upload script I'm getting following error
github.com/crazyheart/elastic-bulk-upload/main.go:70: syntax error: missing operand
github.com/crazyheart/elastic-bulk-upload/main.go:70: unknown escape sequence
github.com/crazyheart/elastic-bulk-upload/main.go:71: syntax error: non-declaration statement outside function body
I feel like I'm making some mistake in mapping that nested field display_spec & filter_spec in golang structure. But can't able to figure out what it is.
main.go
package main
import (
"fmt"
"golang.org/x/net/context"
"gopkg.in/olivere/elastic.v5"
"strconv"
)
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductPrice string `json:"product_price"`
Popularity string `json:"popularity"`
Barcode string `json:"barcode"`
ExclusiveFlag string `json:"exclusive_flag"`
ProductID string `json:"product_id"`
ProductName string `json:"product_name"`
BrandName string `json:"brand_name"`
BrandID string `json:"brand_id"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
func main() {
// Create a context
ctx := context.Background()
client, err := elastic.NewClient()
if err != nil {
fmt.Println("%v", err)
}
// Bulk upload code
n := 0
for i := 0; i < 1000; i++ {
bulkRequest := client.Bulk()
for j := 0; j < 10000; j++ {
n++
product_data := Product{product_displayname:"LG Stylus 2 Plus K535D (16 GB, Brown)",product_price:"24000.00",popularity:"0.00",barcode:"",exclusive_flag:"0",product_id:"17698276",product_name:"Stylus 2 Plus K535D (Brown)",brand_name:"LG",brand_id:"1",product_spec:{display_spec:[{spec_id:"103",sdv:"24000",snv:"24000.0000"},{spec_id:"104",sdv:"GSM",snv:"0.0000"}],filter_spec:[{spec_id:"103",sdv:"24000",snv:"24000.0000"},{spec_id:"105",sdv:"Touch Screen",snv:"0.0000"}]} }
req := elastic.NewBulkIndexRequest().Index("shopfront").Type("products").Id(strconv.Itoa(n)).Doc(product_data)
bulkRequest = bulkRequest.Add(req)
}
bulkResponse, err := bulkRequest.Do(ctx)
if err != nil {
fmt.Println(err)
}
if bulkResponse != nil {
fmt.Println(bulkResponse)
}
fmt.Println(i)
}
}
答案1
得分: 1
工作流程
-
验证你的 JSON（你发布的那个是无效的）。
-
构建一个合适的
struct，你可以使用这个很好的工具来帮助你。
针对你的情况
structs 看起来没问题，除了你没有通过将字段的首字母大写来导出结构体字段（感谢 @ANisus）。
这个（缩短的）看起来更加“自然”。
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
英文:
Workflow
1.- Validate your json (the one you posted is invalid).
2.- Build a proper struct, you can help yourself using this nice tool.
For your case
The structs appears to be fine except you're not exporting the struct fields by capitalizing the initial letter (Thanks @ANisus).
This (shorted) seems more natural.
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论