英文:
Insert nested data into BigQuery using Golang
问题
我可以使用Golang将嵌套数据插入到BigQuery表中。你的BigQuery模式看起来像这样(来自示例):
[{
"name": "kind",
"mode": "nullable",
"type": "string"
},
{
"name": "fullName",
"type": "string",
"mode": "required"
},
{
"name": "visit",
"type": "record",
"mode": "repeated",
"fields": [
{
"name": "time",
"type": "timestamp",
"mode": "nullable"
},
{
"name": "duration",
"type": "integer",
"mode": "nullable"
}
]
}
]
你的第一次尝试插入的代码如下(示例):
func ExampleInsert(f string,) {
jsonRow := make(map[string]bigquery.JsonValue)
bq, _ := bigquery.New(client)
request := new(bigquery.TableDataInsertAllRequest)
rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonRow["kind"] = bigquery.JsonValue(kind)
jsonRow["visit_duration"] = bigquery.JsonValue(duration)
rows[i] = new(bigquery.TableDataInsertAllRequestRows)
rows[i].Json = jsonRow
bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
...
}
这段代码可以成功地将扁平化的数据插入到表中,但只使用了visit_duration
字段。
但是,如果你需要遍历一个切片并将数据添加到visits
记录中,你可以尝试构建一个visit
对象并将其添加到行中。以下是示例代码:
func ExampleInsert(f string,) {
jsonRow := make(map[string]bigquery.JsonValue)
bq, _ := bigquery.New(client)
request := new(bigquery.TableDataInsertAllRequest)
rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonRow["kind"] = bigquery.JsonValue(kind)
visits := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonVisit := make(map[string]bigquery.JsonValue)
jsonVisit["duration"] = rand.Intn(1000)
visits[0] = new(bigquery.TableDataInsertAllRequestRows)
visits[0].Json = jsonVisit
jsonRow["visit"] = visits
rows[i] = new(bigquery.TableDataInsertAllRequestRows)
rows[i].Json = jsonRow
bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
_, err := Call.Do()
}
---【解决方案】---
如评论中建议的,你也可以尝试创建一个切片,然后将visit
添加到其中:
var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)
jsonRow["visit"] = visits
我可以确认这确实可以工作 对于阅读这个回答的其他人,如果一开始没有成功,即使添加了切片,也可能是因为我复制了表。在这样做时,我还扁平化了结果。请注意这一点。
英文:
I can insert a flat object into BigQuery using Golang - how I can insert nested data into a table?
My BigQuery schema looks like this (from the example):
[{
"name": "kind",
"mode": "nullable",
"type": "string"
},
{
"name": "fullName",
"type": "string",
"mode": "required"
},
{ "name": "visit",
"type": "record",
"mode": "repeated",
"fields": [
{
"name": "time",
"type": "timestamp",
"mode": "nullable"
},
{
"name": "duration",
"type": "integer",
"mode": "nullable"
}
]
}
]
My first attempt to insert looked like this (example):
func ExampleInsert(f string,) {
jsonRow := make(map[string]bigquery.JsonValue)
bq, _ := bigquery.New(client)
request := new(bigquery.TableDataInsertAllRequest)
rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonRow["kind"] = bigquery.JsonValue(kind)
jsonRow["visit_duration"] = bigquery.JsonValue(duration)
rows[i] = new(bigquery.TableDataInsertAllRequestRows)
rows[i].Json = jsonRow
bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
...
}
Which flattens and inserts without problems. I'm just using visit_duration
But, I need to loop through a slice and add to the visits record. I've tried to build a visit object (without a loop to test) and add that to the row but it's not inserting and I get no errors:
func ExampleInsert(f string,) {
jsonRow := make(map[string]bigquery.JsonValue)
bq, _ := bigquery.New(client)
request := new(bigquery.TableDataInsertAllRequest)
rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonRow["kind"] = bigquery.JsonValue(kind)
visits := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonVisit := make(map[string]bigquery.JsonValue)
jsonVisit["duration"] = rand.Intn(1000)
visits[0] = new(bigquery.TableDataInsertAllRequestRows)
visits[0].Json = jsonVisit
jsonRow["visit"] = visits
rows[i] = new(bigquery.TableDataInsertAllRequestRows)
rows[i].Json = jsonRow
bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
_, err := Call.Do()
}
---[SOLUTION]----
As suggested in the comments, I have also tried creating a slice and then appending the visit:
var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)
jsonRow["visit"] = visits
I can confirm this does in fact work For those of you reading this, the reason it wasn't initially, even after adding a slice, was because I had copied the table. In doing so, I had also flattened the results. Watch out.
答案1
得分: 2
访问应该是 bigquery.JsonValue
的切片。我不确定为什么你使用了 TableDataInsertAllRequestRows
,它应该只在负载描述符中使用一次。
var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)
jsonRow["visit"] = visits
另外,请确保你的模式是扁平化的。
英文:
Visits should be a slice of bigquery.JsonValue
I am not sure why you have used: TableDataInsertAllRequestRows that should be only used one time for the payload descriptor.
var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)
jsonRow["visit"] = visits
ps. also make sure you have your schema flatten
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论