将嵌套数据使用Golang插入到BigQuery中

huangapple go评论76阅读模式
英文:

Insert nested data into BigQuery using Golang

问题

我可以使用Golang将嵌套数据插入到BigQuery表中。你的BigQuery模式看起来像这样(来自示例):

[{
    "name": "kind",
    "mode": "nullable",
    "type": "string"
  },
  {
    "name": "fullName",
    "type": "string",
    "mode": "required"
  },
  {
    "name": "visit",
    "type": "record",
    "mode": "repeated",
    "fields": [
      {
         "name": "time",
         "type": "timestamp",
         "mode": "nullable"
      },
      {
         "name": "duration",
         "type": "integer",
         "mode": "nullable"
      }
    ]
  }
]

你的第一次尝试插入的代码如下(示例):

func ExampleInsert(f string,) {

  jsonRow := make(map[string]bigquery.JsonValue)

  bq, _ := bigquery.New(client)
  request := new(bigquery.TableDataInsertAllRequest)

  rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)

  jsonRow["kind"] = bigquery.JsonValue(kind)
  jsonRow["visit_duration"] = bigquery.JsonValue(duration)

  rows[i] = new(bigquery.TableDataInsertAllRequestRows)
  rows[i].Json = jsonRow

  bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
  ...
}

这段代码可以成功地将扁平化的数据插入到表中,但只使用了visit_duration字段。

但是,如果你需要遍历一个切片并将数据添加到visits记录中,你可以尝试构建一个visit对象并将其添加到行中。以下是示例代码:

func ExampleInsert(f string,) {

  jsonRow := make(map[string]bigquery.JsonValue)

  bq, _ := bigquery.New(client)
  request := new(bigquery.TableDataInsertAllRequest)

  rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)

  jsonRow["kind"] = bigquery.JsonValue(kind)

  visits := make([]*bigquery.TableDataInsertAllRequestRows, 1)

  jsonVisit := make(map[string]bigquery.JsonValue)
  jsonVisit["duration"] = rand.Intn(1000)
  visits[0] = new(bigquery.TableDataInsertAllRequestRows)
  visits[0].Json = jsonVisit

  jsonRow["visit"] = visits

  rows[i] = new(bigquery.TableDataInsertAllRequestRows)
  rows[i].Json = jsonRow

  bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)

  _, err := Call.Do()
}

---【解决方案】---

如评论中建议的,你也可以尝试创建一个切片,然后将visit添加到其中:

var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)

jsonRow["visit"] = visits

我可以确认这确实可以工作 将嵌套数据使用Golang插入到BigQuery中 对于阅读这个回答的其他人,如果一开始没有成功,即使添加了切片,也可能是因为我复制了表。在这样做时,我还扁平化了结果。请注意这一点。

英文:

I can insert a flat object into BigQuery using Golang - how I can insert nested data into a table?

My BigQuery schema looks like this (from the example):

[{
    "name": "kind",
    "mode": "nullable",
    "type": "string"
  },
  {
    "name": "fullName",
    "type": "string",
    "mode": "required"
  },
  { "name": "visit",
    "type": "record",
    "mode": "repeated",
    "fields": [
    {
       "name": "time",
       "type": "timestamp",
       "mode": "nullable"
    },
    {
       "name": "duration",
       "type": "integer",
       "mode": "nullable"
    }
   ]
  }
]

My first attempt to insert looked like this (example):

func ExampleInsert(f string,) {

  jsonRow := make(map[string]bigquery.JsonValue)

  bq, _ := bigquery.New(client)
  request := new(bigquery.TableDataInsertAllRequest)

  rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)

  jsonRow["kind"] = bigquery.JsonValue(kind)
  jsonRow["visit_duration"] = bigquery.JsonValue(duration)

  rows[i] = new(bigquery.TableDataInsertAllRequestRows)
  rows[i].Json = jsonRow

  bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
  ...
}

Which flattens and inserts without problems. I'm just using visit_duration

But, I need to loop through a slice and add to the visits record. I've tried to build a visit object (without a loop to test) and add that to the row but it's not inserting and I get no errors:

func ExampleInsert(f string,) {

  jsonRow := make(map[string]bigquery.JsonValue)

  bq, _ := bigquery.New(client)
  request := new(bigquery.TableDataInsertAllRequest)

  rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)

  jsonRow["kind"] = bigquery.JsonValue(kind)

  visits := make([]*bigquery.TableDataInsertAllRequestRows, 1)

  jsonVisit := make(map[string]bigquery.JsonValue)
  jsonVisit["duration"] = rand.Intn(1000)
  visits[0] = new(bigquery.TableDataInsertAllRequestRows)
  visits[0].Json = jsonVisit

  jsonRow["visit"] = visits

  rows[i] = new(bigquery.TableDataInsertAllRequestRows)
  rows[i].Json = jsonRow

  bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
  
  _, err := Call.Do()
}

---[SOLUTION]----

As suggested in the comments, I have also tried creating a slice and then appending the visit:

var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)

jsonRow["visit"] = visits

I can confirm this does in fact work 将嵌套数据使用Golang插入到BigQuery中 For those of you reading this, the reason it wasn't initially, even after adding a slice, was because I had copied the table. In doing so, I had also flattened the results. Watch out.

答案1

得分: 2

访问应该是 bigquery.JsonValue 的切片。我不确定为什么你使用了 TableDataInsertAllRequestRows,它应该只在负载描述符中使用一次。

var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)

jsonRow["visit"] = visits

另外,请确保你的模式是扁平化的。

英文:

Visits should be a slice of bigquery.JsonValue I am not sure why you have used: TableDataInsertAllRequestRows that should be only used one time for the payload descriptor.

var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)

jsonRow["visit"] = visits

ps. also make sure you have your schema flatten

huangapple
  • 本文由 发表于 2015年5月22日 19:42:28
  • 转载请务必保留本文链接:https://go.coder-hub.com/30395813.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定