使用Golang将嵌套数据插入到BigQuery中 [英] Insert nested data into BigQuery using Golang
问题描述
我的BigQuery模式看起来像这样(来自示例)我可以使用Golang将一个扁平对象插入到BigQuery中 - 我如何将嵌套数据插入到表中? :
[{
name:kind,
mode:nullable ,
type:string
},
{
name:fullName,
type:string,
模式:需要
},
{名称:拜访,
类型:记录,
模式:重复 ,
fields:[
{
name:time,
type:timestamp,
mode:nullable
},
{
name:duration,
type:integer,
mode:nullable
}
]
}
]
我第一次尝试插入看起来像这样(示例):
func ExampleInsert(f string,){
jsonRow:= make(map [string] bigquery.JsonValue)
bq,_:= bigquery.New (客户端)
request:= new(bigquery.TableDataInsertAllRequest)
rows:= make([] * bigquery.TableDataInsertAllRequestRows,1)
jsonRow [kind ] = bigquery.JsonValue(kind)
jsonRow [visit_duration] = bigquery.JsonValue(duration)
$ b $ rows [i] = new(bigquery.TableDataInsertAllRequestRows)
rows [i] .Json = jsonRow
bq.Tabledata.InsertAll(projectID,visits,visitsv4,request)
...
}
这可以平滑并插入而不会出现问题。我只是使用visit_duration
但是,我需要循环切片并添加到访问记录中。我试图建立一个访问对象(没有一个循环来测试),并将其添加到行中,但它没有插入并且没有错误:
func ExampleInsert(f string,){
jsonRow:= make(map [string] bigquery.JsonValue)
bq,_:= bigquery.New(client)
request:= new(bigquery.TableDataInsertAllRequest)
$ b $ rows:= make([] * bigquery.TableDataInsertAllRequestRows,1)
jsonRow [kind] = bigquery.JsonValue(kind)
访问次数:= make([] * bigquery.TableDataInsertAllRequestRows,1)
$ b $ j jsonVisit:= make(map [string] bigquery.JsonValue)
jsonVisit [duration] = rand.Intn(1000)
visited [0] = new(bigquery.TableDataInsertAllRequestRows)
visits [0] .Json = jsonVisit
$ b jsonRow [visit] =访问
$ b $ rows [i] = new(bigquery.TableDataInsertAllRequestRows)$ b $ rows [i] .Json = jsonRow
bq.Tabledata.InsertAll(projectID,visits,visitsv4,request)
_,err:= Call.Do()
}
--- [SOLUTION ] ----
正如评论中所建议的那样,我也尝试创建一个切片,然后添加访问:
var访问[] bigquery.JsonValue
访问:= make(map [string] bigquery.JsonValue)
visit [duration] = rand。 Intn(100)
访问=追加(访问,访问)
jsonRow [visit] =访问
我可以证实这确实可行:)对于那些阅读此内容的人来说,即使在添加分片之后,原因并非最初,因为我已经复制了表格。在这样做的过程中,我也弄平了结果。注意。
我不确定为什么你使用过:TableDataInsertAllRequestRows,它应该只用于一次有效负载描述符。 var visits [] bigquery.JsonValue
visit:= make(map [string] bigquery.JsonValue)
visit [duration] = rand.Intn(100)
visits = append(visits,visit)
jsonRow [visit] =访问
ps。还要确保你的模式扁平化
I can insert a flat object into BigQuery using Golang - how I can insert nested data into a table?
My BigQuery schema looks like this (from the example):
[{
"name": "kind",
"mode": "nullable",
"type": "string"
},
{
"name": "fullName",
"type": "string",
"mode": "required"
},
{ "name": "visit",
"type": "record",
"mode": "repeated",
"fields": [
{
"name": "time",
"type": "timestamp",
"mode": "nullable"
},
{
"name": "duration",
"type": "integer",
"mode": "nullable"
}
]
}
]
My first attempt to insert looked like this (example):
func ExampleInsert(f string,) {
jsonRow := make(map[string]bigquery.JsonValue)
bq, _ := bigquery.New(client)
request := new(bigquery.TableDataInsertAllRequest)
rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonRow["kind"] = bigquery.JsonValue(kind)
jsonRow["visit_duration"] = bigquery.JsonValue(duration)
rows[i] = new(bigquery.TableDataInsertAllRequestRows)
rows[i].Json = jsonRow
bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
...
}
Which flattens and inserts without problems. I'm just using visit_duration
But, I need to loop through a slice and add to the visits record. I've tried to build a visit object (without a loop to test) and add that to the row but it's not inserting and I get no errors:
func ExampleInsert(f string,) {
jsonRow := make(map[string]bigquery.JsonValue)
bq, _ := bigquery.New(client)
request := new(bigquery.TableDataInsertAllRequest)
rows := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonRow["kind"] = bigquery.JsonValue(kind)
visits := make([]*bigquery.TableDataInsertAllRequestRows, 1)
jsonVisit := make(map[string]bigquery.JsonValue)
jsonVisit["duration"] = rand.Intn(1000)
visits[0] = new(bigquery.TableDataInsertAllRequestRows)
visits[0].Json = jsonVisit
jsonRow["visit"] = visits
rows[i] = new(bigquery.TableDataInsertAllRequestRows)
rows[i].Json = jsonRow
bq.Tabledata.InsertAll(projectID, "visits", "visitsv4", request)
_, err := Call.Do()
}
---[SOLUTION]----
As suggested in the comments, I have also tried creating a slice and then appending the visit:
var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)
jsonRow["visit"] = visits
I can confirm this does in fact work :) For those of you reading this, the reason it wasn't initially, even after adding a slice, was because I had copied the table. In doing so, I had also flattened the results. Watch out.
Visits should be a slice of bigquery.JsonValue
I am not sure why you have used: TableDataInsertAllRequestRows that should be only used one time for the payload descriptor.
var visits []bigquery.JsonValue
visit := make(map[string]bigquery.JsonValue)
visit["duration"] = rand.Intn(100)
visits = append(visits, visit)
jsonRow["visit"] = visits
ps. also make sure you have your schema flatten
这篇关于使用Golang将嵌套数据插入到BigQuery中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!