英文:
Read Google Cloud Pubsub message and write to BigQuery using Golang
问题
我正在使用以下代码从Google Cloud Pubsub读取数据：
pubsubmessage := pubsubio.Read(s, project, *input, &pubsubio.ReadOptions{Subscription: sub.ID()})
以及以下代码将数据写入我的BigQuery数据集：
bigqueryio.Write(s, project, *output, pubsubmessage)
我得到了以下错误：
panic: schema type must be struct: []uint8
unable to convert []uint8/byte to schema type must be struct`
请帮助我。
我正在遵循这些示例：
https://github.com/apache/beam/blob/master/sdks/go/examples/streaming_wordcap/wordcap.go
英文:
I am using this code to read data from Google Cloud Pubsub:
pubsubmessage := pubsubio.Read(s, project, *input, &pubsubio.ReadOptions{Subscription: sub.ID()})
and this code to write to my bigquery data set :
bigqueryio.Write(s, project, *output, pubsubmessage)
I get the following error:
panic: schema type must be struct: []uint8
unable to convert []uint8/byte to schema type must be struct`
Please help me.
I am following these examples:
https://github.com/apache/beam/blob/master/sdks/go/examples/streaming_wordcap/wordcap.go
答案1
得分: 1
pubsubio.Read的返回值是一个Pubsub消息的PCollection。要将其转换为BigQuery行，您需要应用一个DoFn，它接受一个Pubsub消息并将其转换为BigQuery行。这将返回一个包含BigQuery行的PCollection，您可以将其传递给bigqueryio.Write。类似以下代码：
p := beam.NewPipeline()
s := p.Root()
pubsubmessages := pubsubio.Read(s, project, *input, &pubsubio.ReadOptions{Subscription: sub.ID()})
bigqueryrows := beam.ParDo(s, func(message []byte) string {
return ...
}, pubsubmessages)
bigqueryio.Write(s, project, *output, bigqueryrows)
您需要将...替换为将Pubsub消息的原始字节转换为BigQuery行的代码。
英文:
The return value of pubsubio.Read is a PCollection of Pubsub messages. To convert these to a BigQuery row, you will need to apply a DoFn that takes a Pubsub message and converts it to a BigQuery row. This will return a PCollection of BigQuery rows that you can pass to bigqueryio.Write. Something like this:
p := beam.NewPipeline()
s := p.Root()
pubsubmessages := pubsubio.Read(s, project, *input, &pubsubio.ReadOptions{Subscription: sub.ID()})
bigqueryrows := beam.ParDo(s, func(message []byte) string {
return ...
}, pubsubmessages)
bigqueryio.Write(s, project, *output, bigqueryrows)
You replace the ... with your code that converts the raw bytes of the Pubsub message to a BigQuery row.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论