英文:
Determining size of AppendRows request message
问题
我正在使用Golang Storage API库向BigQuery写入数据。为此，我调用AppendRows()函数https://pkg.go.dev/cloud.google.com/go/bigquery/storage/managedwriter#ManagedStream.AppendRows。
这个函数有一个限制，即请求的大小不能超过10 MB，否则会返回错误。我可以通过将数据分成较小的块来避免这个问题。但是我不想发送太小的块，并且希望在一个块中发送尽可能多的数据。
所以，我想知道在将数据发送到函数之前是否有一种方法可以确定请求的大小？这将帮助我调整消息的大小，而不是不断将数据添加到请求中。
英文:
I am using the Golang Storage API library to write to BigQuery. For this I am calling AppendRows() https://pkg.go.dev/cloud.google.com/go/bigquery/storage/managedwriter#ManagedStream.AppendRows.
This function has a limitation that the request size cannot be more than 10 MB otherwise an error will be returned. One way I can avoid this is to send the data in smaller chunks. But I do not want to send too small chunks and want to send as much data as possible in a chunk.
So, I want to know if there is a way I can determine the request size before sending the data to the function? This would help me to adjust the size of the message and not keep adding the data to the request.
答案1
得分: 1
一个选项是在将数据发送到函数之前执行干运行。
使用Golang的干运行的代码片段在这里给出。干运行将估计查询读取的字节数。希望这可以帮到你。
英文:
One options is to execute a dry run before sending the data to function.
The code snippet of the dry one using Golang is given here. The dry run will give estimate the number of bytes read by the query. Hope this helps.
答案2
得分: 1
根据您的要求，您可以考虑@BihagKashikar提到的选项。
另一个选项是重新设计工作，使每个组件生成较小的请求。可以通过更频繁地发送较少行的请求，或者通过检查您尝试发送的数据的大小，并在需要时将其拆分为多个请求来实现。如果插入了单个长行，可以使用'bq load'进行拆分为多个请求，如果有多行，则拆分为多个请求。
有关更多信息，您可以参考此文档。
英文:
For your requirement, you can consider the option mentioned by @BihagKashikar.
Other option is to, Restructure the job so that each component generates smaller requests. This can be done by Sending requests with fewer rows more often, or by checking the size of the data you are trying to send and split it into several requests if needed.
Using ‘bq load’ if you insert a single long row. Splitting into multiple requests if there are multiple rows.
For more information you can refer to this document.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论