英文:
Datastore transaction - hitting entity write limit
问题
问题
使用golang的cloud.google.com/go/datastore
包创建一个事务,在这个事务中执行一系列的getMulti
和putMulti
操作,当提交这个事务时,我遇到了一个实体写入限制错误。
2021/12/22 09:07:18 err: rpc error: code = InvalidArgument desc = cannot write more than 500 entities in a single call
问题是
我的问题是如何创建一个包含超过500个写入操作的事务?
虽然我希望我的操作保持原子性,但是我似乎无法解决这个事务的写入限制错误,而且当我在模拟器上进行测试时,以500个为一批进行写入的查询集运行得很好。
我尝试过的方法
请原谅我使用的是伪代码,但我试图得到我所做的事情的要点。
一次性操作:
transaction, err := datastoreClient.NewTransaction(ctx)
transaction.PutMulti(allKeys, allEntities)
transaction.commit()
// 错误:一次性调用中写入的实体过多
分批操作以避免写入限制:
transaction, err := datastoreClient.NewTransaction(ctx)
transaction.PutMulti(first500Keys, first500Entities)
transaction.PutMulti(second500Keys, second500Entities)
transaction.commit()
// 错误:一次性调用中写入的实体过多
简单的普通putmulti也失败了:
datastoreClient.PutMulti(ctx,allKeys, allEntities)
// 错误:一次性调用中写入的实体过多
有效的方法
非原子性地写入数据存储:
datastoreClient.PutMulti(ctx,first500Keys, first500Entities)
datastoreClient.PutMulti(ctx,second500Keys, second500Entities)
这是我用于写入的真实代码,无论是批处理事务还是普通的putMulti:
for i := 0; i < (len(allKeys) / 500); i++ {
var max int = (i + 1) * 500
if len(allKeys) < max {
max = len(allKeys) % 500
}
_, err = svc.dsClient.PutMulti(ctx, allKeys[i*500:max], allEntities[i*500:max])
if err != nil {
return
}
}
我迷失的地方
所以为了保持我的工作的原子性,有没有办法提交一个包含超过500个实体写入操作的事务?
英文:
The Problem
Using the golang cloud.google.com/go/datastore
package to create a transaction, perform a series of getMulti
's, and putMulti
's, on commit of this transaction I'm confronted with an an entity write limit error.
2021/12/22 09:07:18 err: rpc error: code = InvalidArgument desc = cannot write more than 500 entities in a single call
The Question
My question is how do you create a transaction with more than 500 writes?
While I want my operation to remain atomic, I can't seem to solve this write limit error for a transaction and the set of queries run just fine when I test on an emulator, writing in batches of 500.
What I've Tried
please excuse the sudo code but I'm trying to get the jist of what I've done
All in one
transaction, err := datastoreClient.NewTransaction(ctx)
transaction.PutMulti(allKeys, allEntities)
transaction.commit()
// err too many entities written in a single call
Batched in an attempt to avoid the write limit
transaction, err := datastoreClient.NewTransaction(ctx)
transaction.PutMulti(first500Keys, first500Entities)
transaction.PutMulti(second500Keys, second500Entities)
transaction.commit()
// err too many entities written in a single call
A simple regular putmulti also fails
datastoreClient.PutMulti(ctx,allKeys, allEntities)
// err too many entities written in a single call
What Works
Non-atomic write to the datastore
datastoreClient.PutMulti(ctx,first500Keys, first500Entities)
datastoreClient.PutMulti(ctx,second500Keys, second500Entities)
here's the real code that I used for the write, either as a batched transaction or regular putMulti
for i := 0; i < (len(allKeys) / 500); i++ {
var max int = (i + 1) * 500
if len(allKeys) < max {
max = len(allKeys) % 500
}
_, err = svc.dsClient.PutMulti(ctx, allKeys[i*500:max], allEntities[i*500:max])
if err != nil {
return
}
}
Where I'm Lost
so in an effort to keeping my work atomic, is there any method to commit a transaction that has more than 500 entities written in it?
答案1
得分: 1
你无法做任何事情。这个限制是平台强制执行的,以确保可扩展性并防止性能下降。你不能在单个事务中写入超过500个实体。
在谷歌的一侧可能可以更改这个限制,但在你的一侧无能为力。
英文:
Nothing you can do. This limit is enforced by the platform to ensure scalability and to prevent performance degradation. You can't write more than 500 entities in a single transaction.
It's possible to change the limit on Google's side, but nothing you can do on your side.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论