数据存储事务 – 达到实体写入限制

huangapple go评论72阅读模式
英文:

Datastore transaction - hitting entity write limit

问题

问题

使用golang的cloud.google.com/go/datastore包创建一个事务,在这个事务中执行一系列的getMultiputMulti操作,当提交这个事务时,我遇到了一个实体写入限制错误。

2021/12/22 09:07:18 err: rpc error: code = InvalidArgument desc = cannot write more than 500 entities in a single call

问题是

我的问题是如何创建一个包含超过500个写入操作的事务?

虽然我希望我的操作保持原子性,但是我似乎无法解决这个事务的写入限制错误,而且当我在模拟器上进行测试时,以500个为一批进行写入的查询集运行得很好。

我尝试过的方法

请原谅我使用的是伪代码,但我试图得到我所做的事情的要点。

一次性操作:

transaction, err := datastoreClient.NewTransaction(ctx)
transaction.PutMulti(allKeys, allEntities)
transaction.commit()
// 错误:一次性调用中写入的实体过多

分批操作以避免写入限制:

transaction, err := datastoreClient.NewTransaction(ctx)
transaction.PutMulti(first500Keys, first500Entities)
transaction.PutMulti(second500Keys, second500Entities)
transaction.commit()
// 错误:一次性调用中写入的实体过多

简单的普通putmulti也失败了:

datastoreClient.PutMulti(ctx,allKeys, allEntities)
// 错误:一次性调用中写入的实体过多

有效的方法

非原子性地写入数据存储:

datastoreClient.PutMulti(ctx,first500Keys, first500Entities)
datastoreClient.PutMulti(ctx,second500Keys, second500Entities)

这是我用于写入的真实代码,无论是批处理事务还是普通的putMulti:

for i := 0; i < (len(allKeys) / 500); i++ {
    var max int = (i + 1) * 500
    if len(allKeys) < max {
        max = len(allKeys) % 500
    }

    _, err = svc.dsClient.PutMulti(ctx, allKeys[i*500:max], allEntities[i*500:max])
    if err != nil {
        return
    }
}

我迷失的地方

所以为了保持我的工作的原子性,有没有办法提交一个包含超过500个实体写入操作的事务?

英文:

The Problem

Using the golang cloud.google.com/go/datastore package to create a transaction, perform a series of getMulti's, and putMulti's, on commit of this transaction I'm confronted with an an entity write limit error.

2021/12/22 09:07:18 err: rpc error: code = InvalidArgument desc = cannot write more than 500 entities in a single call

The Question

My question is how do you create a transaction with more than 500 writes?

While I want my operation to remain atomic, I can't seem to solve this write limit error for a transaction and the set of queries run just fine when I test on an emulator, writing in batches of 500.

What I've Tried

please excuse the sudo code but I'm trying to get the jist of what I've done
All in one

    transaction, err := datastoreClient.NewTransaction(ctx)
    transaction.PutMulti(allKeys, allEntities)
    transaction.commit()
// err too many entities written in a single call

Batched in an attempt to avoid the write limit

    transaction, err := datastoreClient.NewTransaction(ctx)
    transaction.PutMulti(first500Keys, first500Entities)
    transaction.PutMulti(second500Keys, second500Entities)
    transaction.commit()
// err too many entities written in a single call

A simple regular putmulti also fails

    datastoreClient.PutMulti(ctx,allKeys, allEntities)
// err too many entities written in a single call

What Works

Non-atomic write to the datastore

    datastoreClient.PutMulti(ctx,first500Keys, first500Entities)
    datastoreClient.PutMulti(ctx,second500Keys, second500Entities)

here's the real code that I used for the write, either as a batched transaction or regular putMulti

	for i := 0; i &lt; (len(allKeys) / 500); i++ {
		var max int = (i + 1) * 500
		if len(allKeys) &lt; max {
			max = len(allKeys) % 500
		}

		_, err = svc.dsClient.PutMulti(ctx, allKeys[i*500:max], allEntities[i*500:max])
		if err != nil {
			return
		}
	}

Where I'm Lost

so in an effort to keeping my work atomic, is there any method to commit a transaction that has more than 500 entities written in it?

答案1

得分: 1

你无法做任何事情。这个限制是平台强制执行的,以确保可扩展性并防止性能下降。你不能在单个事务中写入超过500个实体。

在谷歌的一侧可能可以更改这个限制,但在你的一侧无能为力。

英文:

Nothing you can do. This limit is enforced by the platform to ensure scalability and to prevent performance degradation. You can't write more than 500 entities in a single transaction.

It's possible to change the limit on Google's side, but nothing you can do on your side.

huangapple
  • 本文由 发表于 2021年12月22日 21:25:15
  • 转载请务必保留本文链接:https://go.coder-hub.com/70449795.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定