英文:
GAE/Go: datastore iterator too slow
问题
在GAE/Go中，迭代数据存储查询结果非常慢。
q := datastore.NewQuery("MyStruct")
gaeLog.Infof(ctx, "run") // (1)
it := client.Run(ctx, q)
list := make([]MyStruct, 0, 10000)
gaeLog.Infof(ctx, "start mapping") // (2)
for {
var m MyStruct
_, err := it.Next(&m)
if err == iterator.Done {
break
}
if err != nil {
gaeLog.Errorf(ctx, "datastore read error: %s", err.Error())
<some error handling>
break
}
list = append(list, m)
}
gaeLog.Infof(ctx, "end mapping. count: %d", len(list)) // (3)
结果如下所示。
18:02:11.283 run // (1)
18:02:11.291 start mapping // (2)
18:02:15.741 end mapping. count: 2400 // (3)
在(2)和(3)之间需要大约4.5秒，仅仅是处理2400条记录。这非常慢。
如何提高性能？
[更新]
我在上述代码中添加了查询 q := datastore.NewQuery("MyStruct")。
我尝试检索所有属于 MyStruct 类型的实体。该类型有2400个实体。
英文:
Iteration to datastore query result in GAE/Go is very slow.
q := datastore.NewQuery("MyStruct")
gaeLog.Infof(ctx, "run") // (1)
it := client.Run(ctx, q)
list := make([]MyStruct, 0, 10000)
gaeLog.Infof(ctx, "start mapping") // (2)
for {
var m MyStruct
_, err := it.Next(&m)
if err == iterator.Done {
break
}
if err != nil {
gaeLog.Errorf(ctx, "datastore read error : %s ", err.Error())
<some error handling>
break
}
list = append(list , m)
}
gaeLog.Infof(ctx, "end mapping. count : %d", len(list)) // (3)
The result is below.
18:02:11.283 run // (1)
18:02:11.291 start mapping // (2)
18:02:15.741 end mapping. count : 2400 // (3)
It takes about 4.5 seconds between (2) and (3), just only 2400 record. It is very slow.
How can I improve performance?
[Update]
I added the query in above code q := datastore.NewQuery("MyStruct").
I tried to retrieve all the entities in the kind MyStruct. This kind has 2400 entities.
答案1
得分: 2
我之前使用的是cloud.google.com/go/datastore，发现速度很慢。我迁移到了google.golang.org/appengine/datastore。
结果如下，少于1秒。
13:57:46.216 运行
13:57:46.367 开始映射
13:57:47.063 结束映射。数量：2400
英文:
I was using cloud.google.com/go/datastore and found it is slow. I migrated to use google.golang.org/appengine/datastore.
The result is as follows, less than 1 second.
13:57:46.216 run
13:57:46.367 start mapping
13:57:47.063 end mapping. count : 2400
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论