英文:
Golang dynamodb parallel scan returning same LastEvaluatedKey for all segment
问题
我正在运行一个使用golang编写的程序，它对dynamodb表进行并行扫描。我的ScanInput对象如下所示：
params = &dynamodb.ScanInput{
TableName: aws.String(tableName),
ExclusiveStartKey: lastEvalKey,
Segment: aws.Int64(segment),
TotalSegments: aws.Int64(TOTAL_SEG),
}
除了第一个segment之外，我对所有的segment都遇到了错误。例如，如果totalsegment为4，那么第3个segment将返回以下错误：
ValidationException: The provided starting key is invalid: Invalid ExclusiveStartKey. Please use ExclusiveStartKey with correct Segment. TotalSegments: 4 Segment: 2
通过调试，我检查到出现错误的segment的LastEvaluatedKey与没有抛出错误的segment相同。
我将扫描函数作为goroutine在waitgroup中调用，每个segment一个goroutine。
var wg sync.WaitGroup
wg.Add(int(TOTAL_SEG))
for i := 0; i < int(TOTAL_SEG); i++ {
go func(i int) {
Scan(int64(i))
wg.Done()
}(i)
}
wg.Wait()
有人可以指导一下问题出在哪里吗？提前感谢。
英文:
I am running a golang program that do parallel scan of dynamodb table. My ScanInput object is:
params = &dynamodb.ScanInput{
TableName: aws.String(tableName),
ExclusiveStartKey: lastEvalKey,
Segment: aws.Int64(segment),
TotalSegments: aws.Int64(TOTAL_SEG),
}
I am getting error for all segment except first. For example if totalsegment is 4 than 3 will return below error:
ValidationException: The provided starting key is invalid: Invalid ExclusiveStartKey. Please use ExclusiveStartKey with correct Segment. TotalSegments: 4 Segment: 2
I checked by debugging that for segment that is having error LastEvaluatedKey is same as segment that is not throwing error.
I am calling the function that do scan as goroutine in waitgroup for each segment.
var wg sync.WaitGroup
wg.Add(int(TOTAL_SEG))
for i := 0; i < int(TOTAL_SEG); i++ {
go func(i int) {
Scan(int64(i))
wg.Done()
}(i)
}
wg.Wait()
Can anyone guide what is issue. Thanks in advance.
答案1
得分: 1
解决了这个问题。问题是在第一次扫描的每个段落中，我将ExclusiveStartKey设置为nil，导致每个段落的起始点相同，并返回相同的LastEvaluatedKey值。
对于第一次扫描，我没有为每个段落传递ExclusiveStartKey，而对于后续的每个段落扫描，我使用了前一次扫描中返回的LastEvaluatedKey作为ExclusiveStartKey。
在仔细检查了https://amazon-dynamodb-labs.com/design-patterns/ex2scan/step2.html上的示例后，我解决了这个问题。
英文:
Solve the issue. Problem was for every segment for first scan I was passing ExclusiveStartKey as nil due to which starting point was same for every segment and it was returning same value of LastEvaluatedKey.
For first scan I did not pass ExclusiveStartKey for each segment and for subsequent scan for each segment I used LastEvaluatedKey return in previous scan in ExclusiveStartKey.
After carefully checking example at https://amazon-dynamodb-labs.com/design-patterns/ex2scan/step2.html I solve the issue.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论