DynamoDb写入和获取速度太慢。

huangapple go评论79阅读模式
英文:

DynamoDb write and fetch are too slow

问题

我在我的Springboot应用程序中首次使用Dynamodb,并注意到响应时间非常慢。

我的表结构如下 -

itemid - 每条记录的唯一键,作为主分区键而不带任何排序键。
还有大约10个其他属性,包括2个GSI的属性(item_category(PK), item_date(SK))。

将大约7k条记录持久化需要约30分钟,获取相同记录大约需要一分钟,对于我的情况来说这太高了。我曾以为在DynamoDB中可以在几秒内持久化这么多记录。我的理解错了吗,还是我的表不正确?

DynamoDB表使用默认的读/写容量单位(1-10),并启用了自动扩展。

如何提高DynamoDB事务的读/写速度?

更新:

写入Dynamo表的代码 -

public List<MyItems> saveTransactions(List<MyItems> dynamoItems) {
    DynamoDbTable<MyItems> dynamoDbTable = dynamoDbEnhancedClient.table(tableName, TableSchema.fromBean(MyItems.class));
    WriteBatch.Builder<MyItems> dynamoItemsBuilder = WriteBatch.builder(MyItems.class).mappedTableResource(dynamoDbTable);
    for (MyItems transactions : dynamoItems) {
        dynamoItemsBuilder.addPutItem(builder -> builder.item(transactions));
    }
    BatchWriteResult batchWriteResult = dynamoDbEnhancedClient.batchWriteItem(BatchWriteItemEnhancedRequest.builder().writeBatches(dynamoItemsBuilder.build()).build());
    return batchWriteResult.unprocessedPutItemsForTable(dynamoDbTable);
}

从Dynamo表读取的代码 -

public Optional<List<MyItems>> fetchTransactions(String itemCategory, String itemDate) {
    DynamoDbTable<MyItems> userTable = dynamo.table(tableName, TableSchema.fromBean(MyItems.class));
    DynamoDbIndex<MyItems> gsiIndex = userTable.index("GSI_INDEX");
    QueryConditional q = QueryConditional.keyEqualTo(Key.builder().partitionValue(itemCategory).sortValue(itemDate).build());
    Iterator<Page<MyItems>> result = gsiIndex.query(q).iterator();
    List<MyItems> poslogTransactionsList = new ArrayList<>();
    while (result.hasNext()) {
        poslogTransactionsList.addAll(result.next().items());
    }
    return Optional.of(poslogTransactionsList);
}

Bean类 -

@Data
@NoArgsConstructor
@DynamoDbBean
public class MyItems {
    @EqualsAndHashCode.Include
    @Getter(AccessLevel.NONE)
    String itemId;
    @Getter(AccessLevel.NONE)
    String itemCategory;
    @Getter(AccessLevel.NONE)
    String itemDate;
    String amount;
    String itemNumber;

    @DynamoDbPartitionKey
    public String getItemId() {
        return itemId;
    }
    @DynamoDbSecondaryPartitionKey(indexNames = "GSI_INDEX")
    public String getItemCategory() {
        return itemCategory;
    }

    @DynamoDbSecondarySortKey(indexNames = "GSI_INDEX")
    public String getItemDate() {
        return itemDate;
    }
}

上面的代码基于从Kafka消费者接收的消息运行,每次List可能包含10条或更少的项目。对于这些记录(带有唯一的itemid),持久化也需要约300毫秒。代码或表结构有什么问题吗?

英文:

I am using Dynamodb in one of my Springboot application for the first time and noticing very slow response time.

My table structure is like below -

itemid - A unique key for each record, is made as primary partition key without any sort key.
Around 10 other attributes including 2 of attributes of GSI (item_category(PK), item_date(SK)).

It's taking ~30 minutes to persist ~7k records and approximately a minute to fetch same records, which is too high for my scenario. I was under the impression that we can persist this many records in seconds in DynamoDB. Is my understanding is wrong or is my table is not correct?

DynamoDB table is created with default read/write capacity units(1-10) with auto scaling enabled.

How to improve read/write speed of dynamodb transactions?

Update:

Code which writes to Dynamo table -

public List&lt;MyItems&gt; saveTransactions(List&lt;MyItems&gt; dynamoItems) {
    DynamoDbTable&lt;MyItems&gt; dynamoDbTable = dynamoDbEnhancedClient.table(tableName, TableSchema.fromBean(MyItems.class));
    WriteBatch.Builder&lt;MyItems&gt; dynamoItemsBuilder = WriteBatch.builder(MyItems.class).mappedTableResource(dynamoDbTable);
    for (MyItems transactions : dynamoItems) {
        dynamoItemsBuilder.addPutItem(builder -&gt; builder.item(transactions));
    }
    BatchWriteResult batchWriteResult = dynamoDbEnhancedClient.batchWriteItem(BatchWriteItemEnhancedRequest.builder().writeBatches(dynamoItemsBuilder.build()).build());
    return batchWriteResult.unprocessedPutItemsForTable(dynamoDbTable);
}

Code which reads from Dynamo table -

 public Optional&lt;List&lt;MyItems&gt;&gt; fetchTransactions(String itemCategory, String itemDate) {
    DynamoDbTable&lt;MyItems&gt; userTable = dynamo.table(tableName, TableSchema.fromBean(MyItems.class));
    DynamoDbIndex&lt;MyItems&gt; gsiIndex = userTable.index(&quot;GSI_INDEX&quot;);
    QueryConditional q = QueryConditional.keyEqualTo(Key.builder().partitionValue(itemCategory).sortValue(itemDate).build());
    Iterator&lt;Page&lt;MyItems&gt;&gt; result = gsiIndex.query(q).iterator();
    List&lt;MyItems&gt; poslogTransactionsList = new ArrayList&lt;&gt;();
    while (result.hasNext()) {
        poslogTransactionsList.addAll(result.next().items());
    }
    return Optional.of(poslogTransactionsList);
}

Bean class -

@Data
@NoArgsConstructor
@DynamoDbBean
public class MyItems {
    @EqualsAndHashCode.Include
    @Getter(AccessLevel.NONE)
    String itemId;
    @Getter(AccessLevel.NONE)
    String itemCategory;
    @Getter(AccessLevel.NONE)
    String itemDate;
    String amount;
    String itemNumber;

    @DynamoDbPartitionKey
    public String getItemId() {
        return itemId;
    }
    @DynamoDbSecondaryPartitionKey(indexNames = &quot;GSI_INDEX&quot;)
    public String getItemCategory() {
        return itemCategory;}

    @DynamoDbSecondarySortKey(indexNames = &quot;GSI_INDEX&quot;)
    public String getItemDate() {
        return itemDate;}}

Here above piece of code runs based on message received from kafka consumer and each time List<MyItems> can contain 10 or less items. To persist these 10 records(with unique itemid's) also it's taking ~300milliseconds. Is something wrong with the code or table structure?

答案1

得分: 3

你告诉DynamoDB你想要每秒1次读取和写入,所以你得到的就是这样。是的,你让它自动扩展到10,但需要几分钟才能看到流量并逐步增加到10。

在这种规模下最简单的方法(并假设长时间没有流量)是将表设置为按请求计费的按需定价,这样你将拥有一个极快的数据库。

英文:

You’re telling DynamoDB you want 1 read and write per second, so that’s what you’re getting. Yes, you let it auto scale up to 10 but it takes a few minutes to see the traffic and grow step by step up to 10.

Easiest at this scale (and assuming long periods of zero traffic) is set the table to on-demand per-request pricing and you’ll have an extremely fast database.

huangapple
  • 本文由 发表于 2023年3月20日 22:42:25
  • 转载请务必保留本文链接:https://go.coder-hub.com/75791713.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定