如何使用 Azure.Data.Tables 创建 Azure 表的批量 upserts。

huangapple go评论68阅读模式
英文:

How do I create bulk upserts for azure tables with Azure.Data.Tables

问题

我目前正在尝试使用Azure表存储。

我正在使用Azure.Data.Tables而不是Microsoft.WindowsAzure.Storage.Table,因为我了解到它是更新版本,将得到维护。

所以现在我有这个类,它有两个方法,一个用于插入更新,另一个用于插入更新多个实体:

public class TableRepository
{
    private TableServiceClient _tableServiceClient;
    private TableClient _tableClient;

    public TableRepository(string key, string table)
    {
        _tableServiceClient = new TableServiceClient(key);
        _tableClient = _tableServiceClient.GetTableClient(tableName: table);
    }

    public async Task AddAsync(TableEntity mapping)
    {
        await _tableClient.UpsertEntityAsync(mapping);
    }
    
    public async Task AddEntitiesToTableAsync(IEnumerable<TableEntity> mappings)
    {
        foreach (var m in mappings)
        {
            await _tableClient.UpsertEntityAsync(m);
        }
    }
}

对于我的批量插入,虽然它有效,但对于大量插入来说速度较慢,我想看看是否可以批量插入实体。

我发现对于Microsoft.WindowsAzure.Storage.Table,我可以执行一批操作,做类似这样的事情:

public async Task AddEntitiesToTableAsync(IEnumerable<EmailToIdMapping> mappings)
{
    // 获取表示Azure存储表的CloudTable实例
    CloudTable table = GetCloudTable(); // 用自己的逻辑替换以获取CloudTable

    // 创建TableOperation的列表以保存批处理操作
    List<TableOperation> batchOperations = new List<TableOperation>();

    foreach (var m in mappings)
    {
        var tableEntity = new DynamicTableEntity(m.PartitionKey, m.RowKey)
        {
            Properties = new Dictionary<string, EntityProperty>
            {
                { "Email", new EntityProperty(m.Email) },
                { "Id", new EntityProperty(m.Id) }
            }
        };

        // 为实体创建InsertOrReplace操作
        TableOperation upsertOperation = TableOperation.InsertOrReplace(tableEntity);
        batchOperations.Add(upsertOperation);
    }

    // 使用批处理操作创建TableBatchOperation
    TableBatchOperation batchOperation = new TableBatchOperation();
    batchOperation.AddRange(batchOperations);

    // 执行批处理操作
    await table.ExecuteBatchAsync(batchOperation);
}

但我在Azure.Data.Tables中找不到类似的东西,我唯一能想到的是以更智能的方式并行执行我已经在做的事情,以提高效率。

我可以做什么?

英文:

I am currently dipping my toes in the water with azure table storage.

I am using Azure.Data.Tables rather than Microsoft.WindowsAzure.Storage.Table, since I could read that it was the newer version, which would be maintained.

So now I have this class, which has to methods, one for upserting, and one for upserting many entities:

public class TableRepository
{



    private TableServiceClient _tableServiceClient;
    private TableClient _tableClient;

    public TableRepository(string key, string table)
    {
        _tableServiceClient = new TableServiceClient(key);
        _tableClient = _tableServiceClient.GetTableClient(tableName: table);
    }

    public async Task AddAsync(TableEntity mapping)
    {

        await _tableClient.UpsertEntityAsync(mapping);
    }
    public async Task AddEntitiesToTableAsync(IEnumerable<TableEntity> mappings)
    {
        foreach (var m in mappings)
        {
            await _tableClient.UpsertEntityAsync(m);
        }
    }
}

And for my bulk upsert, this works, but is slow for many upserts, I wanted to see If I could upsert entities in bulk.

I figured out that for Microsoft.WindowsAzure.Storage.Table, I could execute a batch of operations, and do something like this:

public async Task AddEntitiesToTableAsync(IEnumerable<EmailToIdMapping> mappings)
{
    // Retrieve the CloudTable instance representing your Azure Storage table
    CloudTable table = GetCloudTable(); // Replace with your own logic to get the CloudTable

    // Create a List of TableOperation to hold the batch operations
    List<TableOperation> batchOperations = new List<TableOperation>();

    foreach (var m in mappings)
    {
        var tableEntity = new DynamicTableEntity(m.PartitionKey, m.RowKey)
        {
            Properties = new Dictionary<string, EntityProperty>
            {
                { "Email", new EntityProperty(m.Email) },
                { "Id", new EntityProperty(m.Id) }
            }
        };

        // Create an InsertOrReplace operation for the entity
        TableOperation upsertOperation = TableOperation.InsertOrReplace(tableEntity);
        batchOperations.Add(upsertOperation);
    }

    // Create a TableBatchOperation with the batch operations
    TableBatchOperation batchOperation = new TableBatchOperation();
    batchOperation.AddRange(batchOperations);

    // Execute the batch operation
    await table.ExecuteBatchAsync(batchOperation);
}

But I cannot find anything similar for Azure.Data.Tables, the only thing I can think of is to find a smarter way to do the thing I am already doing more effeciently in parralel

What can I do?

答案1

得分: 1

尝试使用Azure.Data.Tables中的SubmitTransactionAsync方法:

try
{
    var actions = new List<TableTransactionAction>();

    // 遍历实体并累加值
    foreach (var entity in entities)
    {
        var upsertAction = new UpsertEntityAction(entity);
        actions.Add(upsertAction);
    }

    var transaction = new TableTransaction(actions);
    var tableClient = client.GetTableClient("<pass-your-table-name-here>");

    // 作为批处理事务执行
    await tableClient.SubmitTransactionAsync(transaction);
}
catch (Exception e)
{
    throw;
}

请注意,为了成功,您应该具有相同的分区键。否则,根据我的了解,您无法实现批处理事务。

英文:

Try out the SubmitTransactionAsync method in Azure.Data.Tables

try
{
    var actions = new List&lt;TableTransactionAction&gt;();

    // Iterate over the entities and add up values
    foreach (var entity in entities)
    {
        var upsertAction = new UpsertEntityAction(entity);
        actions.Add(upsertAction);
    }

    var transaction = new TableTransaction(actions);
    var tableClient = client.GetTableClient(&quot;&lt;pass-your-table-name-here&gt;&quot;);

    // Execute as a batch transaction
    await tableClient.SubmitTransactionAsync(transaction);
}
catch (Exception e)
{
    throw;
}

Note that in order to you to succeed you should have the same partitioning key. Otherwise, as of my knowledge, there is no you can achieve batch transactions

huangapple
  • 本文由 发表于 2023年6月11日 23:42:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/76451227.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定