如何使用 Azure.Data.Tables 创建 Azure 表的批量 upserts。

huangapple
go评论100阅读模式
英文:

How do I create bulk upserts for azure tables with Azure.Data.Tables

问题

我目前正在尝试使用Azure表存储。

我正在使用Azure.Data.Tables而不是Microsoft.WindowsAzure.Storage.Table，因为我了解到它是更新版本，将得到维护。

所以现在我有这个类，它有两个方法，一个用于插入更新，另一个用于插入更新多个实体：

  1. public class TableRepository
  2. {
  3.     private TableServiceClient _tableServiceClient;
  4.     private TableClient _tableClient;
  6.     public TableRepository(string key, string table)
  7.     {
  8.         _tableServiceClient = new TableServiceClient(key);
  9.         _tableClient = _tableServiceClient.GetTableClient(tableName: table);
  10.     }
  12.     public async Task AddAsync(TableEntity mapping)
  13.     {
  14.         await _tableClient.UpsertEntityAsync(mapping);
  15.     }
  16.     
  17.     public async Task AddEntitiesToTableAsync(IEnumerable<TableEntity> mappings)
  18.     {
  19.         foreach (var m in mappings)
  20.         {
  21.             await _tableClient.UpsertEntityAsync(m);
  22.         }
  23.     }
  24. }

对于我的批量插入，虽然它有效，但对于大量插入来说速度较慢，我想看看是否可以批量插入实体。

我发现对于Microsoft.WindowsAzure.Storage.Table，我可以执行一批操作，做类似这样的事情：

  1. public async Task AddEntitiesToTableAsync(IEnumerable<EmailToIdMapping> mappings)
  2. {
  3.     // 获取表示Azure存储表的CloudTable实例
  4.     CloudTable table = GetCloudTable(); // 用自己的逻辑替换以获取CloudTable
  6.     // 创建TableOperation的列表以保存批处理操作
  7.     List<TableOperation> batchOperations = new List<TableOperation>();
  9.     foreach (var m in mappings)
  10.     {
  11.         var tableEntity = new DynamicTableEntity(m.PartitionKey, m.RowKey)
  12.         {
  13.             Properties = new Dictionary<string, EntityProperty>
  14.             {
  15.                 { "Email", new EntityProperty(m.Email) },
  16.                 { "Id", new EntityProperty(m.Id) }
  17.             }
  18.         };
  20.         // 为实体创建InsertOrReplace操作
  21.         TableOperation upsertOperation = TableOperation.InsertOrReplace(tableEntity);
  22.         batchOperations.Add(upsertOperation);
  23.     }
  25.     // 使用批处理操作创建TableBatchOperation
  26.     TableBatchOperation batchOperation = new TableBatchOperation();
  27.     batchOperation.AddRange(batchOperations);
  29.     // 执行批处理操作
  30.     await table.ExecuteBatchAsync(batchOperation);
  31. }

但我在Azure.Data.Tables中找不到类似的东西，我唯一能想到的是以更智能的方式并行执行我已经在做的事情，以提高效率。

我可以做什么？

英文:

I am currently dipping my toes in the water with azure table storage.

I am using Azure.Data.Tables rather than Microsoft.WindowsAzure.Storage.Table, since I could read that it was the newer version, which would be maintained.

So now I have this class, which has to methods, one for upserting, and one for upserting many entities:

  1. public class TableRepository
  2. {
  6.     private TableServiceClient _tableServiceClient;
  7.     private TableClient _tableClient;
  9.     public TableRepository(string key, string table)
  10.     {
  11.         _tableServiceClient = new TableServiceClient(key);
  12.         _tableClient = _tableServiceClient.GetTableClient(tableName: table);
  13.     }
  15.     public async Task AddAsync(TableEntity mapping)
  16.     {
  18.         await _tableClient.UpsertEntityAsync(mapping);
  19.     }
  20.     public async Task AddEntitiesToTableAsync(IEnumerable<TableEntity> mappings)
  21.     {
  22.         foreach (var m in mappings)
  23.         {
  24.             await _tableClient.UpsertEntityAsync(m);
  25.         }
  26.     }
  27. }

And for my bulk upsert, this works, but is slow for many upserts, I wanted to see If I could upsert entities in bulk.

I figured out that for Microsoft.WindowsAzure.Storage.Table, I could execute a batch of operations, and do something like this:

  1. public async Task AddEntitiesToTableAsync(IEnumerable<EmailToIdMapping> mappings)
  2. {
  3.     // Retrieve the CloudTable instance representing your Azure Storage table
  4.     CloudTable table = GetCloudTable(); // Replace with your own logic to get the CloudTable
  6.     // Create a List of TableOperation to hold the batch operations
  7.     List<TableOperation> batchOperations = new List<TableOperation>();
  9.     foreach (var m in mappings)
  10.     {
  11.         var tableEntity = new DynamicTableEntity(m.PartitionKey, m.RowKey)
  12.         {
  13.             Properties = new Dictionary<string, EntityProperty>
  14.             {
  15.                 { "Email", new EntityProperty(m.Email) },
  16.                 { "Id", new EntityProperty(m.Id) }
  17.             }
  18.         };
  20.         // Create an InsertOrReplace operation for the entity
  21.         TableOperation upsertOperation = TableOperation.InsertOrReplace(tableEntity);
  22.         batchOperations.Add(upsertOperation);
  23.     }
  25.     // Create a TableBatchOperation with the batch operations
  26.     TableBatchOperation batchOperation = new TableBatchOperation();
  27.     batchOperation.AddRange(batchOperations);
  29.     // Execute the batch operation
  30.     await table.ExecuteBatchAsync(batchOperation);
  31. }

But I cannot find anything similar for Azure.Data.Tables, the only thing I can think of is to find a smarter way to do the thing I am already doing more effeciently in parralel

What can I do?

答案1

得分: 1

尝试使用Azure.Data.Tables中的SubmitTransactionAsync方法：

  1. try
  2. {
  3.     var actions = new List<TableTransactionAction>();
  5.     // 遍历实体并累加值
  6.     foreach (var entity in entities)
  7.     {
  8.         var upsertAction = new UpsertEntityAction(entity);
  9.         actions.Add(upsertAction);
  10.     }
  12.     var transaction = new TableTransaction(actions);
  13.     var tableClient = client.GetTableClient("<pass-your-table-name-here>");
  15.     // 作为批处理事务执行
  16.     await tableClient.SubmitTransactionAsync(transaction);
  17. }
  18. catch (Exception e)
  19. {
  20.     throw;
  21. }

请注意，为了成功，您应该具有相同的分区键。否则，根据我的了解，您无法实现批处理事务。

英文:

Try out the SubmitTransactionAsync method in Azure.Data.Tables

  1. try
  2. {
  3.     var actions = new List&lt;TableTransactionAction&gt;();
  5.     // Iterate over the entities and add up values
  6.     foreach (var entity in entities)
  7.     {
  8.         var upsertAction = new UpsertEntityAction(entity);
  9.         actions.Add(upsertAction);
  10.     }
  12.     var transaction = new TableTransaction(actions);
  13.     var tableClient = client.GetTableClient(&quot;&lt;pass-your-table-name-here&gt;&quot;);
  15.     // Execute as a batch transaction
  16.     await tableClient.SubmitTransactionAsync(transaction);
  17. }
  18. catch (Exception e)
  19. {
  20.     throw;
  21. }

Note that in order to you to succeed you should have the same partitioning key. Otherwise, as of my knowledge, there is no you can achieve batch transactions

huangapple
  • 本文由 发表于 2023年6月11日 23:42:01
  • 转载请务必保留本文链接：https://go.coder-hub.com/76451227.html
  • azure
  • c#
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定