正在本地处理Azure表存储的批处理时遇到困扰。

huangapple go评论108阅读模式
英文:

Struggling with batch processing in Azure Table Storage locally

问题

  1. 我有这些包含超过100个实体的批次,我试图将它们拆分并发送到我的本地Azure存储账户(Azurite):
  2. ```py
  3. def save_results_in_database(self, operations, table_name):
  4. table = self.table_service_client.get_table_client(table_name)
  5. if table_name is None:
  6. raise Exception(f"表格 {table_name} 不存在!")
  7. try:
  8. table.submit_transaction(operations)
  9. except Exception as e:
  10. logging.exception(f"在表格 {table_name} 中保存结果到数据库时发生错误")
  11. if 0 < len(results) < 100:
  12. helpers.save_results_in_database(results, action)
  13. else:
  14. # 如果结果数大于100,我们需要将它们分成每批50个的批次,并分批保存
  15. 批次大小 = 50
  16. for i in range(0, len(results), 批次大小):
  17. 批次 = results[i : i + 批次大小]
  18. helpers.save_results_in_database(批次, action)
  19. sleep(10)

由于某种原因,有些批次通过了,有些没有。例如,如果我有107个实体要推送,前50个通过了,接下来的50个没有,剩下的7个通过了。

我尝试加了一些延迟,但我不确定它是否真的有帮助。
这是错误消息:

  1. 在表格<表格名称>中保存结果到数据库时发生错误
  2. 文件“/usr/local/lib/python3.10/site-packages/azure/data/tables/_table_client.py”,第734行,在submit_transaction
  3. return self._batch_send(self.table_name, *batched_requests.requests, **kwargs) # type: ignore
  4. 文件“/usr/local/lib/python3.10/site-packages/azure/data/tables/_base_client.py”,第334行,在_batch_send
  5. raise decoded
  6. azure.data.tables._error.TableTransactionError: 2:在处理此请求时发生错误。
  7. 错误代码:InvalidInput
  8. 内容:{"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"2:在处理此请求时发生错误。\nRequestId:994b1eca-1109-4ff9-aa56-0dc73ae97a18\nTime:2023-04-06T13:08:29.278Z"}}}

我是否漏掉了什么?因为我相当确定输入在模式上是有效的。
提前致谢。

  1. <details>
  2. <summary>英文:</summary>
  3. I have these batches that contain more than 100 entities that I&#39;m trying to split to send in my local Azure storage account (Azurite):
  4. ```py
  5. def save_results_in_database(self, operations, table_name):
  6. table = self.table_service_client.get_table_client(table_name)
  7. if table_name is None:
  8. raise Exception(f&quot;Table {table_name} does not exist !&quot;)
  9. try:
  10. table.submit_transaction(operations)
  11. except Exception as e:
  12. logging.exception(
  13. f&quot;Error while saving results in database for table {table_name}&quot;
  14. )
  15. if 0 &lt; len(results) &lt; 100:
  16. helpers.save_results_in_database(results, action)
  17. else:
  18. # if the number of results is greater than 100, we need to split them in batches of 50
  19. # and save them in batches
  20. batch_size = 50
  21. for i in range(0, len(results), batch_size):
  22. batch = results[i : i + batch_size]
  23. helpers.save_results_in_database(batch, action)
  24. sleep(10)

And for some reason somes batches pass and some don't. For example if I have 107 entities to push, the first 50 pass, the second 50 doesn't and the remaining 7 pass.

I tried putting some sleep but I'm not sure it actually helps.
Here's the error message:

  1. Error while saving results in database for table &lt;table name&gt;
  2. File &quot;/usr/local/lib/python3.10/site-packages/azure/data/tables/_table_client.py&quot;, line 734, in submit_transaction
  3. return self._batch_send(self.table_name, *batched_requests.requests, **kwargs) # type: ignore
  4. File &quot;/usr/local/lib/python3.10/site-packages/azure/data/tables/_base_client.py&quot;, line 334, in _batch_send
  5. raise decoded
  6. azure.data.tables._error.TableTransactionError: 2:An error occurred while processing this request.
  7. ErrorCode:InvalidInput
  8. Content: {&quot;odata.error&quot;:{&quot;code&quot;:&quot;InvalidInput&quot;,&quot;message&quot;:{&quot;lang&quot;:&quot;en-US&quot;,&quot;value&quot;:&quot;2:An error occurred while processing this request.\nRequestId:994b1eca-1109-4ff9-aa56-0dc73ae97a18\nTime:2023-04-06T13:08:29.278Z&quot;}}}

Am I missing something? Because I'm fairly certain that the input is valid schema-wise.
TIA

答案1

得分: 0

原来我错了,其中一个 RowKeys 中有一个"/",导致操作失败。请确保检查您的数据!

英文:

Turns out I was wrong, one of the RowKeys had a "/" in it, breaking the operation.
Make sure to check your data!

huangapple
  • 本文由 发表于 2023年4月6日 21:15:44
  • 转载请务必保留本文链接:https://go.coder-hub.com/75949975.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定