How to avoid PopReceiptMismatch errors in Azure durable functions

For my Azure functions application, a fan-out architecture with (sub-)orchestrators and a lot of activities running in parallel is required. Additionally, all these activities have to interact with an Azure blob storage account. When dealing with this high degree of parallelization, I often run into runtime errors like these:

[2023-02-07T14:11:36.859Z] 4c6a215f040141aaaa995f9cfbf1a234:10: An error occurred while processing message [TaskScheduled#46]: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified pop receipt did not match the pop receipt for a dequeued message.[2023-02-07T14:11:36.860Z]  ---> Microsoft.WindowsAzure.Storage.StorageException: The specified pop receipt did not match the pop receipt for a dequeued message.
[2023-02-07T14:11:36.861Z]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)[2023-02-07T14:11:36.874Z]    at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 156[2023-02-07T14:11:36.875Z]    at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)[2023-02-07T14:11:36.876Z]    at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
[2023-02-07T14:11:36.877Z] Request Information
[2023-02-07T14:11:36.879Z] RequestID:7db00ba3-1e70-42fc-84f5-f6dfb9ec630a
[2023-02-07T14:11:36.880Z] RequestDate:Tue, 07 Feb 2023 15:11:36 GMT
[2023-02-07T14:11:36.896Z] StatusMessage:The specified pop receipt did not match the pop receipt for a dequeued message.
[2023-02-07T14:11:36.897Z] ErrorCode:PopReceiptMismatch
[2023-02-07T14:11:36.898Z] ErrorMessage:The specified pop receipt did not match the pop receipt for a dequeued message.
RequestId:7db00ba3-1e70-42fc-84f5-f6dfb9ec630a
Time:2023-02-07T14:11:36.137Z
[2023-02-07T14:11:36.901Z]    --- End of inner exception stack trace ---

These neither trigger try-catch breakpoints nor stop code execution, but are really annoying and cause the orchestrators to restart which leads to data corruption down the road. Is there a proper way of fixing this?

When function projects get created in Visual Studio, the AzureWebJobsStorage value in the local.settings.json file is set to true by default. Configuring Azure functions to use a real storage account for orchestrators can eliminate concurrency issues and faulty state management. Simply replace the value with the connection string of your own storage:

{
  ...
  "Values": {
    "AzureWebJobsStorage": "[YOUR CONNECTION STRING HERE]",
    ...
  }
}