azure function app stalls out during long running orchestrator

I have a long running (node.js) orchestrator in Azure Function App that calls a couple hundred activity functions. Sometimes with a group of 5 or so running in parallel with context.df.Task.all. I find that it will run steadily for about two hours then the function app itself seems to abruptly stop. The logs stop displaying in the log stream. And the records in my database that the activity functions are supposed to be writing stop writing. There are no exceptions in the logs. It will remain paused or stalled like this indefinitely... until I restart the function app. Then it will come back to to life and resume where it stopped before for a time and then stop again.

• Does this behavior sound familiar to anyone?
• Should I update the extension bundle to [4.0.0, 5.0.0)
• Could my storage account be the problem? Should I create a new one?
• We are using the "Premium Plan", Could I be running up against a limit of some kind? If so what and what should I tell the IT team to increase.

As far as I know,

>- Should I update the extension bundle to [4.0.0, 5.0.0)

I believe this issue is not related to extension bundles because this is regarding on the usage compatible extensions, libraries, packages used in the Function App and extension bundle is versioned where each version comprises of Rich set of supported binding extensions to be installed based on the version of the Function App.

If any timeout value is defined in the host.json, make it as unbounded (-1) as the function project is deployed/hosting in the premium plan for the longer timeout duration of function executions.

>- Could my storage account be the problem? Should I create a new one?

• Instead of creating a new account, you can increase the quota of the Storage account to 5 PiB.
• If Storage account is in consideration, then make sure that both the function app and storage account are in same region to reduce latency issues.
• Also, in production environment - it is better to allocate a separate storage account for each azure function app.

>- We are using the "Premium Plan", Could I be running up against a limit of some kind? If so what and what should I tell the IT team to increase.

Also, you mentioned in the question that the function app stalls, with no executions after stalling and works by restart from where it has paused. I have seen some points mentioned by Microsoft even the long running functions hosted in premium plan will stops with no executions like your scenario:

Refer to the MS Doc for more information.