Limit Running Threads to one in ASP.NET Core 6 using TaskScheduler

We have an ASP.NET Core 6 application. The user upload files and we do process those files.

Processing the files is a CPU and memory extensive operation. We are planning to take these operations outside of our main application, but until we do that we decided to try to run the task in background after uploading the file, and send email when the operation is done.

So the process of file is:

public async Task ProcessFile(string filePath, ...) 
{
    var stage1 = await ProcessStage1(...);
    var stage2 = await ProcessStage2(...);
    await SendEmail(..);
}

From the upload the file we run the task.run

public async Task UploadFile(Stream stream....)
{
   // .... uploading the file 

   Task.Factory.StartNew(() => {
     ProcessFile(....);
   })
}

Because the process is heavy, so it is consuming lots of memory, so we decided to limit the process to one file at one time by limiting the threads to only two or one.

Reading the example in .NET TaskScheduler to limit the threads to one , I implemented the code mentioned in the example, and I added the custom TaskScheduler as singleton as the following:

public interface IProcessFileFactory {
 public TaskScheduler Scheduler { get; }
    public TaskFactory Factory { get; }
} 

public ProcessFileFactory: IProcessFileFactory 
{
    public TaskScheduler Scheduler { get; }
    public TaskFactory Factory { get; }

    public ProcessFileFactory()
    {
      Scheduler = new LimitedConcurrencyLevelTaskScheduler(2);
      Factory = new TaskFactory(Scheduler);
    }
   
    // then add it as singlton in DI
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddSingleton(TaskScheduler, LimitedConcurrencyLevelTaskScheduler)
    }

    /// and then use it 

    IProcessFileFactory.Factory.StartNew(() => {
    ProcessFile(...),
    ..., IProcessFileFactory.Scheduler);

And I tried it in a different way

 Services.AddSignlton<TaskScheduler>(x => new LimitedConcurrencyLevelTaskScheduler (2) );

// and then use it
   Task.Factory.StartNew(() {
     ProcessFile() 
    },
CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskScheduler);

Both cases didn't work, and it is processing files for more than 2.

> We are planning to take these operations outside of our main application...

That is the correct solution, as I describe on my blog.

> ... TaskScheduler ...

TaskScheduler is a low-level primitive that isn't usually used with async code. It sounds like you just need a queue service. As long as you process the work items one at a time, nothing more complex is necessary.

As a reminder, in-memory queues such as channels are not durable and thus you can lose work on any application shutdown. A proper solution uses a durable queue, and (preferably) a separate worker process outside your main application to perform the operations.