Is there a difference between Starlette/FastAPI Background Tasks and simply using multiprocessing in Python?

I am looking for different ways to queue up functions that will do things like copy files, scrape websites, and manipulate files (tasks that will take considerable time). I am using FastAPI as a backend API, and I came across FastAPI's background task documentation as well as Starlette's background task documentation and I fail to understand why I couldn't just use multiprocessing.

This is what I do currently using Multiprocessing and it works fine.

from multiprocessing import Process
from fastapi import FastAPI, File, UploadFile
app = FastAPI()

def handleFileUpload(file):
    print(file)
    #handle uploading file here

@app.post("/uploadFileToS3")
async def uploadToS3(bucket: str, file: UploadFile = File(...)):
    uploadProcess = Process(target=handleFileUpload, args(file))
    uploadProcess.start()
    return {
        "message": "Data has been queued for upload. You will be notified when it is ready."
        "status": "OK"
    }

If this works why would FastAPI Background Tasks exist if I can do it just as simply as using Multiprocessing? My only guess is that it has to do with scaling? It may work for myself just testing, but I know that multiprocessing has to do with the number of cores a system has. I may be completely missing the point of multiprocessing. Please help me understand. Thanks.

TL;DR

Those background tasks will always execute in the same process as your main application. They will either just run asynchronously on the event loop or in a separate thread.

For operations that are not primarily I/O, you should probably avoid using them and use multiprocessing instead.

Details

Use multiprocessing (correctly), if you want

> I fail to understand why I couldn't just use multiprocessing.

Not only does the documentation not discourage using multiprocessing, the FastAPI docs explicitly suggest it for computation intensive tasks.

Quote: (emphasis mine)

> If you need to perform heavy background computation and you don't necessarily need it to be run by the same process (for example, you don't need to share memory, variables, etc), you might benefit from using other bigger tools [...].

So you can. And if you want to do CPU-bound work in the background, you almost certainly have to use your own multiprocessing setup.

But in the example you showed in your question, it seems that the operation you want to perform in the background is to upload a file somewhere. Such a task will probably lend itself well to BackgroundTasks-based concurrency because it is I/O-bound. Spawning another process introduces additional overhead that might make it less efficient than what the BackgroundTasks do.

Also, you did not show in your code, when and how you are joining that new process. This is important and mentioned in the guidelines for multiprocessing:

> [...] when a process finishes but has not been joined it becomes a zombie. [...] it is probably good practice to explicitly join all the processes that you start.

Just spawning it and forgetting about it is probably a terrible idea, especially when that happens every time that route is requested.

And a child process can not just join itself because that would cause a deadlock.

Technical distinctions

As you know, the FastAPI background tasks are just a re-import of the BackgroundTasks class from Starlette (see docs). FastAPI just integrates them into its route handling setup in such a way that the user does not need to explicitly return them at any point.

But the Starlette docs clearly state that the class is

> for in-process background tasks.

And if we take a look at the source, we can see that under the hood it's __call__ implementation really just does one of two things:

1. If the function you passed is asynchronous, it simply awaits it.
2. If the function you passed is a "regular" function (not async), it runs it in a thread-pool. (If you go deeper, you'll see that it utilizes the anyio.to_thread.run_sync coroutine.)

This means that at no point is there another process in play. In case 1) it is even scheduled on the same exact event loop as the rest of the application, which means it is all happening in one thread. And in case 2), an additional thread performs the operation.

The implications are very obvious, if you have some experience dealing with concurrency in Python: Do not use BackgroundTasks, if you want to perform CPU-bound operations there. Those would completely block your application because they will either 1) block the event loop in the only available thread or 2) cause the GIL to lock up the main thread.

Legitimate use cases

On the flip side, if your tasks perform some I/O-bound operations (an example given in the docs is connecting to an email server to send something, after the request was processed), the BackgroundTasks machinery is very convenient.

The main benefit of BackgroundTasks to a custom setup in my opinion is that you do not need to worry about how and when exactly the coroutines will be awaited or the threads joined. That is all abstracted away behind the route handler. You just need to specify what function you want executed some time after the response.

You could just e.g. call asyncio.create_task just before the end of your route handler function. That would probably schedule the task right after the request is processed and effectively make it run in the background. But there are three problems with that:

1. There is no guarantee it will be scheduled immediately after. It may take a while, if there are a lot of requests being processed.
2. You have no chance to actually await that task and ensure it actually finishes (as expected or with an error), unless you develop some mechanism yourself to keep track of it outside the route handler.
3. Since the event loop only keeps weak references to tasks, such a task might get garbage collected before it is finished. (That means it will just straight up disappear.)

Multiprocessing Process enables you to make full use of available hardware resources, such as multiple CPU cores. By distributing workload across processes, you can take advantage of parallelism and achieve faster execution times.

The BackgroundTask feature in FastAPI is useful when you want to execute certain functions or methods asynchronously in the background while handling HTTP requests. It allows you to schedule and perform tasks that might take longer to complete or involve I/O operations without blocking the API response. It is useful to use on I/O bound tasks or Periodic/Scheduled Tasks.

However, you can use both together to achieve parallelism and asynchronous execution of tasks in a FastAPI application.

def handleFileUpload(file: UploadFile) -> None:
    print(file)

def check_worker_status(p: Process) -> None:
    while p.is_alive():
        print('Worker is still running...')
        time.sleep(5)
    p.terminate()
    print('Worker terminated')

@router.post("/uploadFileToS3")
async def uploadToS3(background_task: BackgroundTasks, bucket: str, file: UploadFile = File(...)) -> dict:
    uploadProcess = Process(target=handleFileUpload, args=(file,))
    uploadProcess.start()
    background_task.add_task(check_worker_status, uploadProcess)
    return {"message": "File uploaded successfully"}

The join() method of the multiprocessing.Process class is used to block the calling process until the process whose join() method is called terminates. If you want to avoid blocking the calling process, you can use the is_alive() method of the multiprocessing.Process class to check if the process is still running and then terminate it using the terminate() method of the same class.