mod_wsgi: Concise comparison of application groups / process groups / processes / threads?

Background

I'm a Python dev trying to get comfortable using Apache and mod_wsgi in its 'daemon' mode to deploy Python web apps.

Problem

I've read a lot of the mod_wsgi docs but I'm still not sure I have a clear, concise understanding of how to think about and use the different execution contexts(?) available when deploying an app: application groups, process groups, processes, and threads. The docs go into great detail on each of them but I couldn't find a high-level explanation/summary/TLDR of how to use / think about them, so that someone unfamiliar with webserver administration can quickly grasp how they all fit together. I wish there was a TLDR for how to use each of them and how they relate to each other.

After reading through a lot of the docs, I think I have an concise picture of how I'm supposed to think about / use these different execution contexts, but I'm not sure if I'm mistaken or missing something:

They let you tweak how CPU and RAM are allocated to and shared between your web apps

Controlling the number of each type of execution context that exists on your server is like a different knob you can twist to tweak how those resources should be allocated.

They're set up in a hierarchy of parent-child relationships

The structures have a hierarchy of parent-child relationships as follows:

(From top-to-bottom, parent-to-child):

1. Application groups
2. Process groups
3. Processes
4. Threads

So an application group has one or more process groups "underneath" it, a process group has one or more processes, and a process has one or more threads.

What each does

1. Application groups - These are used to allow multiple process groups (web apps) to share a Python "sub-interpreter" so that they don't all have to load their own copies of Python modules. So it's basically a way to save RAM when running multiple process groups (web apps). [1]
2. Process groups - These correspond to your different web apps. Each web app should be in its own process group [2]. If a particular web app becomes more popular and starts getting more users, you can increase the number of processes (instances of the web app) that the process group has allocated to it.
3. Processes and threads:
   • After thinking about it for a long time, I think the easiest way to understand processes and threads is with an analogy:
     • Imagine you run a restaurant where you are trying to cook plates of food based on incoming orders as quickly as possible.
     • To handle these orders, imagine you have a certain number of chef-robots that are controlled by remote employees who connect via the Internet.
     • Imagine that it's allowed for you to have more or fewer remote employees working at one time than you have robot bodies: if you have fewer remote employees than robot bodies on a given day, some of the robot bodies will just sit there doing nothing, whereas if you have more remote employees working on a given day than robot bodies, you'll have remote employees waiting to take turns to use a robot body.
     • Imagine that you can set the maximum number of incoming orders each remote employee is allowed to be assigned to work on at the same time (like, cooking a hamburger, making pancakes, and frying eggs at the same time rather than doing them one-at-a-time).
     • Imagine that the human chefs can end up needing to wait for various things they need to finish the plate of food. For example, they may need to wait for some food to finish cooking, or wait for a busboy to bring ingredients out of storage in the basement, or wait for the dishwasher to finish cleaning some piece of equipment they need to proceed.
     • Imagine that the human controllers can swap in and out of the robot bodies extremely quickly/efficiently, so that, for example, if one human chef-controller doesn't have anything to do for a few seconds (because, for example, they're waiting for their food to finish cooking), they can instantly let another human-controller that was "waiting on the sidelines" swap into the same robot body to work on their own orders.
     • Imagine that you dictate how the human chefs should prepare each plate of food, and if you dictate an efficient method they get each plate done as quickly as possible, and if you dictate an inefficient method then they end up wasting a lot of time.
   • How the analogy applies to a server, processes, and threads:
     • The restaurant is analogous to your server.
     • Each incoming food order the kitchen receives is analogous to an incoming web (HTTP) request.
     • Each finished plate of food sent out of the kitchen is analogous to your server's response to an incoming request.
     • The remote-controlled chefs are analogous to your server's CPU cores.
     • The human controllers of the robot bodies are analogous to your server's processes.
     • The number of orders each human-chef is allowed to try to handle at the same time is analogous to the number of threads you assign per process.
     • The waiting for food to cook or ingredients to arrive is analogous to your server's processes/threads needing to wait for data to arrive from the database or from some external API before they can prepare their response.
     • Your dictating how each plate of food should be prepared is analogous to your application's code and other tools you use (such as caches) to speed up the process of preparing responses.
   • Takeaway from the analogy:
     • How many processes (human chefs) you should have depends on how many CPU cores (robot bodies) you have and how much waiting around each process (human chef) is likely to be doing while preparing a response (plate of food).
       • Note that it can make sense to have more processes (humans) than CPU cores (robot bodies) if the preparation of the responses (plates of food) involves some waiting around.
     • How many threads (simultaneous orders) you should set per process (human chef) depends on how much waiting around the chefs are likely to be doing per order.
       • If you try to get your server (restaurant) to handle incoming requests (food orders) more quickly by increasing the number of threads (simultaneous orders) per process (human chef), you may end up actually slowing things down if the preparation of the response (plate of food) doesn't involve a lot of waiting around, because the process (human chef) will have its "attention" divided.

Example server situation

• Application Group 1: Python packages A, B, C
  • Process Group 1: www.my_first_app.com
    • Process 1:
      • Thread 1: (PAUSED) Currently handling a request for User 41235; waiting for the database to respond with data.
      • Thread 2: (PAUSED) Currently handling a request for User 51234; waiting for an external API to respond with data.
      • Thread 3: (PAUSED) Currently waiting for an incoming request.
      • Thread 4: (ACTIVE) Currently handling a request for User 21455; executing the code in route "get_products()".
    • Process 2:
      • (...)
  • Process Group 2: my_second_app.com
    • (...)
• Application Group 2: Python packages X, Y, Z
  • Process Group 1: resource-heavy-api.my_first_app.com
    • (...)
  • Process Group 2: my_third_app.net
    • (...)