Why do Python objects without circular references still get removed by garbage collection?

If a function creates a lot of objects in a loop that are not referenced elsewhere, they will be removed immediately due to Python's reference counting.
If I store the objects in a list this time, the objects will be removed by garbage collection when the function exits. The objects do not have a circular reference. Why are they not removed by ReferenceCounting when the function ends and the list is removed?
As an example I have the following program with two scenarios where I call the run function once with the parameter with_append = True and the other time with with_append = False.
In the first scenario, the garbage collection jumps in and the whole programme takes much longer to finish. In the second scenario, there is no garbage collection active. You can see this by running the program with py-spy and using the native option.
Below is the example programme and the output for both cases.

import time

COUNT = 10000000


class User:
    def __init__(self, name):
        self.name = name


def run(with_append):
    l = []
    for i in range(COUNT):
        u = User(f"user {i}")
        if with_append:
            l.append(u)


ts = time.time()
run(with_append=True)
print("function completed - ", time.time() - ts)

Scenario 1

Call run with param: with_append = True

Output of: python demo.py

function completed -  10.940133094787598

Output of: py-spy top --native -- python demo.py

Total Samples 1000
GIL: 100.00%, Active: 100.00%, Threads: 1

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                                                                                                                                  
 53.00%  53.00%    4.51s     4.51s   gc_collect_main (libpython3.10.so.1.0)
 38.00% 100.00%    4.41s    10.00s   run (demo3.py)
  9.00%  10.00%    1.07s     3.13s   __init__ (demo3.py)
  0.00%   0.00%   0.010s    0.010s   0x7f91a5b2a746 (libc-2.31.so)
  0.00% 100.00%   0.000s    10.00s   <module> (demo3.py)
  0.00%  53.00%   0.000s     4.51s   gc_collect_with_callback (libpython3.10.so.1.0)

Half of the time is spend in gc_collect_main.

Scenario 2

Call run with param: with_append = False

Output of: python demo.py

function completed -  4.351471424102783

Output of: py-spy top --native -- python demo.py

Total Samples 400
GIL: 100.00%, Active: 100.00%, Threads: 1

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                                                                                                                                  
 85.00% 100.00%    3.38s     4.00s   run (demo3.py)
 13.00%  14.00%   0.560s    0.600s   __init__ (demo3.py)
  1.00%   1.00%   0.020s    0.020s   unicode_dealloc (libpython3.10.so.1.0)
  1.00%   1.00%   0.020s    0.020s   0x7fe05fca5742 (libc-2.31.so)
  0.00%   0.00%   0.010s    0.010s   0x7fe05fca5746 (libc-2.31.so)
  0.00%   0.00%   0.010s    0.010s   0x7fe05fca5724 (libc-2.31.so)
  0.00% 100.00%   0.000s     4.00s   <module> (demo3.py)

Why is reference counting not used to free memory in both cases? And why did garbage collection take so much more time to remove the objects?

Most likely, in both scenarios, objects are deleted by reference counting.

In Scenario 1, gc_collect_main is automatically called in the middle of the loop to detect collectable objects with circular references. No objects are actually deleted by this in Scenario 1. However, this detection is performed tens of thousands of times, so it seems to be time consuming overall.

This automatic detection is triggered by an increase in the number of objects (I am not absolutely sure about this), so it is not executed during the loop in Scenario 2, where the objects are deleted at the end of each loop by reference counting.

You can put the following code at the top of your script to check when the detection was performed.

import gc

gc.set_debug(gc.DEBUG_STATS)

Alternatively, explicitly disabling detection during the loop should significantly improve performance.

import gc

def run(with_append):
    gc.disable()
    l = []
    for i in range(COUNT):
        u = User(f"user {i}")
        if with_append:
            l.append(u)
    gc.enable()

Note that this is mentioned in the official documentation.

> Since the collector supplements the reference counting already used in Python, you can disable the collector if you are sure your program does not create reference cycles.

Obviously, Python cannot automatically determine whether a circular reference exists (i.e., whether garbage collection is needed), so if this is a problem, you must explicitly disable it in this way.

Whenever you reassign u, the reference count for the old User object drops to zero, making it eligible for garbage collection, so before the loop ends it's able to collect the User object contained by u on the fly. So it waits until the end of the function to perform garbage collection on the list instead of doing it during the execution, because the reference count for the list does not reach zero until the list falls out of scope.