Filling memory, what's happening

The simple statement, 1..1GB, fills memory and appears to be hung on a 32 GiB system. Pressing <kbd>Ctrl-C</kbd> does nothing. What is happening inside the interpreter? If it is allowed to run long enough, will the correct result be produced?

PS C:\> (Get-CimInstance -ClassName Win32_ComputerSystem).TotalPhysicalMemory / 1MB
32629.1484375
PS C:\> $PSVersionTable.PSVersion.ToString()
7.3.4
PS C:\> 1..1GB

<!-- language-all: sh -->

.., PowerShell's range operator:

• lazily enumerates the values between the (inclusive) range endpoints (in the case at at hand, 1, 2, 3, ..., 1073741824)

• when the results are captured, such as by enclosing the range operation in (...), assigning it to a variable, or making it participate in a larger expression, PowerShell creates an [object[]] array for the enumerated values (verify with (1..10).GetType()).

Given the sheer number of values, enumerating 1GB == 1,073,741,824 (1+ billion)^[1] values with 1..1GB is very slow - but does eventually return (unless you run out of memory) in PowerShell (Core) 7+, whereas in Windows PowerShell it fails right away; as zett42 notes:

• > in Windows PowerShell (64-bit), $array = 1..1GB fails with an OutOfMemoryException (Array dimensions exceeded supported range.). The actual limit is even lower ($array = 1..300MB still fails, while $array = 1..200MB succeeds).

• The reason is that .NET Framework, the legacy, Windows-only runtime underlying Windows PowerShell, has a limit not just on the element count of an array (close to 2 GB, i.e. 2+ billion elements per dimension), but also on the overall byte size of an array dimension, which is 2 GB as well.

  • Therefore, given that each element of an [object[]] array is the size of a pointer (reference) and given that in 64-bit processes each pointer requires 8 bytes (verify with [IntPtr]::Size), the actual limit is near 2GB / 8; 256MB - 8, to be precise. That is, $array = 1..(256MB - 8) works, but anything with more values fails.

  • However, the limit is higher - and the same for both PowerShell editions - if you stream the range operation instead of causing an array to be created - see next section.

• .NET (Core), the modern, cross-platform runtime underlying PowerShell (Core) 7+, doesn't have this byte-size limit any longer, but is still limited in terms of element count to near 2 GB, i.e. 2+ billion elements; zett42 has worked out the exact limit: 2GB - 57, i.e. 2,147,483,591 elements per dimension ([int]::MaxValue - 56; verify with [object[]]::new(2GB - 57).Count - anything higher, such as - 56, fails with Array dimensions exceeded supported range.)

  • As an aside: In multidimensional arrays an additional constraint is that the combined element count, across all dimensions, must not exceed 4+ billion elements (4,294,967,182 == 4GB - 114, to be precise). Use of such arrays is rare in PowerShell, however, and the more commonly used jagged arrays (arrays whose elements are other arrays) are not subject to this limit.

Note that if you use a range operation as pipeline input, without enclosing it in (...), the lazy enumeration behavior comes apparent - no array is created, and the enumerated values are streamed:

# Prints 1, 2, ..., 10 virtually *instantly*, because the values
# are *streamed* to Select-Object
1..1GB | Select-Object -First 10

If you use this form:

• you are not subject to array size and element-count constraints, so that the above works in Windows PowerShell too.

• however, you are subject to a limitation built into the range operator itself: the count of values to enumerate must fit into an [int] (System.Int32), which limits you to [int]::MaxValue + 1, i.e. 2GB values in the range; that is, with 1 as the start point, the highest end point is [int]::MaxValue ((2GB-1)).

Note that .. is unusual among PowerShell operators, in that it combines non-pipeline and pipeline behaviors: in isolation - when you neither capture it nor send it to another command - it is implicitly captured in an [object[]] array, and only after its creation does the resulting array's to-host (to-console) output start; that is, streaming does not happen unless you use the operation as input to an explicit pipeline.

• Compare the output behavior of 1..50MB to that of 1..50MB | Write-Output: the former creates a noticeable delay before output starts, due to a (large) array getting created first, whereas the latter streams its values and therefore starts producing output instantly.

The situationally streaming behavior also surfaces in the context of the foreach and switch statements:

• The following commands work in both PowerShell editions, due to .. exhibiting streaming behavior there too:

  # Outputs 1, received in a *streaming* fashion, then exits.
  # Subject only to the 2GB-1 limit.
  foreach ($i in 1..(2GB-1)) { $i; break }      
  
  # Ditto
  switch (1..(2GB-1)) { default { $_; break } }
  

^{[1] As mclayton points out, the GB suffix is the (binary) gigabtye multiplier (amounting to a factor of [Math]::Pow(2, 30) == 1,073,741,824), one of several binary multipliers that PowerShell supports in number literals.}