In PowerShell, how can I get stdout and stderr from a native command interleave correctly when redirecting stderr to stdout?

Given a native command output(.exe) that writes output to stdout and/or stderr in the order specified on the command-line.

PowerShell 7.2 on macOS:

PS> ./output -o 1 -e 2 -o 3
1
2
3
PS> ./output -o 1 -e 2 -o 3 2>&1
1
3
2

Compare with Zsh on macOS:

% ./output -o 1 -e 2 -o 3
1
2
3
% ./output -o 1 -e 2 -o 3 2>&1
1
2
3

Windows PowerShell 5.1 (stderr is formatted differently when it's redirected, but other than that, order is the same as PowerShell 7.2 on macOS):

PS> .\output.exe -o 1 -e 2 -o 3
1
2
3
PS> .\output.exe -o 1 -e 2 -o 3 2>&1
1
3
./output.exe : 2
At line:1 char:1
+ ./output.exe -o 1 -e 2 -o 3 2>&1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Windows Command Processor (cmd.exe):

C:\> .\output.exe -o 1 -e 2 -o 3
1
2
3
C:\> .\output.exe -o 1 -e 2 -o 3 2>&1
1
2
3

What causes this behaviour in (Windows) PowerShell, and is there a way to change this behaviour?

tl;dr

• You'e seeing the unfortunate consequence of a design trade-off in PowerShell, discussed in detail in GitHub issue #5424.

• As a result of the design choices summarized below, when merging an external program's stderr output into PowerShell's success output stream with 2>&1, the original output ordering cannot be guaranteed.

• Workaround: Use the native shell's redirection features, which do guarantee output ordering (see bottom section for additional information):

  • Unix-like platforms:

    /bin/sh -c './output -o 1 -e 2 -o 3 2>&1'
    

  • Windows:

    cmd /c '.\output -o 1 -e 2 -o 3 2>&1'
    

Summary of PowerShell's behavior:

When PowerShell redirects an external program's stderr stream - which maps onto PowerShell's loosely analogous error (output) stream (2) into its success (output) stream (1, PowerShell's analog to stdout) via a redirection 2>&1 or *>&1 (the two are equivalent only for external programs), it still captures the external program's output streams separately, before sending them to the same output stream.

This allows it to send the respective stream's output as objects with distinct (.NET) data types to the success output stream (to the pipeline):

• Stdout lines are sent as strings (System.String).

  • Note that, up to at least v7.3, PowerShell invariably parses an external program's output as text,^[1] i.e. it parses it into .NET strings representing the lines of output based on the character encoding stored in [Console]::OutputEncoding], which on Windows reflects the current console's output code page. See this answer for more information.

• Stderr lines are sent as strings wrapped in System.Management.Automation.ErrorRecord instances, i.e. wrapped in the same type PowerShell uses to represent its own errors.

The potential advantage of this approach is:

• You can capture stderr lines in memory (which as of v7.3.x isn't otherwise possible; you can only send them to a file).

  • However, GitHub issue #4332 proposes introducing syntax that would allow capturing stderr output selectively in variables via a redirection such as 2>variable:stdErrOutput (to capture in variable $stdErrOutput)

• You can later separate stdout-originating lines from stderr-originating ones by their data type.

See this answer for details and sample code.

The cost of capturing the streams separately, via separate (system) pipes, before combining them is that their relative output ordering cannot be guaranteed (which appears to be an inherent limitation of such pipes).

This is problematic in lengthy program output, where you generally do want to see stderr output in context, namely right next to stdout output that was emitted just before.

Workaround via the native shells:

Since the platform-native shells (/bin/sh on Unix-like platforms, cmd.exe) do guarantee ordering, namely by merging the streams at the source - resulting in a single, unified output stream of bytes, calling them, using their (analogous) redirection features solves the problem (see top section), at the expense of:

• Being able to tell which lines came from which stream.

• The overhead of an extra child process.

• Needing to know the native shell's syntax rules, which differ from PowerShell's.

• Making cross-platform solutions difficult, due to differing path separators (/ vs. \) and quoting syntax; applied to your call:

  $shell = & ('/bin/sh', 'cmd')[$env:OS -eq 'Windows_NT']
  $opt = ('-c', '/c')[$env:OS -eq 'Windows_NT']
  $pathSep = ('/', '\')[$env:OS -eq 'Windows_NT']
  $cmdLine = '.{0}output -o 1 -e 2 -o 3 2>&1' -f $pathSep
  & $shell $opt $cmdLine
  

• On a related note, needing to use string interpolation to embed PowerShell variables and expressions in the command line passed to the native shell, using an expandable (double-quoted) string ("...")

  • This in turn can necessitate use of embedded quoting, which up to PowerShell 7.2.x is broken with respect to embedded double-quoting ("...") - see this answer

  • With cmd.exe only (Windows), you can often - but not always - mitigate this problem by passing the parts of the command line as individual arguments, making sure that 2>&1 and cmd.exe metacharacters in general are individually quoted; e.g.

    cmd /c echo $HOME '&' dir /b nosuch '2>&1'
    

^{[1] Preview versions of PowerShell (Core) v7.4 have an experimental feature named PSNativeCommandPreserveBytePipe that treats > and | when applied to external (native) programs as raw byte conduits, i.e. it bypasses the usual string-decoding and re-encoding cycle in favor of passing the raw data through.
(Note that, as with any experimental feature, it isn't guaranteed to become a stable feature.) However, the raw-bytes behavior by design does not apply when using a stderr redirection.}