How do i capture interleaved stdout and stderr in a goloang ssh session?

How can i capture interleaved stderr/stdout output from an ssh.Session in go to model shell redirection of the form <code>2>&1</code>?

I tried to it do by combining the output of the stdout and stderr pipes from the session into a multi-reader and then used a scanner to capture the data from the multi-reader asynchronously in a go routine.

That worked, sort of. All of the data was caught but the stderr data was not interleaved. It appeared at the end.

I was able to cause the stderr output to appear at the beginning by reversing the order of the arguments to io.MultiReader() but it was still not interleaved.

Here is the output I expected.

$ ./gentestdata -i 5 -d -l -n 12 -w 32 -a 'Lorem ipsum dolor sit amet'
1 Lorem ipsum dolor sit am
2 Lorem ipsum dolor sit am
3 Lorem ipsum dolor sit am
4 Lorem ipsum dolor sit am
5 Lorem ipsum dolor sit am
6 Lorem ipsum dolor sit am
7 Lorem ipsum dolor sit am
8 Lorem ipsum dolor sit am
9 Lorem ipsum dolor sit am
10 Lorem ipsum dolor sit am
11 Lorem ipsum dolor sit am
12 Lorem ipsum dolor sit am
$ # note that two of the lines were output to stderr
$ ./gentestdata -i 5 -d -l -n 12 -w 32 -a 'Lorem ipsum dolor sit amet' 1>/dev/null
5 Lorem ipsum dolor sit am
10 Lorem ipsum dolor sit am

> The gentestdata program is something I developed to allow me to do this sort of test. The source can be found here: https://github.com/jlinoff/gentestdata.

Here is the output I saw:

$ ./sshx $(pwd)/gentestdata -i 5 -d -l -n 12 -w 32 -a 'Lorem ipsum dolor sit amet'
1 Lorem ipsum dolor sit am
2 Lorem ipsum dolor sit am
3 Lorem ipsum dolor sit am
4 Lorem ipsum dolor sit am
6 Lorem ipsum dolor sit am
7 Lorem ipsum dolor sit am
8 Lorem ipsum dolor sit am
9 Lorem ipsum dolor sit am
11 Lorem ipsum dolor sit am
12 Lorem ipsum dolor sit am
5 Lorem ipsum dolor sit am
10 Lorem ipsum dolor sit am

Note that the last two lines from stderr are out of order.

Here is the complete source code. Note the exec() function.

// Simple demonstration of how I thought that I could capture interleaved
// stdout and stderr output generated during go ssh.Session to model the
// bash 2>&1 redirection behavior.
package main
import (
"bufio"
"fmt"
"io"
"log"
"os"
"os/signal"
"runtime"
"strings"
"syscall"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/terminal"
)
func main() {
user := strings.TrimSpace(os.Getenv("LOGNAME"))
auth := getPassword(fmt.Sprintf("%v's password: ", user))
addr := "localhost:22"
if len(os.Args) > 1 {
cmd := getCmd(os.Args[1:])
config := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(auth),
},
}
exec(cmd, addr, config)
}
}
// Execute the command.
func exec(cmd string, addr string, config *ssh.ClientConfig) {
// Create the connection.
conn, err := ssh.Dial("tcp", addr, config)
check(err)
session, err := conn.NewSession()
check(err)
defer session.Close()
// Collect the output from stdout and stderr.
// The idea is to duplicate the shell IO redirection
// 2>&1 where both streams are interleaved.
stdoutPipe, err := session.StdoutPipe()
check(err)
stderrPipe, err := session.StderrPipe()
check(err)
outputReader := io.MultiReader(stdoutPipe, stderrPipe)
outputScanner := bufio.NewScanner(outputReader)
// Start the session.
err = session.Start(cmd)
check(err)
// Capture the output asynchronously.
outputLine := make(chan string)
outputDone := make(chan bool)
go func(scan *bufio.Scanner, line chan string, done chan bool) {
defer close(line)
defer close(done)
for scan.Scan() {
line <- scan.Text()
}
done <- true
}(outputScanner, outputLine, outputDone)
// Use a custom wait.
outputBuf := ""
running := true
for running {
select {
case <-outputDone:
running = false
case line := <-outputLine:
outputBuf += line + "\n"
}
}
session.Close()
// Output the data.
fmt.Print(outputBuf)
}
func check(e error) {
if e != nil {
_, _, lineno, _ := runtime.Caller(1)
log.Fatalf("ERROR:%v %v", lineno, e)
}
}
// Convert a slice of tokens to a command string.
// It inserts quotes where necessary.
func getCmd(args []string) (cmd string) {
cmd = ""
for i, token := range args {
if i > 0 {
cmd += " "
}
cmd += quote(token)
}
return
}
// Quote an individual token.
// Very simple, not suitable for production.
func quote(token string) string {
q := false
r := ""
for _, c := range token {
switch c {
case '"':
q = true
r += "\""
case ' ', '\t':
q = true
}
r += string(c)
}
if q {
r = "\"" + r + "\""
}
return r
}
func getPassword(prompt string) string {
// Get the initial state of the terminal.
initialTermState, e1 := terminal.GetState(syscall.Stdin)
if e1 != nil {
panic(e1)
}
// Restore it in the event of an interrupt.
// CITATION: Konstantin Shaposhnikov - https://groups.google.com/forum/#!topic/golang-nuts/kTVAbtee9UA
c := make(chan os.Signal)
signal.Notify(c, os.Interrupt, os.Kill)
go func() {
<-c
_ = terminal.Restore(syscall.Stdin, initialTermState)
os.Exit(1)
}()
// Now get the password.
fmt.Print(prompt)
p, err := terminal.ReadPassword(syscall.Stdin)
fmt.Println("")
if err != nil {
panic(err)
}
// Stop looking for ^C on the channel.
signal.Stop(c)
// Return the password as a string.
return string(p)
}

Any insights would be greatly appreciated.

Update #1: Tried suggestion from JimB

Modified the exec function as follows:

// Execute the command.
func exec(cmd string, addr string, config *ssh.ClientConfig) {
// Create the connection.
conn, err := ssh.Dial("tcp", addr, config)
check(err)
session, err := conn.NewSession()
check(err)
defer session.Close()
// Run the command.
b, err := session.CombinedOutput(cmd)
check(err)
// Output the data.
outputBuf := string(b)
fmt.Print(outputBuf)
}

It changed things but the output was still not interleaved. This is the output
from the test run.

     5 9FqBZonjaaWDcXMm8biABker
10 zMd1JTT3ZGR5mEuJOaJCo9AZ
1 bPlNFGdSC2wd8f2QnFhk5A84
2 H9H2FHFuvUs9Jz8UvBHv3Vc5
3 wsp2nChCIwVQztA2n95rXrtz
4 eDZ0tHBxFq6Pysq3N267L1vq
6 DF2EsjYyTQWCfIuilZxV2FCn
7 fGOILa0u1wXnEw1GDGuvdSew
8 fj84Qyu6uRn8CTECWzT5s4ZJ
9 KykqOn91fMwNqsk2Wrc5uhk2
11 0p7opMMsnA87D6TSTAXY5NAC
12 HYixe6pj0dHuKlxQyyNenUNQ

Now the stderr data shows up at the beginning.

Update #2: Showed the SSH also separated the FDs

After JimB's last comment I decided to run experiment using SSH on both a Mac and on a Linux host using gentest. Note that SSH also separates the output so this issue is resolved.

Terminal

$ # Interleaved on the terminal.
$ /user/jlinoff/bin/gentestdata -l -i 5 -w 32 -n 12
1 bPlNFGdSC2wd8f2QnFhk5A84
2 H9H2FHFuvUs9Jz8UvBHv3Vc5
3 wsp2nChCIwVQztA2n95rXrtz
4 eDZ0tHBxFq6Pysq3N267L1vq
5 9FqBZonjaaWDcXMm8biABker
6 DF2EsjYyTQWCfIuilZxV2FCn
7 fGOILa0u1wXnEw1GDGuvdSew
8 fj84Qyu6uRn8CTECWzT5s4ZJ
9 KykqOn91fMwNqsk2Wrc5uhk2
10 zMd1JTT3ZGR5mEuJOaJCo9AZ
11 0p7opMMsnA87D6TSTAXY5NAC
12 HYixe6pj0dHuKlxQyyNenUNQ

SSH

$ ssh hqxsv-cmdev3-jlinoff /user/jlinoff/bin/gentestdata -l -i 5 -w 32 -n 12
1 bPlNFGdSC2wd8f2QnFhk5A84
2 H9H2FHFuvUs9Jz8UvBHv3Vc5
3 wsp2nChCIwVQztA2n95rXrtz
4 eDZ0tHBxFq6Pysq3N267L1vq
6 DF2EsjYyTQWCfIuilZxV2FCn
7 fGOILa0u1wXnEw1GDGuvdSew
8 fj84Qyu6uRn8CTECWzT5s4ZJ
9 KykqOn91fMwNqsk2Wrc5uhk2
11 0p7opMMsnA87D6TSTAXY5NAC
12 HYixe6pj0dHuKlxQyyNenUNQ
5 9FqBZonjaaWDcXMm8biABker
10 zMd1JTT3ZGR5mEuJOaJCo9AZ

Note that the last two lines (stderr) are not interleaved.

Based on @JimB's feedback and my experiment in update #2, you must specify &> or 2>&1 in the shell command to interleave stderr and stdout.