Capture stdout and stderr into different variables

I think before saying “you can't” do something, people should at least give it a try with their own hands…

Simple and clean solution, without using `eval` or anything exotic

1. A minimal version

{
    IFS=$'\n' read -r -d '' CAPTURED_STDERR;
    IFS=$'\n' read -r -d '' CAPTURED_STDOUT;
} < <((printf '\0%s\0' "$(some_command)" 1>&2) 2>&1)

Requires: printf, read

2. A simple test

A dummy script for producing `stdout` and `stderr`: `useless.sh`

#!/bin/bash
#
# useless.sh
#

echo "This is stderr" 1>&2
echo "This is stdout"

The actual script that will capture `stdout` and `stderr`: `capture.sh`

#!/bin/bash
#
# capture.sh
#

{
    IFS=$'\n' read -r -d '' CAPTURED_STDERR;
    IFS=$'\n' read -r -d '' CAPTURED_STDOUT;
} < <((printf '\0%s\0' "$(./useless.sh)" 1>&2) 2>&1)

echo 'Here is the captured stdout:'
echo "${CAPTURED_STDOUT}"
echo

echo 'And here is the captured stderr:'
echo "${CAPTURED_STDERR}"
echo

Output of `capture.sh`

Here is the captured stdout:
This is stdout

And here is the captured stderr:
This is stderr

3. How it works

The command

(printf '\0%s\0' "$(some_command)" 1>&2) 2>&1

sends the standard output of some_command to printf '\0%s\0', thus creating the string \0${stdout}\n\0 (where \0 is a NUL byte and \n is a new line character); the string \0${stdout}\n\0 is then redirected to the standard error, where the standard error of some_command was already present, thus composing the string ${stderr}\n\0${stdout}\n\0, which is then redirected back to the standard output.

Afterwards, the command

IFS=$'\n' read -r -d '' CAPTURED_STDERR;

starts reading the string ${stderr}\n\0${stdout}\n\0 up until the first NUL byte and saves the content into ${CAPTURED_STDERR}. Then the command

IFS=$'\n' read -r -d '' CAPTURED_STDOUT;

keeps reading the same string up to the next NUL byte and saves the content into ${CAPTURED_STDOUT}.

4. Making it unbreakable

The solution above relies on a NUL byte for the delimiter between stderr and stdout, therefore it will not work if for any reason stderr contains other NUL bytes.

Although that will rarely happen, it is possible to make the script completely unbreakable by stripping all possible NUL bytes from stdout and stderr before passing both outputs to read (sanitization) – NUL bytes would anyway get lost, as it is not possible to store them into shell variables:

{
    IFS=$'\n' read -r -d '' CAPTURED_STDOUT;
    IFS=$'\n' read -r -d '' CAPTURED_STDERR;
} < <((printf '\0%s\0' "$((some_command | tr -d '\0') 3>&1- 1>&2- 2>&3- | tr -d '\0')" 1>&2) 2>&1)

Requires: printf, read, tr

5. Preserving the exit status – the blueprint (without sanitization)

After thinking a bit about the ultimate approach, I have come out with a solution that uses printf to cache both stdout and the exit code as two different arguments, so that they never interfere.

The first thing I did was outlining a way to communicate the exit status to the third argument of printf, and this was something very easy to do in its simplest form (i.e. without sanitization).

{
    IFS=$'\n' read -r -d '' CAPTURED_STDERR;
    IFS=$'\n' read -r -d '' CAPTURED_STDOUT;
    (IFS=$'\n' read -r -d '' _ERRNO_; exit ${_ERRNO_});
} < <((printf '\0%s\0%d\0' "$(some_command)" "${?}" 1>&2) 2>&1)

Requires: exit, printf, read

6. Preserving the exit status with sanitization – unbreakable (rewritten)

Things get very messy though when we try to introduce sanitization. Launching tr for sanitizing the streams does in fact overwrite our previous exit status, so apparently the only solution is to redirect the latter to a separate descriptor before it gets lost, keep it there until tr does its job twice, and then redirect it back to its place.

After some quite acrobatic redirections between file descriptors, this is what I came out with.

The code below is a rewriting of the example that I have removed. It also sanitizes possible NUL bytes in the streams, so that read can always work properly.

{
    IFS=$'\n' read -r -d '' CAPTURED_STDOUT;
    IFS=$'\n' read -r -d '' CAPTURED_STDERR;
    (IFS=$'\n' read -r -d '' _ERRNO_; exit ${_ERRNO_});
} < <((printf '\0%s\0%d\0' "$(((({ some_command; echo "${?}" 1>&3-; } | tr -d '\0' 1>&4-) 4>&2- 2>&1- | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)

Requires: exit, printf, read, tr

This solution is really robust. The exit code is always kept separated in a different descriptor until it reaches printf directly as a separate argument.

7. The ultimate solution – a general purpose function with exit status

We can also transform the code above to a general purpose function.

# SYNTAX:
#   catch STDOUT_VARIABLE STDERR_VARIABLE COMMAND [ARG1[ ARG2[ ...[ ARGN]]]]
catch() {
    {
        IFS=$'\n' read -r -d '' "${1}";
        IFS=$'\n' read -r -d '' "${2}";
        (IFS=$'\n' read -r -d '' _ERRNO_; return ${_ERRNO_});
    } < <((printf '\0%s\0%d\0' "$(((({ shift 2; ${@}; echo "${?}" 1>&3-; } | tr -d '\0' 1>&4-) 4>&2- 2>&1- | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)
}

Requires: cat, exit, printf, read, shift, tr

ChangeLog: 2022-06-17 // Replaced ${3} with shift 2; ${@} after Pavel Tankov's comment (Bash-only).

With the catch function we can launch the following snippet,

catch MY_STDOUT MY_STDERR './useless.sh'

echo "The \`./useless.sh\` program exited with code ${?}"
echo

echo 'Here is the captured stdout:'
echo "${MY_STDOUT}"
echo

echo 'And here is the captured stderr:'
echo "${MY_STDERR}"
echo

and get the following result:

The `./useless.sh` program exited with code 0

Here is the captured stdout:
This is stderr 1
This is stderr 2

And here is the captured stderr:
This is stdout 1
This is stdout 2

8. What happens in the last examples

Here follows a fast schematization:

some_command is launched: we then have some_command's stdout on the descriptor 1, some_command's stderr on the descriptor 2 and some_command's exit code redirected to the descriptor 3
stdout is piped to tr (sanitization)
stderr is swapped with stdout (using temporarily the descriptor 4) and piped to tr (sanitization)
the exit code (descriptor 3) is swapped with stderr (now descriptor 1) and piped to exit $(cat)
stderr (now descriptor 3) is redirected to the descriptor 1, end expanded as the second argument of printf
the exit code of exit $(cat) is captured by the third argument of printf
the output of printf is redirected to the descriptor 2, where stdout was already present
the concatenation of stdout and the output of printf is piped to read

9. The POSIX-compliant version #1 (breakable)

Process substitutions (the < <() syntax) are not POSIX-standard (although they de facto are). In a shell that does not support the < <() syntax the only way to reach the same result is via the <<EOF … EOF syntax. Unfortunately this does not allow us to use NUL bytes as delimiters, because these get automatically stripped out before reaching read. We must use a different delimiter. The natural choice falls onto the CTRL+Z character (ASCII character no. 26). Here is a breakable version (outputs must never contain the CTRL+Z character, or otherwise they will get mixed).

_CTRL_Z_=$'\cZ'

{
    IFS=$'\n'"${_CTRL_Z_}" read -r -d "${_CTRL_Z_}" CAPTURED_STDERR;
    IFS=$'\n'"${_CTRL_Z_}" read -r -d "${_CTRL_Z_}" CAPTURED_STDOUT;
    (IFS=$'\n'"${_CTRL_Z_}" read -r -d "${_CTRL_Z_}" _ERRNO_; exit ${_ERRNO_});
} <<EOF
$((printf "${_CTRL_Z_}%s${_CTRL_Z_}%d${_CTRL_Z_}" "$(some_command)" "${?}" 1>&2) 2>&1)
EOF

Requires: exit, printf, read

Note: As shift is Bash-only, in this POSIX-compliant version command + arguments must appear under the same quotes.

10. The POSIX-compliant version #2 (unbreakable, but not as good as the non-POSIX one)

And here is its unbreakable version, directly in function form (if either stdout or stderr contain CTRL+Z characters, the stream will be truncated, but will never be exchanged with another descriptor).

_CTRL_Z_=$'\cZ'

# SYNTAX:
#     catch_posix STDOUT_VARIABLE STDERR_VARIABLE COMMAND
catch_posix() {
    {
        IFS=$'\n'"${_CTRL_Z_}" read -r -d "${_CTRL_Z_}" "${1}";
        IFS=$'\n'"${_CTRL_Z_}" read -r -d "${_CTRL_Z_}" "${2}";
        (IFS=$'\n'"${_CTRL_Z_}" read -r -d "${_CTRL_Z_}" _ERRNO_; return ${_ERRNO_});
    } <<EOF
$((printf "${_CTRL_Z_}%s${_CTRL_Z_}%d${_CTRL_Z_}" "$(((({ ${3}; echo "${?}" 1>&3-; } | cut -z -d"${_CTRL_Z_}" -f1 | tr -d '\0' 1>&4-) 4>&2- 2>&1- | cut -z -d"${_CTRL_Z_}" -f1 | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)
EOF
}

Requires: cat, cut, exit, printf, read, tr

Answer's history

Here is a previous version of catch() before Pavel Tankov's comment (this version requires the additional arguments to be quoted together with the command):

# SYNTAX:
#  catch STDOUT_VARIABLE STDERR_VARIABLE COMMAND [ARG1[ ARG2[ ...[ ARGN]]]]
catch() {
  {
      IFS=$'\n' read -r -d '' "${1}";
      IFS=$'\n' read -r -d '' "${2}";
      (IFS=$'\n' read -r -d '' _ERRNO_; return ${_ERRNO_});
  } < <((printf '\0%s\0%d\0' "$(((({ shift 2; ${@}; echo "${?}" 1>&3-; } | tr -d '\0' 1>&4-) 4>&2- 2>&1- | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)
}

Requires: cat, exit, printf, read, tr

Furthermore, I replaced an old example for propagating the exit status to the current shell, because, as Andy had pointed out in the comments, it was not as “unbreakable” as it was supposed to be (since it did not use printf to buffer one of the streams). For the record I paste the problematic code here:

Preserving the exit status (still unbreakable)

The following variant propagates also the exit status of some_command to the current shell:
{
  IFS= read -r -d '' CAPTURED_STDOUT;
  IFS= read -r -d '' CAPTURED_STDERR;
  (IFS= read -r -d '' CAPTURED_EXIT; exit "${CAPTURED_EXIT}");
} < <((({ { some_command ; echo "${?}" 1>&3; } | tr -d '\0'; printf '\0'; } 2>&1- 1>&4- | tr -d '\0' 1>&4-) 3>&1- | xargs printf '\0%s\0' 1>&4-) 4>&1-)
Requires: printf, read, tr, xargs

Later, Andy submitted the following “suggested edit” for capturing the exit code:

Simple and clean solution saving the exit value

We can add to the end of stderr, a third piece of information, another NUL plus the exit status of the command. It will be outputted after stderr but before stdout
{
  IFS= read -r -d '' CAPTURED_STDERR;
  IFS= read -r -d '' CAPTURED_EXIT;
  IFS= read -r -d '' CAPTURED_STDOUT;
} < <((printf '\0%s\n\0' "$(some_command; printf '\0%d' "${?}" 1>&2)" 1>&2) 2>&1)

His solution seemed to work, but had the minor problem that the exit status needed to be placed as the last fragment of the string, so that we are able to launch exit "${CAPTURED_EXIT}" within round brackets and not pollute the global scope, as I had tried to do in the removed example. The other problem was that, as the output of his innermost printf got immediately appended to the stderr of some_command, we could no more sanitize possible NUL bytes in stderr, because among these now there was also our NUL delimiter.

Trying to find the right solution to this problem was what led me to write § 5. Preserving the exit status – the blueprint (without sanitization), and the following sections.

Technically, named pipes aren't temporary files and nobody here mentions them. They store nothing in the filesystem and you can delete them as soon as you connect them (so you won't ever see them):

#!/bin/bash -e

foo () {
    echo stdout1
    echo stderr1 >&2
    sleep 1
    echo stdout2
    echo stderr2 >&2
}

rm -f stdout stderr
mkfifo stdout stderr
foo >stdout 2>stderr &             # blocks until reader is connected
exec {fdout}<stdout {fderr}<stderr # unblocks `foo &`
rm stdout stderr                   # filesystem objects are no longer needed

stdout=$(cat <&$fdout)
stderr=$(cat <&$fderr)

echo $stdout
echo $stderr

exec {fdout}<&- {fderr}<&- # free file descriptors, optional

You can have multiple background processes this way and asynchronously collect their stdouts and stderrs at a convenient time, etc.

If you need this for one process only, you may just as well use hardcoded fd numbers like 3 and 4, instead of the {fdout}/{fderr} syntax (which finds a free fd for you).

This is for catching stdout and stderr in different variables. If you only want to catch stderr, leaving stdout as-is, there is a better and shorter solution.

To sum everything up for the benefit of the reader, here is an

Easy Reusable `bash` Solution

This version does use subshells and runs without tempfiles. (For a tempfile version which runs without subshells, see my other answer.)

: catch STDOUT STDERR cmd args..
catch()
{
eval "$({
__2="$(
  { __1="$("${@:3}")"; } 2>&1;
  ret=$?;
  printf '%q=%q\n' "$1" "$__1" >&2;
  exit $ret
  )";
ret="$?";
printf '%s=%q\n' "$2" "$__2" >&2;
printf '( exit %q )' "$ret" >&2;
} 2>&1 )";
}

Example use:

dummy()
{
echo "$3" >&2
echo "$2" >&1
return "$1"
}

catch stdout stderr dummy 3 $'\ndiffcult\n data \n\n\n' $'\nother\n difficult \n  data  \n\n'

printf 'ret=%q\n' "$?"
printf 'stdout=%q\n' "$stdout"
printf 'stderr=%q\n' "$stderr"

this prints

ret=3
stdout=$'\ndiffcult\n data '
stderr=$'\nother\n difficult \n  data  '

So it can be used without deeper thinking about it. Just put catch VAR1 VAR2 in front of any command args.. and you are done.

Some if cmd args..; then will become if catch VAR1 VAR2 cmd args..; then. Really nothing complex.

Addendum: Use in "strict mode"

catch works for me identically in strict mode. The only caveat is, that the example above returns error code 3, which, in strict mode, calls the ERR trap. Hence if you run some command under set -e which is expected to return arbitrary error codes (not only 0), you need to catch the return code into some variable like && ret=$? || ret=$? as shown below:

dummy()
{
echo "$3" >&2
echo "$2" >&1
return "$1"
}

catch stdout stderr dummy 3 $'\ndifficult\n data \n\n\n' $'\nother\n difficult \n  data  \n\n' && ret=$? || ret=$?

printf 'ret=%q\n' "$ret"
printf 'stdout=%q\n' "$stdout"
printf 'stderr=%q\n' "$stderr"

Discussion

Q: How does it work?

It just wraps ideas from the other answers here into a function, such that it can easily be resused.

catch() basically uses eval to set the two variables. This is similar to https://stackoverflow.com/a/18086548

Consider a call of catch out err dummy 1 2a 3b:

let's skip the eval "$({ and the __2="$( for now. I will come to this later.
__1="$("$("${@:3}")"; } 2>&1; executes dummy 1 2a 3b and stores its stdout into __1 for later use. So __1 becomes 2a. It also redirects stderr of dummy to stdout, such that the outer catch can gather stdout
ret=$?; catches the exit code, which is 1
printf '%q=%q\n' "$1" "$__1" >&2; then outputs out=2a to stderr. stderr is used here, as the current stdout already has taken over the role of stderr of the dummy command.
exit $ret then forwards the exit code (1) to the next stage.

Now to the outer __2="$( ... )":

This catches stdout of the above, which is the stderr of the dummy call, into variable __2. (We could re-use __1 here, but I used __2 to make it less confusing.). So __2 becomes 3b
ret="$?"; catches the (returned) return code 1 (from dummy) again
printf '%s=%q\n' "$2" "$__2" >&2; then outputs err=3a to stderr. stderr is used again, as it already was used to output the other variable out=2a.
printf '( exit %q )' "$ret" >&2; then outputs the code to set the proper return value. I did not find a better way, as assigning it to a variable needs a variable name, which then cannot be used as first or second argument to catch.

Please note that, as an optimization, we could have written those 2 printf as a single one like printf '%s=%q\n( exit %q ) "$__2" "$ret"` as well.

So what do we have so far?

We have following written to stderr:

out=2a
err=3b
( exit 1 )

where out is from $1, 2a is from stdout of dummy, err is from $2, 3b is from stderr of dummy, and the 1 is from the return code from dummy.

Please note that %q in the format of printf takes care for quoting, such that the shell sees proper (single) arguments when it comes to eval. 2a and 3b are so simple, that they are copied literally.

Now to the outer eval "$({ ... } 2>&1 )";:

This executes all of above which output the 2 variables and the exit, catches it (therefor the 2>&1) and parses it into the current shell using eval.

This way the 2 variables get set and the return code as well.

Q: It uses eval which is evil. So is it safe?

As long as printf %q has no bugs, it should be safe. But you always have to be very careful, just think about ShellShock.

Q: Bugs?

No obvious bugs are known, except following:
- Catching big output needs big memory and CPU, as everything goes into variables and needs to be back-parsed by the shell. So use it wisely.
- As usual $(echo $'\n\n\n\n') swallows all linefeeds, not only the last one. This is a POSIX requirement. If you need to get the LFs unharmed, just add some trailing character to the output and remove it afterwards like in following recipe (look at the trailing x which allows to read a softlink pointing to a file which ends on a $'\n'):
```
    target="$(readlink -e "$file")x"
    target="${target%x}"
```
- Shell-variables cannot carry the byte NUL ($'\0'). They are simply ignores if they happen to occur in stdout or stderr.
The given command runs in a sub-subshell. So it has no access to $PPID, nor can it alter shell variables. You can catch a shell function, even builtins, but those will not be able to alter shell variables (as everything running within $( .. ) cannot do this). So if you need to run a function in current shell and catch it's stderr/stdout, you need to do this the usual way with tempfiles. (There are ways to do this such, that interrupting the shell normally does not leave debris behind, but this is complex and deserves it's own answer.)

Q: Bash version?

I think you need Bash 4 and above (due to printf %q)

Q: This still looks so awkward.

Right. Another answer here shows how it can be done in ksh much more cleanly. However I am not used to ksh, so I leave it to others to create a similar easy to reuse recipe for ksh.

Q: Why not use ksh then?

Because this is a bash solution

Q: The script can be improved

Of course you can squeeze out some bytes and create smaller or more incomprehensible solution. Just go for it ;)

Q: There is a typo. : catch STDOUT STDERR cmd args.. shall read # catch STDOUT STDERR cmd args..

Actually this is intended. : shows up in bash -x while comments are silently swallowed. So you can see where the parser is if you happen to have a typo in the function definition. It's an old debugging trick. But beware a bit, you can easily create some neat sideffects within the arguments of :.

Edit: Added a couple more ; to make it more easy to create a single-liner out of catch(). And added section how it works.

Ok, it got a bit ugly, but here is a solution:

unset t_std t_err
eval "$( (echo std; echo err >&2) \
        2> >(readarray -t t_err; typeset -p t_err) \
         > >(readarray -t t_std; typeset -p t_std) )"

where (echo std; echo err >&2) needs to be replaced by the actual command. Output of stdout is saved into the array $t_std line by line omitting the newlines (the -t) and stderr into $t_err.

If you don't like arrays you can do

unset t_std t_err
eval "$( (echo std; echo err >&2 ) \
        2> >(t_err=$(cat); typeset -p t_err) \
         > >(t_std=$(cat); typeset -p t_std) )"

which pretty much mimics the behavior of var=$(cmd) except for the value of $? which takes us to the last modification:

unset t_std t_err t_ret
eval "$( (echo std; echo err >&2; exit 2 ) \
        2> >(t_err=$(cat); typeset -p t_err) \
         > >(t_std=$(cat); typeset -p t_std); t_ret=$?; typeset -p t_ret )"

Here $? is preserved into $t_ret

Tested on Debian wheezy using GNU bash, Version 4.2.37(1)-release (i486-pc-linux-gnu).

Capture stdout and stderr into different variables

Simple and clean solution, without using `eval` or anything exotic

1. A minimal version

2. A simple test

A dummy script for producing `stdout` and `stderr`: `useless.sh`

The actual script that will capture `stdout` and `stderr`: `capture.sh`

Output of `capture.sh`

3. How it works

4. Making it unbreakable

5. Preserving the exit status – the blueprint (without sanitization)

6. Preserving the exit status with sanitization – unbreakable (rewritten)

7. The ultimate solution – a general purpose function with exit status

8. What happens in the last examples

9. The POSIX-compliant version #1 (breakable)

10. The POSIX-compliant version #2 (unbreakable, but not as good as the non-POSIX one)

Answer's history

Preserving the exit status (still unbreakable)

Simple and clean solution saving the exit value

Easy Reusable `bash` Solution

Addendum: Use in "strict mode"

Discussion

Tags:

Shell

Command Line

Bash

Io Redirection

Related

Recent Posts

Capture stdout and stderr into different variables

Simple and clean solution, without using eval or anything exotic

1. A minimal version

2. A simple test

A dummy script for producing stdout and stderr: useless.sh

The actual script that will capture stdout and stderr: capture.sh

Output of capture.sh

3. How it works

4. Making it unbreakable

5. Preserving the exit status – the blueprint (without sanitization)

6. Preserving the exit status with sanitization – unbreakable (rewritten)

7. The ultimate solution – a general purpose function with exit status

8. What happens in the last examples

9. The POSIX-compliant version #1 (breakable)

10. The POSIX-compliant version #2 (unbreakable, but not as good as the non-POSIX one)

Answer's history

Preserving the exit status (still unbreakable)

Simple and clean solution saving the exit value

Easy Reusable bash Solution

Addendum: Use in "strict mode"

Discussion

Tags:

Shell

Command Line

Bash

Io Redirection

Related

Simple and clean solution, without using `eval` or anything exotic

A dummy script for producing `stdout` and `stderr`: `useless.sh`

The actual script that will capture `stdout` and `stderr`: `capture.sh`

Output of `capture.sh`

Easy Reusable `bash` Solution