I/O redirection
(Written by Paul Cobbaut, https://github.com/paulcobbaut/, with contributions by: Alex M. Schapelle, https://github.com/zero-pytagoras/)
One of the powers of the Unix command line is the use of
input/output redirection and pipes.
This chapter explains redirection of input, output and error streams.
stdin, stdout, and stderr
The bash shell has three basic streams; it takes input from the standard input
stdin (stream 0), it sends output to the standard output stdout
(stream 1) and it sends error messages to the standard error stderr
(stream 2).
The drawing below has a graphical interpretation of these three streams.

The keyboard often serves as stdin, whereas stdout and stderr both
go to the display. This can be confusing to new Linux users because
there is no obvious way to recognize stdout from stderr. Experienced
users know that separating output from errors can be very useful.

The next sections will explain how to redirect these streams.
output redirection
> stdout
stdout can be redirected to a file with a greater than sign. While scanning
the line, the shell will see the > sign and will clear
the file.

The > notation is in fact the abbreviation of 1> (stdout being
referred to as stream 1).
[student@linux ~]$ echo It is cold today!
It is cold today!
[student@linux ~]$ echo It is cold today! > winter.txt
[student@linux ~]$ cat winter.txt
It is cold today!
[student@linux ~]$
Note that the bash shell effectively removes the redirection from the command line before argument 0 is executed. This means that in the case of this command:
echo hello > greetings.txt
the shell only counts two arguments (echo = argument 0, hello = argument 1). The redirection is removed before the argument counting takes place.
output file is erased
While scanning the line, the shell will see the > sign and
will clear the file! Since this happens before resolving argument 0,
this means that even when the command fails, the file will have been
cleared!
[student@linux ~]$ cat winter.txt
It is cold today!
[student@linux ~]$ zcho It is cold today! > winter.txt
-bash: zcho: command not found
[student@linux ~]$ cat winter.txt
[student@linux ~]$
noclobber
Erasing a file while using > can be prevented by setting the
noclobber option.
[student@linux ~]$ cat winter.txt
It is cold today!
[student@linux ~]$ set -o noclobber
[student@linux ~]$ echo It is cold today! > winter.txt
-bash: winter.txt: cannot overwrite existing file
[student@linux ~]$ set +o noclobber
[student@linux ~]$
overruling noclobber
The noclobber option can be overruled with >|.
[student@linux ~]$ set -o noclobber
[student@linux ~]$ echo It is cold today! > winter.txt
-bash: winter.txt: cannot overwrite existing file
[student@linux ~]$ echo It is very cold today! >| winter.txt
[student@linux ~]$ cat winter.txt
It is very cold today!
[student@linux ~]$
>> append
Use >> to append output to a file.
[student@linux ~]$ echo It is cold today! > winter.txt
[student@linux ~]$ cat winter.txt
It is cold today!
[student@linux ~]$ echo Where is the summer ? >> winter.txt
[student@linux ~]$ cat winter.txt
It is cold today!
Where is the summer ?
[student@linux ~]$
error redirection
2> stderr
Redirecting stderr is done with 2>. This can be very
useful to prevent error messages from cluttering your screen.

The screenshot below shows redirection of stdout to a file, and
stderr to /dev/null. Writing 1> is the
same as >.
[student@linux ~]$ find / > allfiles.txt 2> /dev/null
[student@linux ~]$
2>&1
To redirect both stdout and stderr to the same file, use
2>&1.
[student@linux ~]$ find / > allfiles_and_errors.txt 2>&1
[student@linux ~]$
Note that the order of redirections is significant. For example, the command
ls > dirlist 2>&1
directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command
ls 2>&1 > dirlist
directs only the standard output to file dirlist, because the standard error made a copy of the standard output before the standard output was redirected to dirlist.
output redirection and pipes
By default you cannot grep inside stderr when using pipes on the
command line, because only stdout is passed.
student@linux:~$ rm file42 file33 file1201 | grep file42
rm: cannot remove ‘file42’: No such file or directory
rm: cannot remove ‘file33’: No such file or directory
rm: cannot remove ‘file1201’: No such file or directory
With 2>&1 you can force stderr to go to stdout. This enables the
next command in the pipe to act on both streams.
student@linux:~$ rm file42 file33 file1201 2>&1 | grep file42
rm: cannot remove ‘file42’: No such file or directory
You cannot use both 1>&2 and 2>&1 to switch stdout and stderr.
student@linux:~$ rm file42 file33 file1201 2>&1 1>&2 | grep file42
rm: cannot remove ‘file42’: No such file or directory
student@linux:~$ echo file42 2>&1 1>&2 | sed 's/file42/FILE42/'
FILE42
You need a third stream to switch stdout and stderr after a pipe symbol.
student@linux:~$ echo file42 3>&1 1>&2 2>&3 | sed 's/file42/FILE42/'
file42
student@linux:~$ rm file42 3>&1 1>&2 2>&3 | sed 's/file42/FILE42/'
rm: cannot remove ‘FILE42’: No such file or directory
output redirection and pipes
Pipes only pass stdout by default. For example, you cannot grep inside
stderr when using pipes on the command line, because only stdout is passed.
student@linux:~$ rm file42 file33 file1201 | grep file42
rm: cannot remove ‘file42’: No such file or directory
rm: cannot remove ‘file33’: No such file or directory
rm: cannot remove ‘file1201’: No such file or directory
With 2>&1 you can force stderr to go to stdout. This enables the
next command in the pipe to act on both streams. In our example, it
allows grep to filter the stderr + stdout data.
student@linux:~$ rm file42 file33 file1201 2>&1 | grep file42
rm: cannot remove ‘file42’: No such file or directory
You cannot use both 1>&2 and 2>&1 to switch stdout and stderr.
Redirections are applied from left to right, and a>&b means "make stream
a point to wherever stream b points right now". It copies the destination,
it does not link the two streams.
The examples below use a command that writes one line to each stream,
and pipes the results into a file with | cat > piped.txt. Whatever appears
on screen went straight to the terminal before the pipe, and whatever ends up
in the file went through the pipe. By default only stdout goes through the
pipe.
student@linux:~$ { echo "I am stdout"; echo "I am stderr" >&2; } | cat > piped.txt
I am stderr
student@linux:~$ cat piped.txt
I am stdout
When you try to switch stdout and stderr by using both 2>&1 and 1>&2,
everything will be redirected to stdout and will be passed through the pipe.
Nothing will be redirected to the terminal.
student@linux:~$ { echo "I am stdout"; echo "I am stderr" >&2; } 2>&1 1>&2 | cat > piped.txt
student@linux:~$ cat piped.txt
I am stdout
I am stderr
You need a third stream to switch stdout and stderr after a pipe
symbol. Stream 3 works like a temporary variable when swapping two
values.
student@linux:~$ { echo "I am stdout"; echo "I am stderr" >&2; } 3>&1 1>&2 2>&3 | cat > piped.txt
I am stdout
student@linux:~$ cat piped.txt
I am stderr
Now stdout goes to the terminal and stderr goes through the pipe.
joining stdout and stderr
The &> construction will put both stdout and stderr in one stream similar to 2>&1.
student@linux:~$ rm file42 &> out_and_err
student@linux:~$ cat out_and_err
rm: cannot remove ‘file42’: No such file or directory
student@linux:~$ echo file42 &> out_and_err
student@linux:~$ cat out_and_err
file42
student@linux:~$
input redirection
< stdin
Redirecting stdin is done with < (short for 0<).
[student@linux ~]$ cat < text.txt
one
two
[student@linux ~]$ tr 'onetw' 'ONEZZ' < text.txt
ONE
ZZO
[student@linux ~]$
<< here document
The here document (sometimes called here-is-document) is
a way to append input until a certain sequence (usually EOF) is
encountered. The EOF marker can be typed literally or
can be called with Ctrl-D.
[student@linux ~]$ cat <<EOF > text.txt
> one
> two
> EOF
[student@linux ~]$ cat text.txt
one
two
[student@linux ~]$ cat <<brol > text.txt
> brel
> brol
[student@linux ~]$ cat text.txt
brel
[student@linux ~]$
<<< here string
The here string can be used to directly pass strings to
a command. The result is the same as using echo string | command (but
you have one less process running).
student@linux~$ base64 <<< linux-training.be
bGludXgtdHJhaW5pbmcuYmUK
student@linux~$ base64 -d <<< bGludXgtdHJhaW5pbmcuYmUK
linux-training.be
See rfc 3548 for more information about base64.
confusing redirection
The shell will scan the whole line before applying redirection. The following command line is very readable and is correct.
cat winter.txt > snow.txt 2> errors.txt
But this one is also correct, but less readable.
2> errors.txt cat winter.txt > snow.txt
Even this will be understood perfectly by the shell.
< winter.txt > snow.txt 2> errors.txt cat
quick file clear
So what is the quickest way to clear a file ?
>foo
And what is the quickest way to clear a file when the noclobber option
is set ?
>|bar
practice: input/output redirection
-
Activate the
noclobbershell option. -
Verify that
noclobberis active by repeating anlson/etc/with redirected output to a file. -
When listing all shell options, which character represents the
noclobberoption ? -
Deactivate the
noclobberoption. -
Make sure you have two shells open on the same computer. Create an empty
tailing.txtfile. Then typetail -f tailing.txt. Use the second shell toappenda line of text to that file. Verify that the first shell displays this line. -
Create a file that contains the names of five people. Use
catand output redirection to create the file and use ahere documentto end the input.
solution: input/output redirection
-
Activate the
noclobbershell option. -
Verify that
noclobberis active by repeating anlson/etc/with redirected output to a file. -
When listing all shell options, which character represents the
noclobberoption? -
Deactivate the
noclobberoption. -
Make sure you have two shells open on the same computer. Create an empty
tailing.txtfile. Then typetail -f tailing.txt. Use the second shell toappenda line of text to that file. Verify that the first shell displays this line.in the other shell:
-
Create a file that contains the names of five people. Use
catand output redirection to create the file and use ahere documentto end the input.