Pipes and Redirection

Pipes and redirection are the two shell features that let programs be wired together and connected to files without any program knowing about the others. They rest on a single idea: every Unix process has standard streams, standard input (file descriptor 0), standard output (file descriptor 1), and standard error (file descriptor 2), and the shell can attach those streams to whatever it likes before the program runs.

Redirection points an individual stream at a file. The GNU Bash Reference Manual’s Redirections section explains that if a file descriptor number is omitted, a redirection operator beginning with < refers to standard input (descriptor 0) and one beginning with > refers to standard output (descriptor 1). So command > file sends output into a file, overwriting it; command >> file appends instead; and command < file feeds the file in as input. The same section covers redirecting standard error and combining streams, for example sending both standard output and standard error to one file.

The pipe goes a step further and connects two running programs directly. The manual’s Pipelines section defines a pipeline as a sequence of one or more commands separated by the control operator | (or |&). The shell runs the commands concurrently and connects the standard output of each to the standard input of the next, so data flows through the chain as it is produced rather than being staged in a temporary file. A search tool can feed a sort tool which feeds a counter, all in one line, with no intermediate files and no program aware it is part of a pipeline.

The pipe is one of the defining inventions of Unix, credited to Doug McIlroy and added by Ken Thompson. It is the concrete mechanism behind the Unix philosophy of small programs that each do one thing well: because every tool reads text from standard input and writes text to standard output, tools written years apart can be combined in ways their authors never anticipated. Text becomes a universal interface, and the pipe is the connector.

The exit behavior of a pipeline matters for scripting. The Pipelines section notes that, by default, the exit status of a pipeline is the status of its last command, unless the pipefail option is enabled, in which case the pipeline reports the rightmost command that exited non-zero. This detail is a common source of silent bugs, since a failure early in a pipeline can be masked by a successful final stage unless pipefail is set.

Together, pipes and redirection turn the shell from a command launcher into a dataflow language. With a handful of operators a user assembles custom one-off programs out of existing tools, which is why this small set of symbols, |, >, <, and >>, is among the most consequential notation in computing.