Purging the process

Introduction to pipes, filters, and redirection, Part 1

Summary
If you've arrived at Unix from the graphical user interface (GUI) world of Windows or Mac OS, you're probably not familiar with pipes and filters. Even among character-based interfaces, only a few of them, such as MS-DOS, provide even rudimentary pipes and redirection. This month, Mo Budlong walks you through the process of piping and redirecting standard and error outputs. (1,900 words)

Purging the process: Read the whole series!
Part 1. The basics of pipes and redirections Part 2. Pipes and redirection: More advanced features

When a typical Unix utility starts up, three files are automatically opened for you inside of it. These files are given file descriptor numbers inside the program -- 0, 1, and 2 -- but they're more commonly known as stdin (standard in -- file descriptor: 0), stdout (standard out -- file descriptor: 1) and stderr (standard error -- file descriptor: 2). When the program starts, default assignments for these files are made to /dev/tty, which is the device name for your terminal. The stdin file is assigned to the keyboard of your terminal, while stdout and stderr are assigned to its screen.

Let's start with a simple example using grep. Type a grep command to find lines containing the word hello, then type the following lines at your terminal. At the end of each line press Enter to move down to the next line. Watch what happens as you type say hello.

$ grep "hello"
Now is the time
for every good person to
say hello.

The screen repeats the last line.

$ grep "hello"
Now is the time
for every good person to
say hello.
say hello.

Hold down the Control key and press D to end the input to grep. Control-D is an end-of-file marker and can be entered as a keystroke to stop any utility that is taking its input from the keyboard.

The grep "hello" line is a command to search standard input for lines containing hello and echo any such line found to standard output. The Unix console automatically echoes anything you type, so the three lines appear on the screen as you type them. Then grep hits a line containing hello and decides to output it to standard out, and say hello appears on the screen a second time. The second appearance is the output from grep.

Standard output can be redirected to a file using the right angle bracket (>) as shown in the example below. The same grep command is redirected to send its output to a file named junk.txt. The say hello line doesn't appear a second time because it's been directed to the junk.txt file. After the user presses Control-D, cat is used to display the contents of junk.txt, which contains grep's single output line.

$ grep "hello" >junk.txt
Now is the time
for every good person to
say hello.
(type control-D here)
$ cat junk.txt
say hello.
$

Standard input can be redirected to come from a file by using the left angle bracket (<). In order to demonstrate this, we need a file that can be used for input. Use vi to create the following sample file and save it as hello.txt.

Now is the time
for every good person to
say hello.

When you type the following command, notice that the output from grep is the single say hello. Because input is being drawn from a file, you don't need to use Control-D to stop the process.

$ grep "hello" <hello.txt
say hello.

Both standard input and output are redirected in the following example. Once grep starts up, it takes its input from hello.txt and outputs the result to junk.txt. There is no output on the screen, but you can use cat to display junk.txt and verify the contents.

$ grep "hello" <hello.txt>junk.txt
$ cat junk.txt
say hello.
$

If a redirection to an output file encounters a file that already exists, that file is destroyed and a new one, containing the new output, is created, assuming the user has appropriate permissions to delete and create a new file. You can confirm this by using the previous example to search for a different line of text. In this example, the earlier version of junk.txt has been replaced with the new output from grep, the single line Now is the time

$ grep "Now" <hello.txt >junk.txt
$ cat junk.txt
Now is the time
$

There is a convention used in Unix programs which dictates that, if a file is expected as input to a program but no file is named on the command line, standard input is used. Because grep is designed to search for a string in a file, or files, it uses a command-line syntax that lets you name a file on the command line, and the input redirection symbol is not needed. Internally, grep checks if a file is named on the command line and opens and uses it. If no file name is found, standard input is used. The following command lines for grep have the identical effect.

Internally, the first command reassigns hello.txt to standard input and uses it for input; the second command opens hello.txt as a file and uses it for input. grep doesn't expect an output file to be named on the command line. To get the output into a file, you must use output redirection. It doesn't hurt to redirect grep input, but in the case of grep, the redirection is already taken care of for you on the command line.

$ grep "Now" <hello.txt >junk.txt
$ grep "Now" hello.txt >junk.txt

If you want to preserve the existing output file and append new information to it, use a double right angle bracket (>>). The following example uses echo, which normally outputs to the screen, to create the hello.txt file without using an editor. The output of the echo command is redirected into the file, and two more lines are appended to it.

$ echo "Now is the time" >hello.txt
$ echo "for every good person to" >>hello.txt
$ echo "say hello." >>hello.txt
$ cat hello.txt
Now is the time
for every good person to
say hello.
$

Pipes are created as a means of taking the output of one program and using it as the input to another. The pipe symbol (|) is used as a connector between the two programs. In the following example, look at the first part of the command up to the first pipe symbol. The cat command normally outputs to the screen; in this case, however, the output has been sent into a pipe. On the righthand side of the pipe, this output becomes the input to grep "hello". The output from grep "hello" is in turn sent into another pipe. On the right side of that pipe, the output is used as standard input to a sed command that searches for hello and replaces it with bye. The final result is redirected to a file named result.txt which cat displays on the screen as say bye.

$cat hello.txt | grep "hello" | sed -e "s/hello/bye/" > result.txt
$cat result.txt
say bye.
$

If this were broken down step by step using simple redirection, you would need several commands, as well as the final rm steps to clean up the intermediate work files that were created.

$cat hello.txt >wrk1.txt
$ grep "hello" <wrk1.txt >wrk2.txt
$ sed -e "s/hello/bye/" &ltwrk2.txt >result.txt
$cat result.txt
say bye.
$rm wrk1.txt wrk2.txt

The initial step of getting hello.txt into the grep command could also be done in several other ways. Two examples are shown below. The first redirects input to grep from hello.txt on the lefthand side of the pipe; the second puts parentheses around the grep and sed commands, groups them as a subprocess, then redirects input and output to the grouped process.

$ grep "hello" < hello.txt | sed -e "s/hello/bye/" > result.txt
$( grep "hello" | sed -e "s/hello/bye/" ) < hello.txt > result.txt
$

Redirecting standard error output
So far I've only shown you how to pipe and redirect standard output, but it's frequently useful to do something with error output. In the following example, find is being used to search the entire system (starting at / ) for files with a .txt extension. Whenever one is found, its full directory entry is placed in a file named textfiles. The example below shows sample error messages that are generated when find attempts to access an unavailable directory.

$ find / -name *.txt -exec ls -l {} \; >textfiles
find: /some/directory: Permission denied
find: /another/one: Permission denied
$

The error messages can be suppressed by redirecting them to /dev/null, which is a special device that can be thought of as a wastebasket for bytes written to it on output. Everything that goes to /dev/null disappears. To redirect standard error, use a right angle bracket preceded by a 2, which is the file descriptor number for standard error. If you don't care about error messages, send them to the /dev/null byte bucket.

$ find / -name *.txt -exec ls -l {} \; 2>/dev/null >textfiles
$

The following command combines redirection and pipes to extract and bring a full list of all .txt files sorted in order by the third field in the ls -l directory entry, the owner's name.

$ find / -name *.txt -exec ls -l {} \; 2>/dev/null |sort -k 3 >textfiles
$

Shell scripts can also redirect their output, so the above command could be put into a shell script without redirection, but the output can be redirected when the command is executed.

#!/usr/bin/sh
# usertexts
#    outputs a listing of texts files on the system, ordered by owner id

find / -name *.txt -exec ls -l {} \; 2>/dev/null |sort -k 3

This shell's script could be executed with the output redirection done at the shell script level.

$ usertexts >textfiles
$

Pipes and redirection can be combined to create very powerful tools that start a text stream and then apply different tools to that stream, filtering it as it passes through different processes.

Next month, I'll take a look at more advanced uses of pipes and redirection.

Contact us for a free consultation.