Purging the process, Part 2

Advanced topics in pipes, filters, and redirection

Summary
Unix pipes and redirection are as useful as your text editor or a shell script. In Part 2 of this series, Mo Budlong explores some of the more advanced features of pipes and redirection. (1,900 words)

Last month I covered several basics, such as input redirection:

$ grep "hello" <hello.txt
say hello.

Purging the process: Read the whole series!
Part 1. The basics of pipes and redirections Part 2. Pipes and redirection: More advanced features

Output redirection:

$ grep "hello" >junk.txt
Now is the time
for every good person to
say hello.
(type control-D here)
$ cat junk.txt
say hello.
$

Input and output redirection, and the use of input files on the command line instead of redirected input:

$ grep "Now" <hello.txt >junk.txt
$ grep "Now" hello.txt >junk.txt

Appending additional data to a file using an output redirection:

$ echo "Now is the time" >hello.txt
$ echo "for every good person to" >>hello.txt
$ echo "say hello." >>hello.txt
$ cat hello.txt
Now is the time
for every good person to
say hello.
$

Redirecting standard output and standard error, and redirecting standard error to the /dev/null byte wastebasket:

$ find / -name *.txt -exec ls -l {} \; 2>/dev/null >textfiles
$

Basic pipes:

$ grep "hello" < hello.txt | sed -e "s/hello/bye/" > result.txt
$( grep "hello" | sed -e "s/hello/bye/" ) < hello.txt > result.txt
$

I also stated that redirecting output to an existing file would delete the file and create a new version of it. In the following example, the fourth line causes hello.txt to be overwritten with a new version of the file containing only a single line, bye.

$ echo "hello" >hello.txt
$ cat hello.txt
hello
$ echo "bye" >hello.txt
$ cat hello.txt
bye

You can set the noclobber option to prevent redirected files from automatically overwriting their predecessors. In the following example, the option causes an error message at line six when the user tries to overwrite the hello.txt file.

$ set noclobber
$ echo "hello" >hello.txt
$ cat hello.txt
hello
$ echo "bye" >hello.txt
File "hello.txt" already exists
$ cat hello.txt
hello
unset noclobber

If noclobber is set, you can force a redirection to clobber any pre-existing file by using the >| redirection operator. This operator looks like a redirection to a pipe, but it's actually just a force redirect to override the noclobber option. In the following example the forced redirection operator prevents any error messages.

$ set noclobber
$ echo "hello" >|hello.txt
$ cat hello.txt
hello
$ echo "bye" >|hello.txt
$ cat hello.txt
bye
unset noclobber

Combining standard output and standard error
Redirection is frequently used for jobs that run for a long period of time, or for jobs that produce a lot of output. For such jobs, redirection can capture the results in a file. When this is done, it's also necessary to capture any output errors. Remember that if you redirect standard output but not standard error, output will go to a file and error messages will still go to your screen. The following find command will save the results to found.txt, although errors still appear on the screen.

$ find / -name *.txt -exec ls -l {} \; >found.txt
find: /some/directory: Permission denied
find: /another/one: Permission denied
$

The redirection operator is actually a number followed by the redirection symbol, as in the following example. If number is omitted, 1 is the default.

$ find / -name *.txt -exec ls -l {} \; 1>found.txt
$

The following commands are equivalent:

$ find / -name *.txt -exec ls -l {} \; 1>found.txt
$ find / -name *.txt -exec ls -l {} \; >found.txt
$

Unix utilities open three files automatically when a program starts up. These files are given file descriptor numbers inside the program -- 0, 1, and 2 -- but they're more commonly known as stdin (standard input -- file descriptor 0), stdout (standard output -- file descriptor 1), and stderr (standard error -- file descriptor 2). When the program starts, default assignments for these files are made to /dev/tty, which is the device name for your terminal. The stdin file is assigned to the keyboard of your terminal, while stdout and stderr are assigned to the screen of your terminal. The output redirection operator defaults to 1; thus > and 1> are equivalent. The input redirection operators < and <0 are equivalent. Redirecting standard error, file descriptor 2, requires that its number be explicitly included in the redirection symbol.

The following examples use 1> to redirect standard output because it helps clarify how the redirection works. When reviewing these examples remember that > and 1> are the same.

One method of handling the logging problem would be to create separate logs for each of the outputs, as in the following example.

$ find / -name *.txt -exec ls -l {} \; 1>found.txt 2>errors.txt
$

It is also possible to redirect an output by attaching it to an already open redirection using the >& redirection operator. In the following example, the standard output of find is redirected to the file result.txt. The 2>&1 redirection command instructs the shell to attach the output from standard error (2) to the output of standard output (1). Now both standard output and standard error are sent to result.txt.

$ find / -name *.txt -exec ls -l {} \; 1>result.txt 2>&1
$

The order of redirection is important. In the following example, the output of file descriptor 2 (standard error) is attached to file descriptor 1. At this point, standard output is still attached to the terminal, so standard error is sent to the terminal. The next redirection sends standard output to result.txt. This redirection doesn't drag file descriptor 2 along with it, so standard error is left pointing to the terminal device.

$ find / -name *.txt -exec ls -l {} \; 2>&1 1>result.txt
find: /some/directory: Permission denied
find: /another/one: Permission denied
$

Input redirection from here documents
Perhaps one of the most useful forms of redirection is redirecting input from a here document. A shell script can be written that executes a command and serves all input to the command. This is frequently used for a command that is normally run interactively. As an extreme example, I will show you how to do this with the editor vi. I am using vi for two reasons: first, it's interactive, and second, you're probably fairly familiar with it already and so will have a better understanding of what the script's doing. Normally, hands-off editing is done with the sed command.

First, create a text file with several hello strings in it, as in the following example, then name it hello.txt.

sample hello.txt
hello world
hello broadway
hello dolly

Create a file named here.sh that contains the lines in the example below. The second line starts the vi editor on the hello.txt file and the <<END-OF-INPUT option states that vi will run taking its input from this current file, here.sh, reading in a line at a time until a single line containing END-OF-INPUT is read in. The subsequent lines are vi commands to globally search for hello, replace each instance of it with bye, write the file back out, then quit. The next line is the END-OF-INPUT line and final echo statement to indicate that the editing is complete.

# here.sh - sample here document
vi hello.txt <<END-OF-INPUT
:g/hello/s//bye/g
:w
:q!
END-OF-INPUT
echo "Editing complete"

Change the mode on the file to make it executable:

$ chmod a+x here.sh

When you execute the here.sh script, you may receive a warning from vi that it's not running in interactive mode. Next, the actual editing takes place; afterwards, you can cat out the hello.txt file and see your handiwork.

$ ./here.sh
Vim: Warning: Input is not from a terminal
Editing complete
$ cat hello.txt
sample bye.txt
bye world
bye broadway
bye dolly

If you really want to suppress the vi warning, redirect the error to the /dev/null device, as in the following version of here.sh:

# here.sh - sample here document
vi hello.txt 2>/dev/null <<END-OF-INPUT
:g/hello/s//bye/g
:w
:q!
END-OF-INPUT
echo "Editing complete"

here documents frequently appear as small pieces of larger scripts. In order to make the here portion stand out, it's helpful to indent that section of the shell. Using a minus (-) in front of the end-of-input marker eats the white spaces at the beginning of a line and prevents them from being passed on to the program. The following is an example:

# here.sh - sample here document
vi hello.txt 2>/dev/null <<-STOP-HERE
:g/hello/s//bye/g
:w
:q!
STOP-HERE
echo "Editing complete"

Because it's an interactive program, the ftp utility is a common candidate for here document status. The following example starts ftp and redirects standard output and standard error to xfr.log. The process logs in to a remote system named nj_system, switches to binary transfer mode, creates two directories, transfers a file named newstuff.a to the remote system, and signs out again. Using a here document makes it possible to execute ftp through a shell script while seeing what the script is doing. The second example below is another method of doing this, but it involves a separate file with the ftp commands.

# xfr.sh - Transfers to a remote system
district=nj
ftplog=xfr.log
insbase=/usr/installations
insdir=$insbase/new
inskit=newstuff.a
echo "Transferring to" $district
ftp 1>>$ftplog 2>&1 $district"_system" <<-ALL-DONE
        user mo ddd789
        binary
        mkdir $insbase
        chmod 777 $insbase
        mkdir "$insdir"
        chmod 777 $insdir
        put $inskit $insdir/$inskit
        chmod 777 $insdir/$inskit
        bye
ALL-DONE
echo "Transfer to" $district "complete."

The first file would have to contain nothing but the commands for ftp, and couldn't take advantage of script variables. Here's a sample input for ftp:

user mo ddd789
binary
mkdir /usr/installations
chmod 777 /usr/installations
mkdir /usr/installations/new
chmod 777 /usr/installations/new
put newstuff.a /usr/installations/new/newstuff.a
chmod 777 /usr/installations/new /newstuff.a
bye

# xfr.sh - Transfers to a remote system
district=nj
ftplog=xfr.log
echo "Transferring to" $district
ftp 1>>$ftplog 2>&1 $district"_system" <ftp_commands
echo "Transfer to" $district "complete."

In our next installment, I'll cover Unix system and global variables. What are they and how do you use them? I have been meaning to do this one for a while, and now seems like a good time.

Contact us for a free consultation.