Unix command line 101: How much do you know?

Arguments and options on the half-shell

Summary
Arguments and options are those mysterious little nuggets preceded by minus signs, file names, and other Unix arcana that appear on a command line following the command to be executed. This month Mo unveils the power of these commands, but not before he breaks it down with an examination of the Unix command line itself. (2,600 words)

A Unix command line is a sequence of characters in the syntax of the target shell language. Of the characters found there, some are known as metacharacters, which have a special meaning to the shell. The metacharacters in the Korn shell are:

• ; Separates multiple commands on a command line
• & Causes the preceding command to execute synchronously (i.e., at the same time as the next command on the command line)
• () Launches commands enclosed in parentheses in a separate shell
• | Pipes the output of the command to the left of the pipe to the input of the command on the right of the pipe
• > Redirects output to a file or device
• < Redirects input from a file or device
• Newline Ends a command or set of commands
• Space Separator between command words
• Tab Separator between command words

(Note: Some of these metacharacters can be used in combinations, such as || and &&. Consult your manual for a complete description.)

With these metacharacters in mind, you can define a command line word -- a sequence of characters separated by one or more nonquoted metacharacters. In the following example, the passwd file is piped through the cut program, and fields 1 and 3 are output based on a colon delimiter.

In the following command line,

cat /etc/passwd|cut -d ":" -f 1,3 >usruid.txt

the words are:

cat
/etc/passwd
cut
-d
":"
-f
1,3
usruid.txt

Note that the metacharacters |, >, and the space have been removed, and that the metacharacters &, |, (), ;, and the newline are used to separate or terminate multiple commands within a command line.

In our example command line, there are two commands separated by the pipe (|) symbol:

cat /etc/passwd

and

cut -d ":" -f 1,3

The final portion of the command line -- >usruid.txt -- could be thought of as the command "and output the result to usruid.txt," although redirection is not usually considered part of a command.

When a command executes a Unix program, utility, or shell script, it's usual for the command to include arguments. In the example above, the argument to cat is /etc/passwd. The arguments to cut are -d, ":", -f, and (1,3).

In general, arguments are all the words (note the definition of word above) that follow an executable program name in a command. Arguments within a command are separated from one another by spaces or tabs (metacharacters). Most Unix programs were written with standards for the arrangement of arguments and options.

Options are the letters or numbers that follow a minus sign. A simple example of arguments and options would be the use of the cat command:

cat -v -e -t doodah.txt

In the above example, the arguments are -v, -e, -t, and doodah.txt, while the options are the entries for -v, -e, and -t. The -v option asks cat to display all characters, even nonprintable ones; the -e option specifies that the end of a line will be displayed as $; and the -t option specifies that a tab should be displayed as ^I instead of expanding the tab into spaces on the screen.

Unfortunately, no standard terminology has been developed to differentiate an option from a nonoption argument, which is all the more confusing when one considers that an option can itself have an argument. In the first example using cut, the -d option has an option argument of ":", and the -f option has an option argument of (1,3). In order to clarify these, various manuals have adopted standards for naming conventions for the parts of a Unix command. The following examples illustrate the parts:

cat -v -e -t doodah.txt

cat /etc/passwd|cut -d ":" -f 1,3 >usruid.txt

The program name itself, cat in the first example and cat and cut in the second, is variously called the name, progname, executable, or program-name. The nonoption arguments to a command, doodah.txt in the first example and /etc/passwd in the second, is called an operand or cmdarg.

The options -v,-e, and -t in the first example, and -d and -f in the second, are called options, opts, or switches. The arguments to options, ":" and (1,3) in the second example are called option-arguments, or optargs.

The standards used in creating Unix executables are:

1. Command names must be between two and nine characters long

2. Command names must include only lowercase letters and digits

3. Option names (options above) must be only one character long

4. All options must be preceded by -

5. Options with no arguments may be grouped after a single - (e.g., -v -e -t could also be written as -vet)

6. The first option-argument following an option must be preceded by white space

7. Option-arguments cannot be optional

8. Groups of option-arguments following an option must either be separated by commas or by white space, and quoted (e.g., -f 1,3 or -o "xxx z yy")

9. All options must precede operands on the command line

10. -- may be used to indicate the end of the options

11. The order of the options relative to one another should not matter

12. The relative order of the operands (cmdargs) may affect their significance in ways determined by the command with which they appear

13. - preceded and followed by white space should only be used to mean standard input

Not all Unix commands follow these rules, although all the newer ones do. Older executables were written before the standard was established, but executables dating from these times are in such regular use that it was decided not to change them. For example, cut will function with or without rule number six, which requires a space before the option-argument. Both of the following commands will work on most systems.

cat /etc/passwd|cut -d ":" -f 1,3 >usruid.txt
cat /etc/passwd|cut -d: -f1,3 >usruid.txt

The find command is another example of an antiquated program still in use today. It uses options longer than a single character, which violates rule number three, and allows options to appear after the operand, thus violating rule number nine. In the following example, dot (.) is the operand, -name and -print are options, and data.txt is the option-argument for -name.

find . -name data.txt -print.

The getopts function
You're probably wondering what all this blather is leading up to. Well, Unix provides a handy tool for separating option arguments and operands, and it's known as the getopts function. This function is called by following getopts with a string (which contains the list of valid option characters) and a shell variable (which receives the result of searching the arguments). The function can be called several times, and each time it steps forward through the list of arguments and picks up the next option. It can also pick up an option-argument, and the index of the argument that it has processed.

To illustrate this, imagine a shell script that will archive a file by copying it to an archive directory. The default directory is /u/arch, but the path of the archive directory can be changed on the command line. The archive program will also stop and ask you if it is about to overwrite an earlier archive, but an option can be set to overwrite without warning. A sample command line for this archive program would be:

arch [-r] [-a /new/archive/path] filename

In this example, the -r option will automatically replace an existing archive file without warning, though the default is to warn. The -a option is followed by an alternative archive directory to use instead of /u/archive. Finally, filename is the name of the file to archive.

The following is a listing for arch that covers the processing of the option arguments. It does not include the logic for doing the actual archive operation. Below the complete listing is a step-by-step analysis of how the program works.

#! /bin/sh
#:------------------------------------------------------------
--
usage () {
 echo "Usage:"
 echo "arch - archives a file to /u/arch directory"
 echo "syntax:"
 echo "    arch [-r] [-a /new/archive/path] filename"
 echo "where"
 echo "    -r will automatically replace an existing archive
file"
 echo "       (default is to warn)"
 echo "    -a specifies an alternative archive directory"
 echo "    filename is the name of the file to archive"
 exit
}
#:------------------------------------------------------------
--

replace="w"
arch="/u/arch"
filename=""

optstr=":ra:"

while getopts $optstr opt
do

    case $opt in
        r) replace="r";;
        a) arch=$OPTARG;;
        *) usage;;
    esac
done

shift `expr $OPTIND - 1`

filename=$1

echo "Archiving" $filename " to " $arch "with" $replace
"replace option"

# rest of the code goes here

The getopts function does not always work correctly with the Korn shell, so line 1 forces the script to run in the Bourne shell. The program begins lines 2 through 15 with a comment describing its actions that also doubles as a usage function, which is called when the user makes a mistake.

 1   #! /bin/sh
 2  
#:------------------------------------------------------------
--
 3   usage () {
 4    echo "Usage:"
 5    echo "arch - archives a file to /u/arch directory"
 6    echo "syntax:"
 7    echo "    arch [-r] [-a /new/archive/path] filename"
 8    echo "where"
 9    echo "    -r will automatically replace an existing
archive file"
10    echo "       (default is to warn)"
11    echo "    -a specifies an alternative archive
directory"
12    echo "    filename is the name of the file to archive"
13    exit
14   }
15  
#:------------------------------------------------------------
--

Shell variables are set up at lines 17 and 18 and contain the default values used for archiving and the archive directory, and makes a replacement warning default behavior. Line 19 sets up a variable for the file to be archived:

17   replace="w"
18   arch="/u/arch"
19   filename=""

This program has two possible options: -r and -a. The -a option requires an option-argument that names the directory to use. An options string should contain the list of single character identifiers to be used for options, or ra. In addition, if an option is to be preceded by an option argument, a colon should immediately follow it. Finally, getopts will produce an error message if an invalid option is placed on the command line. In order to suppress the error message, start the option string with a colon or ":ra:". That string is set up at line 21 of the script.

21   optstr=":ra:"

Whenever getopts is called, it locates the next available option, retrieves the character, and places it in the passed variable name. At line 23 this variable, $opt, is passed as the second argument to getopts after $optstr. The getopts function returns true as long as it continues to find arguments that start with a leading -. When it finds -r on the command line, it places r in $opt. When it finds -a, it places a in $opt. Whenever getopts finds an option expecting an option-argument, it retrieves the argument and places it in a variable named $OPTARG. The loop at lines 23 through 31 processes all options and option-arguments by repetitively calling getopts. Inside a case statement the various results are processed.

If -r was encountered, it will appear in $opt and $replace will be set to r. If -a was encountered, a will appear in $opt, and the value in $OPTARG will be used to set the value of $arch. If anything else is encountered, the user has entered an invalid option. This calls the usage function, which displays a usage message and exits the program.

23   while getopts $optstr opt
24   do
25
26       case $opt in
27           r) replace="r";;
28           a) arch=$OPTARG;;
29           *) usage;;
30       esac
31   done

The getopts function also retains one other variable, $OPTIND, which contains the index of the next argument to be processed. When the shell script is first started, $OPTIND is set to 1. If -r is processed as the first argument, $OPTIND will contain 2. If -a is processed as the second argument (and the name of an archive directory as the third argument), $OPTIND will contain 4. On the next call to getopts, getopts returns false, and the loop at lines 23 through 31 ends.

At this point, $OPTIND still contains the value 4, which can be used as the index of the next argument -- the first argument not beginning with a hyphen (-). This should be the name of the file to archive. At line 33, the shift command is used to shift all arguments by $OPTIND - 1; this causes the argument that was at position 4 ($4) to be shifted left three positions, making it argument $1. At line 35 this value is picked up and stored in $filename.

33   shift `expr $OPTIND - 1`
34   
35   filename=$1

At this point, a good script would execute further error checking, such as making sure the file named in $filename and the archiving directory in $arch both exist. In the following example, results of the extracted values are displayed:

37   echo "Archiving" $filename " to " $arch "with" $replace
"replace option"
38   
39   # rest of the code goes here

Using getopts is an excellent way to create scripts that comply with the Unix command standard. It also makes it fairly easy to add additional features to your scripts. For example, let's assume you want to enhance your arch script to put a date and time stamp on an archive. Simply extend the $optstr variable to allow for a -d option, add a variable, and extend the case statement. Voilà! You've just added a -d option to the arch command. Of course, you have to add the code to handle $datestamp="Y", but the user interface is easily taken care of.

replace="w"
arch="/u/arch"
filename=""
datestamp="N"

optstr=":ra:d"

while getopts $optstr opt
do

    case $opt in
        r) replace="r";;
        a) arch=$OPTARG;;
        d) datestamp="Y";;
        *) usage;;
    esac
done

End of article.

Contact us for a free consultation.