Simple shared logging

Using the cron daemon to create a shared log file

Summary
This month we show you how to write a simple logging script that logs messages into a single directory and then collect them into a master log. Once you've got that down, we'll show you how to use the cron daemon to automatically run this process on a regular basis. (1,200 words)

Last month, a reader wrote in with a question about logging from within a shell script and sharing files. He wanted to know how to create a shared script log file.

Sharing a file in shell script is fairly complex, but there is a simpler method of implementing shared logging. The technique involves using a resource that is automatically shared, a directory. The basic plan for sharing would be to write small log files into a logging directory, and then, every so often, collect up the log files and attach them to a larger log.

A simple version of this is illustrated in Listing 1. At line 6, logging directory is chosen by absolute path name. In other words it shouldn't be something like:

LOGDIR=$HOME/logs

because $HOME is relative to the user and process running the program.

The command at line 7 creates a log file name composed of the log directory, date, and time as a string of digits ( `date +%Y%m%d%H%M%S` ) and the process ID of the current process (.$$).

Lines 8 and 9 create open and closed version of the log file name by adding .x and .l extensions.

The actual logging takes place at line 10. The information that is logged is the date and everything that was passed on the command line to logit.sh.

The command:

logit.sh One two three

would result in a line of logged information that would look like:

Wed Nov 19 16:50:57 PST 1997 One two three

Listing 1: logit.sh -- A simple logging script
Line numbers are included for explanation.

 1 #! /bin/sh
 2 # logit.sh
 3 # log messages to a logging directory by using
 4 # a time and job stamp for the file name.
 5 
 6 LOGDIR=/joblogs ; export LOGDIR
 7 LOGFILE=$LOGDIR/`date +%Y%m%d%H%M%S`.$$ ; export LOGFILE
 8 OPENLOG=$LOGFILE.x ; export OPENLOG
 9 CLOSEDLOG=$LOGFILE.l ; export CLOSEDLOG
10 echo `date` "$*" >> $OPENLOG
11 mv $OPENLOG $CLOSEDLOG

One question that immediately pops up is why go through the process of renaming the log files? It is possible to leave all the little log files sitting in the logging directory, but it is much tidier to pick them all up and attach them into one log file. A single log file is easier to read and search.

The problem with grabbing the little log files and collating them into one larger file is that you don't want to try to collate a file that is currently receiving logging information. So a file with a .x extension is a file that is open and should be left alone. A file with a .l extension is finished with, and can be collected into the larger logging file.

The process of collecting the log files into a single file is simple enough on the surface, as in Listing 2. All of the .l files in the log directory are appended to the master log, and the individual files are deleted.

Listing 2: Collecting the log files in a master log

 1 #! /bin/sh
 2 # bldmaster.sh
 3 # collects up all .l files in the logging
 4 # directory and appends them to the master log.
 5 
 6 LOGDIR=/joblogs ; export LOGDIR
 7 MASTERLOG=master.log ; export MASTERLOG
 8 MASTERPATH=$LOGDIR/$MASTERLOG ; export $MASTERPATH
 9 
10 for name in $LOGDIR/*.l
11 do
12     cat $name >> $MASTERPATH
13     rm -f $name
14 done

The question that arises from listing 2 is, who does all this collecting? If any or all processes could do it, we would just have the same sharing problem all over again as multiple tasks tried to add data to the master log.

So we give the problem to the cron daemon. The cron daemon runs in the background on Unix systems. It is usually started when the system is first booted up. Its job is to launch tasks that have to be executed at regular intervals. The types of tasks that you will find a cron daemon performing include nightly backups, weekly cleanups of temporary files, and logging of system messages for the system administrator. Cron is designed to allow tasks to run as often as once a minute or as little as once a year.

The cron daemon reads one or more files containing entries that are made up of five columns specifying the times when a process should run, one column naming the program or script with its command line arguments, and an optional comment field.

A sample cron table for nightly backups might look like Listing 3. The weekday varies from 0 for Sunday through 6 for Saturday. The backup time is 11:20 p.m., except for Sunday and Saturday when it runs at 7:20 p.m.. The day and month fields are marked with an asterisk meaning any legal value. Or in this case, any month or any day.

Listing 3: A sample cron file

#Min Hr Day Month WeekDay Job
#-------------------------------------------------------------
20  19   *   *     0      /prgms/backup/sunbu.sh #Sunday Backup
20  23   *   *     1      /prgms/backup/monbu.sh #Monday
20  23   *   *     2      /prgms/backup/tuebu.sh #Tuesday
20  23   *   *     3      /prgms/backup/wedbu.sh #Wednesday
20  23   *   *     4      /prgms/backup/thubu.sh #Thursday
20  23   *   *     5      /prgms/backup/fribu.sh #Friday
20  23   *   *     6      /prgms/backup/satbu.sh #Saturday

You can specify multiple values for a column by separating the values with commas. In Listing 4 the bldmaster.sh process has been added to run every 15 minutes. Note that the hour, day, month, and weekday are filled in with asterisks, so the process runs every 15 minutes of every day of the week.

Listing 4: A sample cron file with the master logging process.

#Min Hr Day Month WeekDay Job
#-------------------------------------------------------------
20  19   *   *     0      /prgms/backup/sunbu.sh #Sunday Backup
20  23   *   *     1      /prgms/backup/monbu.sh #Monday
20  23   *   *     2      /prgms/backup/tuebu.sh #Tuesday
20  23   *   *     3      /prgms/backup/wedbu.sh #Wednesday
20  23   *   *     4      /prgms/backup/thubu.sh #Thursday
20  23   *   *     5      /prgms/backup/fribu.sh #Friday
20  23   *   *     6      /prgms/backup/satbu.sh #Saturday
0,15,30,45 * * *   *      /prgms/bldmaster.sh #Collate logs

The cron process keeps a list of cron jobs for each user legally allowed to create cron jobs. If you are root, you will be able to create cron jobs by editing the root cron table. Proceed with caution as you type:

crontab -e

Your root cron table will be opened by the vi editor. If you are not root, but you do have permission to create cron jobs, typing crontab -e will open your personal cron table for editing. If you are not familiar with cron, this will probably be a new file. If you are just learning about or experimenting with cron, be sure to back up your cron tables. These are probably in /var/spool/cron/crontabs, but see your manual for the location of cron tables.

If you do not have cron rights, you must create your bldmaster.sh process, and then get your system administrator to install it as a cron job.

Contact us for a free consultation.