ASCII chart with decimal values

Most of the nonprintable characters were and are used for communications protocols and have no real use for most applications programmers today. Even the others seem primitive in today's world of slick GUIs and laser printers. For example, the value 13 (CR) is a carriage return. When this value is sent to a printer, it causes the print head to return to column 1. A CR also is sometimes sent by the Return or Enter key on the keyboard, although Unix usually translates this as value 10, a line feed (LF). This latter control character is used to move a printer or terminal up one line. Value 7 (BEL), when sent by the computer to the terminal, usually causes a beep or rings an alarm. HT (horizontal tab, or just plain tab), value 9, is sent to a printer or a screen and causes the cursor or print head to advance to the next print column. SO (shift out) and SI (shift in), values 14 and 15, are also used in printer control. Many printers are set up with two built-in fonts. Sending an SI causes the printer to shift to the second font, while an SO causes it to shift back to the original typeface.

The values from 33 through 126 are printable characters. Value 32 (SP) is a space. Whether a space is actually a printable character is a debatable point, since a space does not usually put ink on the paper. Instead, it places a character containing no image. Some printers render this by simply advancing the print head one position.

The characters in the range below 32 are used extensively in telecommunications. For example, 2 and 3, STX (start of transmission) and ETX (end of transmission), are often used at the start and end of a block of transmitted information, respectively. 6 and 21, ACK and NAK, are often used by a receiving computer to signal an acknowledgement (ACK for well received) or a negative acknowledgement (NAK for not well received, please retransmit).

Control characters are also used inside text files to indicate the end of a line, and here is where our problem lies. Unix uses a single LF (line feed, ASCII value 10) character to designate this. DOS and Windows use a combination of CR (carriage return, value 13) and LF. Moving a text file back and forth between these two systems without translating the end-of-line marker causes some unusual results. For example, the MS-DOS Edit utility is smart enough to recognize a file that only has a line feed for an end-of-line marker and displays it correctly, but the Windows Notepad utility is not. Notepad displays an untranslatable control character as a thick black vertical bar that looks like a black box. In the following listings this black box is shown as a pair of square bracket (like this: []).

A Unix text file in MS-DOS Edit

These are the times
that try men's souls.
The Metropolitan Transit Authority,
better known as the MTA
etc.

Notepad cannot figure out where the lines in the same file end.

A Unix text file in Notepad These are the times[]that try men's souls.[]The Metropolitan Transit Authority,[]better known as the MTA[]etc.

In the reverse case, a Windows text file has too many control characters for vi. The extra carriage return shows up as a control-M (^M) in the vi display.

A Windows/DOS text file in vi

These are the times^M
that try men's souls.^M
The Metropolitan Transit Authority,^M
better known as the MTA^M
etc.^M

Many Unix/Windows transfer utilities include a switch that can be set to indicate that a text file is being transferred, and the resulting file has its end-of-line character(s) translated. Some utilities have text translation as the default, and you must set a switch to suppress the translation when you are transferring binary files.

Unfortunately, most serious movement of files in volume is done by combining and compacting the files using one of the versions of zip, tar, or what have you, and the resulting file must be transferred as a binary file. The individual files within such an archive do not have their end-of-line characters translated when they are combined and transferred as a binary.

Conversion scripts
The extra carriage return can be removed or inserted with two simple scripts. I use scripts here so that you can save and reuse them. The first one takes two command arguments, the Unix file name and the DOS/Windows file name. It adds a carriage return to a Unix text file and outputs it under the DOS text file name so that it can be transferred to DOS. To enter this with vi, when you get to the ^M, type control-V then control-M. The control-V causes the next character to be inserted as a literal control character. After you have saved it as lf2crlf, change its mode to allow execution (chmod a+x lf2crlf).

# lf2crlf
# adds an extra carriage return in a unix
# text file so that end of line matches
# the Windows/DOS convention

usage()
{
echo "usage: lf2crlf unix.txt dos.txt
exit
}

if [ $# != 2 ]
then
usage
fi

sed 's/$/^M/g' <$1 >$2

For text files coming from Windows or DOS to Unix, the second script strips the extra CR from the end of each line. There is an additional hook in MS-DOS files. Some DOS editors and utilities append a control-Z (value 26 or SUB in ASCII) to the end of a text file, which will display in vi as a ^Z. This script also removes that character. Note that the single quoted portion starts on one line and ends on the second. Use control-V, control-M to create the ^M and control-V, control-Z to create the ^Z.

# crlf2lf
# removes an extra carriage return in a dos/widows
# text file so that end of line matches
# the Unix convention.
# Also removes a control-Z at end of file

usage()
{
echo "usage: crlf2lf dos.txt unix.txt
exit
}

if [ $# != 2 ]
then
usage
fi

sed 's/^M//g
s/^Z//g' <$1 >$2

Just as a final note: some DEC systems used only a carriage return to mark an end of line. I once ported a C application from Unix to DOS and then from Unix to VAX. The end-of-line terminators had to be handled to move the source code from one system to another.

Contact us for a free consultation.