Learn Linux 07: The Way Shell Sees The World

Published Jan 8, 2022

Contents


Introduction

In this chapter, we are going to look at some of the "magic" that occurs on the command line when we press the enter key. While we will examine several interesting and complex features of the shell, we will do it with just one new command:

  • echo - Display a line of text

Expansion

Each time we type a command and press the enter key, bash performs several substitutions upon the text before it carries out our command. The process that makes this happen is called expansion. With expansion, we enter something, and it is expanded into something else before the shell acts upon it.

[user@linux ~]$ echo Hello World
Hello World
[user@linux ~]$ echo *
Desktop Documents Music Pictures Public Templates Videos

So why echo didn't print *? The * character means match any characters in a filename, but how the shell does that? The simple answer is that the shell expands the * into something else (in this instance, the names of the files in the current working directory) before the echo command is executed. When the enter key is pressed, the shell automatically expands any qualifying characters on the command line before the command is carried out, so the echo command never saw the *, only its expanded result. Knowing this, we can see that echo behaved as expected.

Pathname Expansion

The mechanism by which wildcards work is called pathname expansion.

[user@linux ~]$ ls
Desktop Documents Music Pictures Public Templates Videos
[user@linux ~]$ echo D*
Desktop Documents
[user@linux ~]$ echo *s
Documents Pictures Templates Videos
[user@linux ~]$ echo [[:upper:]]*
Desktop Documents Music Pictures Public Templates Videos
[user@linux ~]$ echo /usr/*/share
/usr/kerberos/share /usr/local/share

Tilde Expansion

The tilde character (~) has a special meaning. When used at the beginning of a word, it expands into the name of the home directory of the named user or, if no user is named, the home directory of the current user.

[user@linux ~]$ echo ~
/home/user
[user@linux ~]$ echo ~john
/home/john

Arithmetic Expansion

The shell allows arithmetic to be performed by expansion. This allows us to use the shell prompt as a calculator.

[user@linux ~]$ echo $((5 + 9))
14
[user@linux ~]$ echo $((5 - 9))
-4
[user@linux ~]$ echo $((5 * 9))
45
[user@linux ~]$ echo $((5 / 9))
0
[user@linux ~]$ echo $((5 % 9))
5
[user@linux ~]$ echo $((5 % 9))
1953125
[user@linux ~]$ echo $(((5**2) * 3))
75
[user@linux ~]$ echo $(($((5**2)) * 3))
75

Brace Expansion

Perhaps the strangest expansion is called brace expansion. With it, you can create multiple text strings from a pattern containing braces.

[user@linux ~]$ echo Front-{A,B,C}-Back
Front-A-Back Front-B-Back Front-C-Back
[user@linux ~]$ echo Item_{1..5}
Item_1 Item_2 Item_3 Item_4 Item_5
[user@linux ~]$ echo {Z..A}
Z Y X W V U T S R Q P O N M L K J I H G F E D C B A

In bash version 4.0 and newer, integers may also be zero-padded.

[user@linux ~]$ echo {01..15}
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15
[user@linux ~]$ echo {001..15}
001 002 003 004 005 006 007 008 009 010 011 012 013 014 015

Brace expansions may be nested too.

[user@linux ~]$ echo a{A{1,2},B{3,4}}b
aA1b aA2b aB3b aB4b

We can do pretty slick things with brace expansions.

[user@linux ~]$ mkdir Pictures
[user@linux ~]$ cd Pictures
[user@linux Pictures]$ mkdir {2007..2009}-{01..12}
[user@linux Pictures]$ ls
2007-01 2007-02 2007-03 2007-04 2007-05 2007-06 2007-07 2007-08 2007-09 2007-10 2007-11 2007-12
2008-01 2008-02 2008-03 2008-04 2008-05 2008-06 2008-07 2008-08 2008-09 2008-10 2008-11 2008-12
2009-01 2009-02 2009-03 2009-04 2009-05 2009-06 2009-07 2009-08 2009-09 2009-10 2009-11 2009-12

Parameter Expansion

Parameter expansion a feature that is more useful in shell scripts than directly on the command line. Many of its capabilities have to do with the system’s ability to store small chunks of data and to give each chunk a name. Many such chunks, more properly called variables, are available for your examination. For example, the variable named USER contains your username.

[user@linux ~]$ echo $USER
user

Command Substitution

Command substitution allows us to use the output of a command as an expansion.

[user@linux ~]$ echo $(ls)
Desktop Documents Music Pictures Public Templates Videos
[user@linux ~]$ ls -l $(which cp)
-rwxr-xr-x 1 root root 71516 2017-05-09 12:00 /bin/cp
[user@linux ~]$ file $(ls -d /usr/bin/* | grep zip)
/usr/bin/bunzip2: symbolic link to `bzip2'
/usr/bin/bzip2: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
/usr/bin/bzip2recover: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
/usr/bin/funzip: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped

Here is an alternate syntax for command substitution in older shell programs that is also supported in bash. It uses backquotes instead of the dollar sign and parentheses.

[user@linux ~]$ ls -l `which cp`
-rwxr-xr-x 1 root root 71516 2017-05-09 12:00 /bin/cp

Quoting

Now that we’ve seen how many ways the shell can perform expansions, it’s time to learn how we can control it. In the first example, word splitting by the shell removed extra whitespace from the echo command’s list of arguments. In the second example, parameter expansion substituted an empty string for the value of $1 because it was an undefined variable. The shell provides a mechanism called quoting to selectively suppress unwanted expansions.

[user@linux ~]$ echo Hello     World
Hello World
[user@linux ~]$ echo The total is $100.00
The total is 00.00

Double Quotes

The first type of quoting we will look at is double quotes. If we place text inside double quotes, all the special characters used by the shell lose their special meaning and are treated as ordinary characters. The exceptions are $ (dollar sign), \ (backslash), and ` (backtick).

[user@linux ~]$ ls -l two words.txt
ls: cannot access two: No such file or directory
ls: cannot access words.txt: No such file or directory
[user@linux ~]$ ls -l "two words.txt"
-rw-rw-r-- 1 user user 18 2017-05-09 12:00 two words.txt
[user@linux ~]$ echo Hello     World
Hello World
[user@linux ~]$ echo "Hello     World"
Hello     World

Single Quotes

If we need to suppress all expansions, we use single quotes. Here is a comparison of unquoted, double quotes, and single quotes.

[user@linux ~]$ echo text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER
text /home/user/ls-output.txt a b foo 4 user
[user@linux ~]$ echo "text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER"
text ~/*.txt {a,b} foo 4 user
[user@linux ~]$ echo 'text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER'
text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER

Escaping Characters

Sometimes we want to quote only a single character. To do this, we can precede a character with a backslash, which in this context is called the escape character. Often this is done inside double quotes to selectively prevent an expansion.

[user@linux ~]$ echo "The balance for user $USER is: \$9.00"
The balance for user user is: $9.00
[user@linux ~]$ mv bad\&filename good_filename

Backslash Escape Sequences

In addition to its role as the escape character, the backslash is used as part of a notation to represent certain special characters called control codes. The first 32 characters in the ASCII coding scheme are used to transmit commands to Teletype-like devices. Some of these codes are familiar (tab, backspace, line feed, and carriage return), while others are not (null, end-of-transmission, and acknowledge).

Here is a list of backslash escape sequences:

  • \a - Bell (an alert that causes the computer to beep)
  • \b - Backspace
  • \n - Newline; on Unix-like systems, this produces a line feed
  • \r - Carriage return
  • \t - Tab

The idea behind this representation using the backslash originated in the C programming language and has been adopted by many others, including the shell.

Adding the -e option to echo will enable interpretation of escape sequences. You can also place them inside $' '. Here, using the sleep command, a simple program that just waits for the specified number of seconds and then exits, we can create a primitive countdown timer.

[user@linux ~]$ sleep 3; echo -e "Time's up\a"
[user@linux ~]$ sleep 3; echo "Time's up" $'\a'

Summary

It could be argued that expansions and quoting are the most important subjects to learn about the shell. Without a proper understanding of expansion, the shell will always be a source of mystery and confusion, with much of its potential power wasted.