3. Advanced Shell Features¶
Last lesson we focused on very basic principles and features of the UNIX shell.
This lesson we want to go one step further and make ourselves more productive.
There are lots of more features, little tricks and efficiency hacks, we can use on the shell.
Most presented features are still POSIX-compliant, i.e. work in every UNIX shell. But some may only work on certain more advanced shells such as Bash or ZSH.
I Know What You Typed Last Night: Shell History¶
Most modern shells save your commands in a history file. For Bash this is ~/.bash_history, for ZSH it is ~/.zsh_history.
Every line in that file corresponds to one command you ran.
To look at the last 5 entries from your history, run
history 5
2066 head -n-1 examplefile 2067 echo $? 2068 mkdir --help 2069 echo $? 2070 history 5
The main purpose of the history is convenience. In the shell use the Up and Down arrow keys to recall previously executed commands, edit them as needed and hit enter to run them again.
History shortcut commands¶
Instead of the arrow keys you can also use shortcut commands:
!N– run the Nth command from the history again!-N– run the Nth-last command again!!– run the last command again (same as!-1)
For example:
~$ echo "Hello World"
Hello World
~$ !!
echo "Hello World"
Hello WorldHow is that useful?
Imagine, you mistyped a command and want to corrrect it. Instead of typing everything again, use the Up arrow, correct the command and run it again.
Similarly, you maybe forgot sudo in front of a command that requires root privileges:
~$ mkdir -v /etc/somefolder
mkdir: cannot create directory ‘/etc/somefolder’: Permission denied
~$ sudo !!
sudo mkdir /etc/somefolder
[sudo] password for cmdline:
mkdir: created directory /etc/somefolderIncidents¶
Be aware, though, that on our seminar system, you won't be able to run commands with sudo. Doing so will result in a message like this:
$ sudo somecommand
cmdline is not in the sudoers file. This incident will be reported.
Fun with Filenames: Globbing¶
As we saw in the last lesson, many commands can take more than one file as arguments.
But imagine we have a directory with 100 files and want to copy all of them to a new folder. Do you want to write all their names manually into the command line? Surely not.
This is where globbing (wildcarding) comes into play.
Globbing is a very powerful, yet sometimes dangerous, feature of every modern UNIX shell.
Four basic operators exist:
? one arbitrary character
* 0 or more arbitrary characters
[abc], [a-z] one character from the given set
{abc,def} one string from the given setThese work in basically every shell. Some shells specify more advanced globbing patterns. For example, Bash also supports [^abc] for negative patterns and ZSH supports recursive globbing via **.
Example 1: Move all files from one directory to another
ls -l exampledirectory1
total 8 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:04 file1 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:04 file2
ls -l exampledirectory2
total 0
mv ./exampledirectory1/* exampledirectory2
ls -l exampledirectory2
total 8 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:04 file1 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:04 file2
Example 2: Copy file1 and file2
cp ./exampledirectory2/{file1,file2} exampledirectory1
ls -l exampledirectory1
total 8 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:28 file1 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:28 file2
Example 3: List all files from both folders that end with the number 2 or 3
ls -l ./exampledirectory?/*[23]
-rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:28 ./exampledirectory1/file2 -rw-r--r-- 1 cmdline cmdline 6 Oct 26 18:04 ./exampledirectory2/file2
Swimming with the Stream: Redirection and Pipes¶
There are three standard streams which are used for all input and output data:
- Standard Input (
STDIN) - Standard Output (
STDOUT) - Standard Error Output (
STDERR)
If a program asks you interactively to enter some text, it goes through STDIN.
If a program prints some text to the console, it usually goes through STDOUT.
If that output is an error, the program may also send it through STDERR.
Every of the three streams can be accessed independently by its handle number (0, 1 and 2).
For redirecting these three streams to (or in case of STDIN from) files, we have the three operators
0<1>2>
Because people are lazy, 1> can be abbreviated with > and 0< with <.
To make this clearer, let's write Hello World to a file:
echo "Hello World" > examplefile2
cat examplefile2
Hello World
We redirected STDOUT of the echo command (which would have gone to the screen otherwise) to a file.
What happens when we redirect STDERR instead?
echo "Hello World" 2> examplefile2
Hello World
cat examplefile2
Now Hello World goes to the screen (because we didn't touch STDOUT) and the file is empty because there was no error output.
Now it also makes sense why we have two standard output streams. This way we can handle “normal” command output and any error output separately.
Appending lines in files¶
You probably noticed that > and 2> truncate the file if it exists already. If you want to append lines, use >> instead:
echo "Hello World" > examplefile2
echo "Hello Line2" >> examplefile2
cat examplefile2
Hello World Hello Line2
Where nothing comes ever back¶
In context of output redirection, let me introduce a very helpful device file: /dev/null.
/dev/null is basically the “black hole” of your Linux system. Everything that goes past its event horizon never comes back.
With this unique property, /dev/null is really helpful, when you are only interested in, e.g., STDOUT and want to disregard any (potential) error messages:
cat exampledirectory1/file1 2> /dev/null
File1
In the event that this file doesn't exist, the command would silently fail instead of printing an error message.
In the same way you can also redirect STDOUT to /dev/null or both:
cat exampledirectory1/file2 > /dev/null 2> /dev/null
Redirecting streams to each other¶
The last example of writing both STDOUT and STDERR to a file is very verbose. We can make it shorter:
cat exampledirectory1/file2 > /dev/null 2>&1
What did we do here?
We told STDOUT to go to hell /dev/null.
After that we told STDERR (2>) to go to STDOUT (&1).
The ugly duckling: STDIN redirection¶
There is nothing special to redirection of STDIN compared to STDOUT or STDERR. As shown before, we use < for writing a file to STDIN of the given command.
The only downside is: we hardly ever need it.
Almost all commands, that accept input from STDIN also let you specify files on the command line.
The following usage of cat is possible, but also pointless. (WHY?)
cat < exampledirectory1/file2
File2
Easier would be
cat exampledirectory1/file2
File2
Input from /dev/null is also possible, but only rarely useful when you need “nothing” as input for some reason:
cat < /dev/null
Where Things Get Interesting: Piping¶
Redirection to files is nifty, but not very exciting when you know about piping.
With piping we can redirect STDOUT of one command directly to STDIN of another command. You can pipe as many commands together as you like.
The syntax is:
command1 | command2 | command3 | ...Want to get the second word of "Hello World"? Easy. (HOW?)
echo "Hello World" | cut -d" " -f2
World
Using echo to pipe arbitrary strings into commands is a pretty common shell pattern. But you can do way more advanced things.
Using only commands already presented in this course, we can cut out the third word of the second line of a file. (HOW?)
cat examplefile
This Is Line1 This Is Line2 This Is Line3 This Is Line4
Cut out the wanted part:
head -n2 examplefile | tail -n1 | cut -d" " -f3
Line2
Alternatively:
cut -d$'\n' -f2 examplefile | cut -d" " -f3
(works only with GNU Coreutils 8.23 and above)
Piping STDERR¶
Pipes only work in STDOUT. If we want to pipe STDERR, we need to redirect it to STDOUT first.
As a stupid example, let's count the words of the produced error message:
cat nonexistentfile 2>&1 | wc -m
48
Background Jobs¶
By default, any command that you run, will block the input prompt until it has finished.
By sending a program to the background, you can continue to use your shell while the process is doing its work.
For demonstrating this, we need a process that runs forever (or at least quite some time). We'll use:
bash -c "while true; do sleep 1; done"Notice how your shell seems to hang because we just started an endless loop.
To interrupt this process, hit Ctrl+Z (^Z). This will send a STOP signal (SIGSTP) to the process. The STOP signal causes the process to suspend and gives control back to the shell.
If you now run the jobs command, you will see a list of all suspended processes in the current shell (i.e. “Jobs”):
$ jobs
[1] + suspended bash -c "while true; do sleep 1; done"To resume the suspended process, we have the commands fg and bg to continue execution in either foreground or background mode.
Resuming the process in the foreground is like you never suspended it. Interesting is the bg command:
$ bg %1
[1] + continued bash -c "while true; do sleep 1; done"Nothing seems to change, but in fact, the process is now running again. This time, however, in the background.
We can verify this by running jobs:
[1] + running bash -c "while true; do sleep 1; done"You can bring the process back to the foreground at any time by typing fg:
$ fg %1
[1] + running bash -c "while true; do sleep 1; done"Terminating a hanging process¶
Since your process is running forever, we want to manually terminate it at some point. To do this, hit Ctrl+C (^C). This will send an INTERRUPTION signal (SIGINT) which kindly asks the process to exit.
If the process is running in the background, you can also use kill instead of Ctrl+C:
$ kill -INT %1
[1] + interrupt bash -c "while true; do sleep 1; done"Forcefully closing processes¶
If you use kill without the -INT flag, it will send a TERMINATION signal (SIGTERM) instead of an INTERRUPTION signal. SIGTERM is a direct request to the program to terminate itself. In most cases, this has the same behavior as SIGINT.
If the program is really frozen and doesn't react anymore, you can also ask the kernel to terminate that process for you by sending a KILL signal using `kill -KILL':
$ kill -KILL %1
[1] + killed bash -c "while true; do sleep 1; done"This will end the process instantly, no matter what.
Signal numbers¶
Every signal also has a number that you can use instead. For instance, the number for SIGKILL is 9, therefore you can also run:
$ kill -9 %1
[1] + killed bash -c "while true; do sleep 1; done"To get a list of all available signals and their numbers use
kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8 43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1 64) SIGRTMAX
Using kill with process IDs¶
kill can, of course, not only be used with job numbers, but also directly with process numbers.
Every process on a Linux system has a unique number which you can find out by running ps a or by using top (or htop).
If you're using process numbers, omit the leading % symbol in front of the number.
Starting processes directly in the background¶
If you want to start a process in the background, but don't want to go use Ctrl+Z and then bg %1, you can add an ampersand & to the end of the command line:
$ bash -c "while true; do sleep 1; done" &
[1] 13004This will start the process in the background and print both the job and the process number.
Configuration at Runtime: Environment Variables¶
Probably one of the most import feature of any shell are environment variables.
Environment variables can be used to strore arbitrary strings or numbers and configure certain aspects of your working environment (therefore environment variables).
Your shell usually comes with a lot of pre-defined environment variables. For a list of all currently defined variables, write env.
Defining custom variables is easy:
VARNAME="some string"
Note, that there must not be any spaces around the assignment operator =.
Since shell variables are also called parameters, the replacement of a variable with its value is called parameter expansion. Expanding (i.e. using) variables is equally simple as defining them:
echo $VARNAME
some string
Alternative notation:
echo ${VARNAME}
some string
Parameter expansion in strings¶
Shell variables can also be used within double-quoted strings:
echo "This is ${VARNAME}"
This is some string
If you want to prevent expansion within strings, either escape the variable symbol or use single-quoted strings:
echo "This is \${VARNAME}"
This is ${VARNAME}
echo 'This is ${VARNAME}'
This is ${VARNAME}
Pitfalls and when quotes are really needed¶
Parameter expansion in shell environment works a little different than in other programming languages.
The contents of a variable can easily get unexpected syntactic meaning.
The following code snippet will most likely do something else than you intended:
touch "My Example File.txt"
alias cp="echo cp: target ‘File.txt’ is not a directory; true"
mkdir somedirectory
FILENAME="My Example File.txt"
cp $FILENAME somedirectory/$FILENAME
cp: target ‘File.txt’ is not a directory
(WHY?)
unalias cp
The cp expression gets expanded to:
cp My Example File.txt somedirectory/My Example File.txtHowever, what you probably wanted was:
cp "My Example File.txt" "somedirectory/My Example File.txt"To achieve that, you have to put the variables in quotes:
cp "$FILENAME" "somedirectory/$FILENAME"
rm "My Example File.txt"
rm -R somedirectory
Variable scopes¶
Shell variables have three different scopes in which they're visible:
- local
- global
- exported
By default, any defined variable is global. That means, they're visible for every sub-routine in the current shell instance.
However, separate sub-processes will not inherit global variables:
unset FOO
FOO="bar"
bash -c 'echo "FOO says ${FOO}"'
FOO says
To make the value of $FOO visible to sub-processes, we have to export the variable:
export FOO
bash -c 'echo "FOO says ${FOO}"'
FOO says bar
Exported variables are what we actually call environment variables.
Variables can also be declared locally which is mainly useful for scripting when you use sub-routines.
Local variables are declared with the keyword local:
some_sub_routine() {
local local_variable="local"
GLOBAL_VARIABLE="global"
echo "From sub-routine: ${GLOBAL_VARIABLE}, ${local_variable}"
}
some_sub_routine
From sub-routine: global, local
echo "From main program: ${GLOBAL_VARIABLE}, ${local_variable}"
From main program: global,
Pre-defined environment variables¶
There is quite a number of pre-defined environment variables. The three most important ones are:
$HOME– contains the absolute path to your home directory$LANG– your current system locale$PATH– contains a colon-separated (:) list of directories to search for commands
$PATH is the reason why you are able to type vim instead of the full path /usr/bin/vim. Because /usr/bin is in your $PATH, the shell searches that directory for an executable called vim.
The contents of a typical $PATH variable may look like:
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Command substitution¶
Similar to parameter substitution, also commands can be evaluated and substituted.
Any command that appears between backticks `...` or within $(...) is evaluated and the result is passed back as a string to the command line:
echo "My current username is $(id -un)"
My current username is cmdline
Basic Scripting¶
If we can use variables, can we also use other elements of a regular programming (scripting) language?
Of course we can. In fact, any sequence of shell commands can also be written as a script file and then executed as a program.
In order to create a working shell script the script file needs to
- be executable (
+xpermissions) - start with a so-called hashbang
Hashbang¶
A hashbang (also called shebang) is a “magic” character sequence at the beginning of an executable text file that specifies the interpreter that is used to execute the following script.
A hashbang consists of the sequence #! followed by a path.
For example:
#!/bin/bashor
#!/usr/bin/python“Dynamic” hashbangs¶
By using #!/bin/bash you do nothing wrong in most cases, but you might want to not hard-code the location of Bash (especially when you're coding for low-end machines such as embedded devices).
In that case you can use /usr/bin/env to find the path out for you:
#!/usr/bin/env bash/usr/bin/env will look in in your $PATH variable for a suitable path and try to execute Bash from there.
Even though it's not needed in most cases, it's best practice to use /usr/bin/env.
Control structures¶
The hashbang can be followed by any code that is executable by the given interpreter. In this case, we will continue to use Bash.
Since we want to to “real” scripting, we need more than the ability to write a static sequence of commands. That's where control structures come into play.
Like most other programming languages, Bash also supports if (and elif/else), for and while. The syntax may be a little strange at first, though.
if, elif, else¶
A conditional statement is introduced with if which is then followed by any executable commands that returns a true (zero*) or false (non-zero) exit code. The block is terminated by fi.
if true; then
echo "Hello World"
fi
Hello World
*) no, that's not a mistake
A condition may be notated within [ and ] (mind the spaces between brackets and statement):
if [ "a" != "b" ]; then
echo "Alphabet still unchanged"
fi
Alphabet still unchanged
More branches can be introduced with elif and else:
if [ "a" == "b" ]; then
echo "Huh?"
elif [ "b" == "c" ]; then
echo "Woot?"
else
echo "Nothing of that"
fi
Nothing of that
Special comparison operators exist for numeric values:
-eq– is equal to-ne– is not equal to-gt– is greater than-ge– is greater than or equal to-lt– is less than-le– is less than or equal to
Example:
if [ 1 -lt 2 ]; then
echo "Math still works"
fi
Math still works
There is also a set of (unary) file testing operators. The most common are:
-e– file exists-f– file exists and is a regular file (not a directory or device file)-s– file is not zero size-d– file is a directory-h– file is a symbolic link-r– file has read permissions-w– file has write permissions-x– file has execute permissions
Example:
if [ -f examplefile ]; then
cat examplefile
fi
This Is Line1 This Is Line2 This Is Line3 This Is Line4
Statements can also be negated with !. Multiple conditions can be chained with && (logical AND) or || (logical OR):
if [ ! -d examplefile ] && [ -f examplefile ]; then
cat examplefile
fi
This Is Line1 This Is Line2 This Is Line3 This Is Line4
for¶
The for statement can be used to iterate over a set of space-separated values, such as a file listing. The loop body is notated between do and done:
for i in ./*; do
echo "File name: $i"
done
File name: ./exampledirectory1 File name: ./exampledirectory2 File name: ./examplefile File name: ./examplefile2 File name: ./seminar
while¶
The while statement executes a code block as long as its condition evaluates to true. The loop syntax is the same as for for, the syntax of the conditional expression the same as for if:
i=0
while [ $i -lt 5 ]; do
echo $i
i=$(($i + 1))
done
0 1 2 3 4
while may also be used together with piping. It's really simple to read a file line by line:
cat examplefile | while read line; do
echo "Current line: $line"
done
Current line: This Is Line1 Current line: This Is Line2 Current line: This Is Line3 Current line: This Is Line4
The read command will read one line of input and write it into the variable $line. If an EOF (end of file) appears, the command will exit with a non-zero exit code (which evaluates to false).
You can use read also without piping to ask the user of a script to enter a line of text into your program.
Special variables¶
When you use Bash for scripting, you may also find the following special variables useful:
$0– first positional parameter, i.e. the script name (similar toargv[0]in C)$1..$9– more positional parameters, i.e. script parameters${10}..${N}– two-digit positional parameters$@– All positional parameters, except$0$#– The number of given arguments, not counting$0$?– The return code of the last command you executed
End of Lesson 2¶
That marks the end of lesson 2.
We covered the most basic principles and techniques. Next time we'll focus on a list of useful commands that make your life a lot easier and prepare your way to ultimate shell mastery. :-)
Thanks for listening.