sh
vs. ash
vs. bash
vs. everything else, "REPL”, interactive vs. scripts, command history,
tab expansion, environment variables and "A path! A path!"
"If you hold a shell up to your ear, you can hear the OS." - me
To avoid getting all pedantic, I am just going to define a shell as an environment in which you can execute commands. People tend to think of a shell as a "command prompt," but you can run a shell without running a command prompt, but not vice versa - an interactive command prompt is an instance of a shell environment almost by definition.
Examples of shells:
-
CMD.EXE
- yes, Windows has a shell. \drios{Windows} \drshl{CMD.EXE} -
PowerShell.exe
- in fact, it has at least two! \drshl{Powershell}
In UNIX-land:
-
sh
- the "original" Bourne shell in UNIX, which spawned: \drshl{sh} -
csh
- C shell, historically it is the default shell on BSD systems (although there are arguments on why you should never use it). \drshl{csh} -
...and many more! - tons, really.
Most Linux distros use bash
, but the BSDs are all over the place. We're going to assume bash
for the rest of this tutorial. With few modifications, anything in the sh
hierarchy above can
usually run in the other members of the same tree.
Every shell has some "built-in" commands that are implemented as part of the shell and not as an
external command or program, and bash
has its share, as shown by running the
help
command in a bash
terminal:
\drdoc{help}
\drshl{bash}
\drcap{Built-in commands in \texttt{bash}}
~ $ help
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
These shell commands are defined internally. Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.
A star (*) next to a name means that the command is disabled.
job_spec [&] history [-c] [-d offset] [n] or hist>
(( expression )) if COMMANDS; then COMMANDS; [ elif C>
. filename [arguments] jobs [-lnprs] [jobspec ...] or jobs >
: kill [-s sigspec | -n signum | -sigs>
[ arg... ] let arg [arg ...]
[[ expression ]] local [option] name[=value] ...
alias [-p] [name[=value] ... ] logout [n]
bg [job_spec ...] mapfile [-n count] [-O origin] [-s c>
bind [-lpsvPSVX] [-m keymap] [-f file> popd [-n] [+N | -N]
break [n] printf [-v var] format [arguments]
builtin [shell-builtin [arg ...]] pushd [-n] [+N | -N | dir]
caller [expr] pwd [-LP]
case WORD in [PATTERN [| PATTERN]...)> read [-ers] [-a array] [-d delim] [->
cd [-L|[-P [-e]] [-@]] [dir] readarray [-n count] [-O origin] [-s>
...and so on...
Why does this matter? Because if you are in an environment and something as fundamental as echo
isn't working, you may not be working in a shell that is going to act like a "sh
" shell. In
general, sh
, ash
, bash
, dash
and ksh
all act similarly enough that you don't care, but
sometimes you may have to care. Knowing if you are on a csh
variant or even something more
esoteric can be key.
\drtxt{echo}
\drshl{csh}
Pay attention to the first line in script files, which will typically have a "shebang" line that looks like this: \index{*@\texttt{#"!} (shebang)}
\drcap{\texttt{bash} "shebang"}
#!/bin/bash
In this case we know the script is expecting to be executed by bash
, and in fact should throw an
error if /bin/bash
doesn't exist. For example, on the FreeBSD system I have access to, dash
is
not installed. So consider the following hello.sh
script:
\drbsd{FreeBSD}
\drcap{Script with dash
"shebang"}
#!/bin/dash
echo Hello, World!
When I try to run it on FreeBSD, I get:
\drcap{"Shebang" error}
% ./hello.sh
./hello.sh: Command not found.
This is confusing, because it seems to be saying that hello.sh
is not found! But in reality it is
complaining about missing dash
. If I change the script to point to bash
(which is installed on
that FreeBSD system), it works as expected:
\drcap{Hello, World!}
% ./hello.sh
Hello, World!
Note that on some systems #!/bin/sh
points to an alias of bash
, and on some it is a different
implementation of the original sh
command, such as ash
or dash
. Now you know what to search
for if you hit problems as simple as an expected "built-in" command not being found.
CMD.EXE
has a lineage that is a mish-mash of CP/M and UNIX excreted through three decades of
backwards compatibility to that devil's spawn we call DOS. It has gotten even muddier over the
years as Microsoft has added more commands, PowerShell, POSIX subsystems, etc.
\drios{CP/M}
\drios{DOS}
\drshl{CMD.EXE}
But even so, there are some similarities between CMD.EXE
and a Linux shell like bash
. In both
bash
and CMD.EXE
the set
command shows you all environment
variables that have been set. Here's bash
:
\drcmd{set}{set shell options}
\drshl{bash}
\drshl{CMD.EXE}
\index{environment variables!displaying!setcommand@\texttt{set} command}
\drcap{\texttt{set} command in \texttt{bash}}
~ $ set
BASH=/bin/bash
BASHOPTS=checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob:extquote
:force_fignore:histappend:interactive_comments:login_shell:progcomp:promptvars:s
ourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_COMPLETION_COMPAT_DIR=/etc/bash_completion.d
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="11" [3]="1" [4]="release" [5]="x86_64-pc-lin
ux-gnu")
BASH_VERSION='4.3.11(1)-release'
COLORTERM=gnome-terminal
COLUMNS=80
DIRSTACK=()
DISPLAY=:0
EUID=1003
GROUPS=()
HISTCONTROL=ignoreboth
HISTFILE=/home/myuser/.bash_history
...and so on...
And CMD.EXE
:
\drcap{\texttt{SET} command in \texttt{CMD.EXE}}
C:\Users\myuser>SET
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\myuser\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=JLEHMER650
ComSpec=C:\Windows\system32\cmd.exe
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\Users\myuser
LOCALAPPDATA=C:\Users\myuser\AppData\Local
LOGONSERVER=\\JLEHMER650
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\system32
\config\systemprofile\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program F
iles (x86)\nodejs\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program
Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL
Server\130\DTS\Binn\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\P
rogram Files (x86)\Microsoft SDKs\Azure\CLI\wbin;C:\Windows\System32\WindowsPowe
rShell\v1.0\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
...and so on...
Similarly, the echo
command can be used to show you the
contents of an environment variable like HOME
on bash
:
\drtxt{echo}
\drenv{HOME}{current user's home directory}
\index{environment variables!displaying!echocommand@\texttt{echo} command}
\drcap{\texttt{echo} the \texttt{HOME} environment variable in \texttt{bash}}
~ $ echo $HOME
/home/myuser
Versus the HOMEPATH
variable under CMD.EXE
:
\drcap{\texttt{echo} the \texttt{HOMEPATH} environment variable in \texttt{CMD.EXE}}
C:\> ECHO %HOMEPATH%
\Users\myuser
This example shows some valuable differences between shells. Even though both have the concept of
environment variables and displaying their contents using the "same" echo
command, note that:
-
The syntax for accessing an environment variable is
$variable
inbash
and%variable%
inCMD.EXE
. \index{environment variables!syntax} -
bash
is case-sensitive and soecho $HOME
works butecho $home
does not.CMD.EXE
is not case-sensitive, so eitherecho %homedrive%
orecho %HOMEDRIVE%
(orEcHo %hOmEdRiVe%
) would work.
One final note of caution. You can set up command aliases in bash
and other shells that allow you
to define a CMD.EXE
-style dir
command as a substitute for the ls
command in bash
, or
copy
for cp
, del
for rm
, and so on. I recommend you don't do this for at least two reasons:
\drfnd{cp}{copy}
-
It is difficult to get these right in terms of being able to map all the various parameters from the
bash
command to the appropriate parameters for aCMD.EXE
-style command. Most people don't go that far, which means you then end up with a "toy" substitute for theCMD.EXE
command, and have to fall back to the native commands anyway. -
It simply delays you actually learning about the "UNIX" environment. You end up relying on a crutch that then must be replicated on every system you touch. In my opinion it is better to just learn the native commands, because then you are instantly productive at any shell window.
It is much more common to set up environment variables to control run-time execution in Linux than
in Windows. In fact, it is quite common to assign a given environment variable for the single
execution of a program, to the point that bash
has built-in "one-line" support for it:
\index{environment variables!assigning}
\drcap{Assign \texttt{FOO} environment variable before executing script}
~ $ FOO=myval /home/myuser/myscript
This sets the environment variable FOO
to "myval" but only for the duration and scope of running
myscript
.
By convention, environment variables are named all uppercase, whereas all scripts and programs tend to be named all lowercase. Remember, almost without exception "UNIX" is case-sensitive and Windows is not.
You can assign multiple variables for a single command or script execution simply by separating them with spaces:
\drcap{Set multiple environment variables at once}
~ $ FOO=myval BAR=yourval BAZ=ourvals /home/myuser/myscript
Note that passing in values in this way does not safeguard sensitive information from other users
on the system who can see the values at least while the script is running using the ps -x
command.
In addition, the entire command will be written to your .bash_history
file, too. Theoretically
that should be safe, but if you are using this to pass in a password to a command, for example, and
your id gets compromised, your .bash_history
will be just as exposed as if you had the password
saved in a script file.
\drsys{ps}{list processes}
You can also set the value of environment variables to the output of another command by surrounding it with paired ` ("back ticks", or "grave accents"):
\drcap{Set environment variable to output from a command}
~ $ FILETYPE=`file --brief --mime-type header.tex`
~ $ echo $FILETYPE
text/plain
Sometimes you want to keep certain sensitive commands from being records in your .bash_history
file, since it is a simple text file and if you ever got hacked, the attacker could peruse it.
For example, some commands take userids and passwords as parameters. To keep a command like that
from being recorded in your command history, export the following, preferably in the .profile
or .bashrc
scripts in your home directory:
\drenv{HISTIGNORE}{commands to ignore in command history}
\drcap{Hiding commands from command history}
export HISTIGNORE="*smbclient*"
When writing scripts that can be run by any user, it may be helpful to know their user name at
run-time. There are at least two different ways to determine that. The first is via the
USER
environment variable:
\drenv{USER}{current user}
\drcap{\texttt{USER} environment variable}
~ $ echo $USER
myuser
The second is with a command with one of the best names, ever -
whoami
:
\drcmd{whoami}{existential question}
\drcap{\texttt{whoami} command}
~ $ whoami
myuser
Some environments set the USER
environment variable, some set a USERNAME
variable, and some
like Mint set both. I think it is better to use whoami
, which tends to be on almost all systems.
\drenv{USERNAME}{current user}
The concept of a "path" for finding executables is almost identical between "UNIX" and Windows, and
Windows lifted it from UNIX (or CP/M, which lifted it from UNIX). Look at the output of the PATH
environment variable under bash
:
\drenv{PATH}{execution search path}
\drios{CP/M}
\drcap{\texttt{PATH} environment variable in \texttt{bash}}
~ $ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
Echoing the PATH
environment variable under CMD.EXE
works, too:
\drenv{PATH}{execution search path}
\drcap{\texttt{PATH} environment variable in \texttt{CMD.EXE}}
C:\Users\myuser>ECHO %PATH%
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\system32\conf
ig\systemprofile\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files
(x86)\nodejs\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program File
s\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Serv
er\130\DTS\Binn\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Progra
m Files (x86)\Microsoft SDKs\Azure\CLI\wbin;C:\Windows\System32\WindowsPowerShel
l\v1.0\
Note the differences and similarities. Both the paths are evaluated left to right. Both use
separators between path components, a ;
for DOS and Windows, a :
for Linux. Both delimit their
directory names with slashes, with \
for DOS and Windows and /
for Linux. But Linux has no
concept of a "drive letter" like C:
in Windows, and instead everything is mounted in a single
namespace hierarchy starting at the root /
. We'll be talking more about directories, paths and
file systems in the next chapter.
\index{*@\texttt{/} (path separator)}
Just to muddy the waters further, notice how Cygwin under Windows shows the PATH
environment
variable with bash
syntax but a combination of both Cygwin and Windows directories, and Windows
drive letters like C:
mapped to /cygdrive/c
:
\drenv{PATH}{execution search path}
\drunx{Cygwin}
\drcap{\texttt{PATH} environment variable in Cygwin}
$ echo $PATH
/usr/local/bin:/usr/bin:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdri
ve/c/Windows/System32/Wbem:/cygdrive/c/Windows/system32/config/systemprofile/.dn
x/bin:/cygdrive/c/Program Files/Microsoft DNX/Dnvm:/cygdrive/c/Program Files (x8
6)/nodejs:/cygdrive/c/Program Files/Microsoft/Web Platform Installer:/cygdrive/c
/Program Files/Microsoft SQL Server/130/Tools/Binn:/cygdrive/c/Program Files (x8
6)/Microsoft SQL Server/130/DTS/Binn:/cygdrive/c/Program Files/Microsoft SQL Ser
ver/120/Tools/Binn:/cygdrive/c/Program Files (x86)/Microsoft SDKs/Azure/CLI/wbin
:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0
The actual "command prompt" is when you run a shell in an "interactive session" in a terminal
window. This might be from logging into the console of a Linux VM, or starting a terminal window in
a X window manager like GNOME or KDE, or ssh
'ing into an interactive session of a remote machine,
or even running a Cygwin command prompt under Windows.
\drnet{ssh}
\drunx{Cygwin}
Command prompts allow you to work in a so-called "REPL" environment (Read, Evaluate, Print, Loop). You can run a series of commands once, or keep refining a command or commands until you get them working the way you want, then transfer their sequence to a script file to capture it.
Real wizards at using the shell can often show off their magic with an incredible one-liner typed from memory with lots of obscure commands piped together and invoked with cryptic options.
I am not a real shell wizard. See chapter 9 for how you can fake it like I do.
Most modern interactive shells like bash
and CMD.EXE
allow for tab expansion and command
history, at least for the current session of the shell.
\drshl{CMD.EXE}
Tab expansion is "auto-complete" for the command prompt. Let's say you have some files in a directory:
\drcap{List some files}
~/Documents $ ls
Disabled User Accounts.csv elsewhere LOLcatz.jpg MyResume.md
Without tab expansion, typing out something like this is painful:
\drcap{Lots of typing and escape characters}
~/Documents $ mv Disabled\ User\ Accounts.csv elsewhere/.
But with tab expansion, we can simply type mv D^t
, where ^t
represents hitting the Tab
key,
and since there is only one file that starts with a "D", tab expansion will fill in the rest of the
file name for us:
\drcap{Tab expansion magic}
~/Documents $ mv Disabled\ User\ Accounts.csv
Then we can go about our business of finishing our command.
One place tab completion in bash
is different than CMD.EXE
is that in bash
if you hit Tab
and there are multiple candidates, it will expand as far as it can and then show you a list of
files that match up to that point and allow you to type in more characters and hit Tab
again to
complete it. Whereas in CMD.EXE
it will "cycle" between the multiple candidates, showing you each
one as the completion option in turn. Both are useful, but each is subtly different and can give
you fits when moving between one environment and another.
\drshl{CMD.EXE}
Pro Tip: Remember, UNIX was built by people on slow, klunky teletypes and terminals, and they hated to type! Tab expansion is your friend and you should use it as often as possible. It gives at least three benefits:
-
Saves you typing.
-
Helps eliminate misspellings in a long file or command name.
-
Acts as an error checker, because if the tab doesn't expand, chances are you are specifying something else (the beginning part of the file name) wrong.
The other thing to remember about the interactive shell is command history. Again, both CMD.EXE
and bash
give you command history, but CMD.EXE
only remembers it for the session, while bash
stores it in one of your hidden "profile" or "dot" files in your home directory called
.bash_history
, which you can display with ls -a
:
\drshl{CMD.EXE}
\index{files and directories!special!bashhistory@\texttt{.bash_history}}
\drcap{\texttt{ls} command showing hidden files}
~ $ ls -a
. .config .gconf .mozilla Templates
.. .dbus .gnome2 Music Videos
.bash_history Desktop .gnome2_private Pictures .xsession-errors
.bash_logout .dmrc .ICEauthority .profile
.cache Documents .linuxmint Public
.cinnamon Downloads .local .ssh
Inside, .bash_history
is just a text file, with the most recent commands at the bottom.
The bash
shell supports a rich interactive environment for searching for, editing and saving
command history. However, the biggest thing you need to remember to fake it is simply that the up
and down arrows work in the command prompt and bring back your recent commands so you can update
them and re-execute them.
Note: If you start multiple sessions under the same account, the saved history will be of the
last login to successfully write back out .bash_history
.