Linux in Science and Engineering
Working and programming in a Linux environment

Every tool should fulfill one purpose - and fulfill it as good as possible, adding only as much complexity as absolutely necessary.

In UNIX everything is a file: network, hard disks, USB sticks, keyboards & mice, … even the main memory

ahi:~ $ LANG=C ls -lh /dev/input/by-id/
total 0
lrwxrwxrwx 1 root root 9 Okt   2 23:40 usb-CHICONY_HP_Basic_USB_Keyboard-event-kbd -> ../event3
lrwxrwxrwx 1 root root 9 Okt   2 23:40 usb-Microsoft_Microsoft_Basic_Optical_Mouse_v2.0-event-mouse -> ../event2
lrwxrwxrwx 1 root root 9 Okt   2 23:40 usb-Microsoft_Microsoft_Basic_Optical_Mouse_v2.0-mouse -> ../mouse0

ahi:~ $ LANG=C ls -lh /dev/input/event3
crw-rw---- 1 root input 13, 67 Okt   2 23:40 /dev/input/event3

ahi:~ $ LANG=C ls -l /dev/sda*
brw-rw---- 1 root disk 8, 0 Okt  2 23:40 /dev/sda
brw-rw---- 1 root disk 8, 1 Okt  2 23:40 /dev/sda1
brw-rw---- 1 root disk 8, 2 Okt  2 23:40 /dev/sda2

ahi:~ $ LANG=C sudo /sbin/fdisk -l  /dev/sda
Disk /dev/sda: 465,78 GiB, 500107862016 bytes, 976773168 sectors
Disk model: ST3500630AS     
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00084420

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1  *         2048 960937983 960935936 458,2G 83 Linux
/dev/sda2       960937984 976771071  15833088   7,6G 82 Linux swap / Solaris

ahi:~ $ ls -lh /dev/mem
crw-r----- 1 root kmem 1, 1 Okt  2 23:40 /dev/mem

ahi:~ $ sudo od -t x1 -a /dev/mem | head -n 4
0000000  f3  ee  00  f0  f3  ee  00  f0  c3  e2  00  f0  f3  ee  00  f0
          s   n nul   p   s   n nul   p   C   b nul   p   s   n nul   p
0000020  f3  ee  00  f0  54  ff  00  f0  2f  20  00  f0  d7  1f  00  f0
          s   n nul   p   T del nul   p   /  sp nul   p   W  us nul   p

The history and technology of Unix is strongly tied to the programming language "C", which was largely developed to implement the operating system.

AT&T UNIX

from 1969 onwards, source available free in the beginning

Berkeley Software Distribution (BSD) Unix

from 1977 to 1995, a collection of free derivatives (FreeBSD, OpenBSD, NetBSD, …) exist till now

commercial UNIX

some workstation vendors adopted BSD for their hardware, some used evolved this system later to AT&T system V UNIX. The latest offspring is Apples macOS, whose kernel consists of parts of FreeBSD and the "Mach" microkernel.

Linux

developed starting 1991 in the spirit of Unix, technically not UNIX since no code has been inherited and no fees have been paid for the UNIX trademark. Linux is just the operating system kernel, most of the user visible code comes from the GNU project → so you will often find the term GNU/Linux.

Here documents are a special of input redirection with the operator << allowing multiline input right inside a shell script without using an additional file.

software <<EOF
  1st line of input
  2nd line of input
EOF

The delimiters for text can bee choosen freely, EOF is just a common string reminding the ASCII character EOF – End Of File. Trailing blanks in here documents are ommited.

A variant of here documents, allowing for shorter text snippets on a single line.

cat <<< "one line of input"

If the shell finds any of the latter characters in the words of a command line, it tries to match those to files in the working directory and replaces them with an alphabetically sorted list of those files.

*

matches any sequence of characters, even the null string

?

matches exactly one character

[…]

matches any of the enclosed characters. It is possible to use range expressions like [A-J] meaning every capital character between "A" and "J" in the character set in use (see locale(1)).

${parameter}

evaluates to the value of the paramter

${parameter:-text}

substitute text if parameter is not defined

${parameter:offset:length}

extract a substring

${parameter#text}

remove prefix matching text

${parameter%text}

remove sufffix matching text

file="/tmp/tmp.TEST.jpeg"
touch $file
mv $file ${file%.jpeg}.jpg
ls -l /tmp/tmp.*.jpg

Command substitution allows the output of a command to replace the command name.

$(command)
`command`

Both forms are roughly equivalent; the shell executes the command in a subshell and replaces command substitution with output from command.

Arithmetic expansion evaluates an arithmetic expression and substites the result.

$((expression))

Evaluation is done in fixed-width integers with no check for overflow. Operators and their precedence, associativity, and values are the same as in the C language

The if construct tests whether a evaluation of an expression returns zero (success!) and executes different compound commands:

if [ -d $HOME/Download/new ];
then
    DESTINATION=$HOME/Download/new/
else
    DESTINATION=/tmp/
fi

case $(hostname -s) in
    faepop?? )
        echo "Workstation for staff and diploma students."
        ;;
    faepcr?? )
        echo "Computer in students lab."
        ;;
    * )
        echo "I know nothing about $(hostname -f)"
        ;;
esac

• Each test line follows the globbing rules and ends with a right parenthesis ")".
• Each condition block ends with a double semicolon ;;.

for loops come in iterative and arithmetic flavour:

for file in *.jpeg
do
    mv $file ${file%.jpeg}.jpg
done

LIMIT=10
for ((a=1; a <= LIMIT ; a++))  # Double parentheses, and naked "LIMIT"
do
        echo -n "$a "
done 

while and until loops execute the compound statement list-2 as long as the statemant list-1 is evaluating to true or respectively false.

while list-1; do list-2; done
until list-1; do list-2; done

Copy the leading or last n lines of input to STDOUT. An especially usefull paramter to tail is -f which allows monitoring for additional data.

cut allows the extraction of columns from file in tabular form.

head -n 1  /etc/passwd
echo
head -n 1  /ect/passwd | cut -d : -f 6,7

paste writes corresponding lines from several files side by side

paste /etc/passwd /etc/group | head -n 3

The find command allows to search recursively for files meeting various constraints:

-name

search for a file with a name pattern - globbing rules apply

-iname

same as search, but case insensitive

-type

search for directories, regular files, links, …

-ctime n

files last changed n days ago

-empty

file or directory is empty

There is a simpler (and sometimes faster) alternative to find: fdfind.

xargs is an efficient way to collect output from find as parameters to another executable:

find /var/tmp/$USER -type d -empty -print0 \
    | xargs --null --no-run-if-empty \
            echo rmdir 

grep collects matching lines. There are several matching algorithms regular expressions (re) and simple text match to chose from - in our implementation there are three variants. Globbing rules do not apply :)

ls /afs/itp.tugraz.at/common/www/Lokales-TU/Arbeiten/Bakk/  \
    | grep "\-_200.-"

fzf & fzy allow for interactive and fuzzy search on the provided input. They are both new to me; waiting for your input!

Simply a better history?

alias H='history | fzf'

Or a helper to sort files?

function MV() {
    echo mv -v "$1" "$(find $HOME/Download/ -type d \
                 | fzf)"
}
# oder 
MV() { echo mv -v "$1" "$(find $HOME/Download/ -type d | fzf)" ;}

ed and sed are line oriented editors, quite usefull in scripting. ed (editor) is specialised on inline editing files while sed (stream editor) is suited for stream editing. They share a similar, but not identical command set.

ed -s mein_datei <<< 'H\ng/muster/d\n,w'
sed 's/regexp/replacement/g' inputFileName > outputFileName

Others know a lot more on ed!

module provides for the dynamic modification of your environment:

module avail
module load gcc
module list
gcc -v

Find the real path to an programm you intend to run - in case of weird problems this lets you verify which programm is really run.

which gcc
gcc ~/TEST/Compiler/hello.c -o test
test
which test
./test

Creates a temporary file or directory -

mktemp /tmp/$USER_XXXXXXX

install is usually called from within a shell script or a Makefile - more on make maybe in a later installment.

PREFIX=/temp/$USER
PROJECT=fritz

install --mode 755 --directory     $(PREFIX)/bin/
install --mode 755 --directory     $(PREFIX)/man/man1/
install --mode 755 $(PROJECT)      $(PREFIX)/bin/
install --mode 644 $(PROJECT).1.gz $(PREFIX)/man/man1/

Short digression to mode (see chmod(1) for more information):

755

means read and execute rights for everybody, write access for the owner of the file

644

means read rights for everybody, write access for the owner of the file, execute rights for nobody

Make is at its heart a solver of dependency graphs.

It reads a description of rules, targets and dependencies and tries to minimise the necessary steps to create requested targets.

In general, a rule looks like this:

targets : dependencies
        recipe
        ....

Targets are file names, separated by spaces - wildcards may be used. Usually there is only one target per rule.

A target is out of date if it does not exist or if it is older than any of the dependencies - also file names separated by spaces and optional wildcards.

The recipe describes the way to generate the targets from their dependencies. One major roadblock is indentation - a Tab character is necessary!

If you write a rule whose recipe will not create the target file, the recipe will be executed every time the target comes up for remaking.

clean:
        rm *.o *~

If later on a file "clean" is created it would be considered up to date - no dependencies - therefore the cleaning action will never be performed. Make clean an dependency of the special target .PHONY to run this recipe unconditionally.

.PHONY: clean
clean:
        rm *.o *~

Variables can represent lists of file names, options to pass to compilers, programs to run, …

To reference a variable use $(variable) or ${variable}:

src = hallo.c io.c numeric.c
program : $(src)
       gcc program $(src)

Some variables are set automatically inside rules:

$@

The file name of the rules target of the rule.

$<

The name of the first dependency.

$^

The names of all the prerequisites.

$?

The names of all the prerequisites newer than the target.

The last example could also be written as:

program : hallo.c io.c numeric.c
        gcc -o $@ $^

Some actions are performed quite regularly - like compiling C/C++ or FORTRAN code to object files and later linking them to executables.

For a lot of these recipes are already known to make; but you can create your own implicit rules:

%.pdf : %.tex
        rubber --pdf $<

The behaviour of builtin implicit rules is controlled by implicit variables; eg.

CXX

compiler for C++

CXXFLAGS

flags given to the C++ compiler

LDFLAGS

flags given to the linker, such as -L.

LDLIBS

libraries to link

-n

Print the commands that would be executed, but do not execute them.

-f <file>

Specify input should be read from <file>; if not present, the default of ./Makefile is used.

-C <directory>

Change to <directory> before doing anything else

• Easier notation for dependencies with auto-detection
• makes out-of-tree builds easy
• easier cross compilation
• optional interactive User Interface ccmake

• very few external dependencies: C compiler and native build tools
  • supports make, nmake, …
  • supports Apple XCode
  • supports MS Visual Studio

• module "CTest" provides a testing framework
• module "CPack" enables easy build of installation packages
  • for Microsoft Windows
  • for Apple macOS
  • and various UNIX/Linux package formats: Debian’s dpkg, Redhat’s RPM, …

For a minimal examples just create a file CMakeLists.txt next to your sources. Specify a project name, the executable and its sources:

project (Hallo)
add_executable (hallo hallo.c io.c numeric.c)

project (Hallo)
add_executable (hallo hallo.c io.c numeric.c)

find_package (BLAS)
if (BLAS_FOUND)
  target_link_libraries (hallo ${BLAS_LIBRARIES})
endif (BLAS_FOUND)

See /usr/share/cmake-X.XX/Modules/ for libraries known to CMake.