Raspberry Pi: Measure, Record, Explore

Monday, 11 April 2016

Scripting in Linux

The following post is a section of the book 'Just Enough Linux'.  The entire book can be downloaded in pdf format for free from Leanpub or you can read it online here.
Since this post is a snapshot in time. I recommend that you download a copy of the book which is updated frequently to improve and expand the content.
---------------------------------------

A shell script is a file containing a series of commands. The process of scripting is the writing of those commands in the file. The shell can read this file and act on the commands as if they were typed at the keyboard.
Because there are many commands available to use we can realise the real power of computing and the reduction of repetitive tasks by automating processes.
Think of shell scripts as program commands which are chained together and which the system can execute as a scripted event. They make use of functions such as command substitution where we can invoke a command, like date, and use it’s output as part of a file-naming scheme. Scripts are programs in their own right which use programming functions such as loops, variables and if/then/else statements. The beauty of scripting is that we don’t have to learn another programming language to script effectively, we simply use the commands we use at the command line.
To successfully write a script to do something we have to do three things;
  1. Write a script (a file with a ‘.sh’ extension) that uses common commands.
  2. Set the permissions on the file so that it can be executed
  3. Place the file somewhere that the shell can locate it when it’s invoked

Writing our Script

The script itself is just a text file. The only thing fancy about it is that it contains commands and we’re going to make it executable.
We can start with a simple example that demonstrates echoing a simple message to the screen.
Starting in our home directory (/home/pi) we can use the nano editor to create our script by running the following command;
This will open the nano editor and we can type in something like the following;
#!/bin/bash
# Hello World Script

echo "Hello World!"
The first line of the script is important because it provides information to the shell about what program should be used to interpret and run the script. The line starts with a ‘shebang’ (#!) and then the path to the program (in this case /bin/bash).
The second line is a comment that we would place in a script to tell ourselves or others that open the file what is going on. Anything that appears after a # mark (except for on the first line with the ‘shebang’) will be ignored by the script and is entered to make notes for the reader.
The last line is the command we will run. In this case it only one command, but it could also be any number. The command in this case is a very simple echo command to print the words ‘Hello World!’ to the screen.
Once finished we can close and save the file (CTRL-x to close and say ‘y’ to save);
That’s our script written.

Make the script executable

If we list the properties of our file sayhello.sh with ls as follows;
… we will see the details of the file something like this;
-rw-r--r-- 1 pi pi 54 Feb 17 17:18 sayhello.sh
Linux permissions specify what the owning user can do, what a member of the owning group can do and what other users can do with the file. For any given user, we need three bits to specify access permissions: the first to denote read (r) access, the second to denote (w) access and the third to denote execute (x) access.
We also have three levels of ownership: ‘user’, ‘group’ and ‘others’ so we need a triplet (three sets of three) for each, resulting in nine bits.
The following diagram shows how this grouping of permissions can be represented on a Linux system where the user, group and others had full read, write and execute permissions;
Linux permissions as rwx
The permissions of our sayhello.sh file tell us that the user can read and write the file, members of the group and anyone else can read the file. What we need to do is to make it executable using the chmod command
If we check again with ls -l sayhello.sh we should see something similar to this;
-rwxrwxr-x 1 pi pi 54 Feb 17 17:18 sayhello.sh
Here everyone has execute permissions and the ‘pi’ user and the ‘pi’ group have read and write permissions.

Place the script somewhere that the shell can find it

Currently we’re working in the ‘pi’ home directory where we’ve saved our file. This should be the easiest example of making the script accessible and as such we should be able to simply run the command directly from the command line as follows;
The output from the command should look something like this;
Hello World!
The ./ notation in front of the script designates the path to the file being our current directory. We could have also provided the full path as follows;
(Since /home/pi/ is the directory that the file is currently in.)
So that works, but in the perfect world we wouldn’t need to designate a path to the script in order to execute it. We can make that happen by placing the script in one of the directories in the users ‘PATH’.
PATH is an environmental variable that tells the shell which directories to search for executable files. It should not be confused with the term ‘path’ which refers to a file’s or directory’s address in a files system
A user’s PATH consists of a series of colon-separated absolute paths. When we type in a command at the command line that is not distributed with the operating system or which does not include its absolute path and then the shell searches through the directories in PATH, which constitute the our search path, until it finds an executable file with that name.
We can check what the paths available are by echo-ing the PATH variable to the screen like so;
This will then report PATH to the screen something like the following
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
We can therefore add our script to any of those directories and it will work like a normal command. However, this will make it available to all users on the system and if we want to only make it available to ourselves as the current user we will need to add a new path to PATH.
The good news is that this is simple to do and in Debian Jessie all we need to do is to create a directory in our home directory (if we are logged in as the ‘pi’ user, that would be /home/pi) called bin. We can place any script that we want to be available to the ‘pi’ user in there and they will have access to it. The reason they will have access is that while it doesn’t appear in PATH at the moment, when we add the bin directory to our home directory, there is a script called .profile which gets checked whenever we start our bash shell. If .profile sees that we have a bin directory in our home directory it adds it to our PATH.
So if we add the directory using mkdir as follows;
… we will be set up with a new path in PATH the next time we start our shell.
Let’s not forget that we should move our file into our new directory with mv as follows;
If we open a new terminal and log on as our user (it’s ‘pi’ in the example) we can how go anywhere (cd) in the directory structure and simply run the command;
… and get the response;
Hello World!
That’s the completion of our simple first example. More advanced work with loops, variables and if/then/else statements will have to be an exercise for the reader. Hopefully that’s ‘Just Enough’ to get you started.


The post above (and heaps of other stuff) is in the book 'Just Enough Linux' that can be downloaded for free (or donate if you really want to :-)).

Saturday, 9 April 2016

File Editing in Linux

The following post is a section of the book 'Just Enough Linux'.  The entire book can be downloaded in pdf format for free from Leanpub or you can read it online here.
Since this post is a snapshot in time. I recommend that you download a copy of the book which is updated frequently to improve and expand the content.
---------------------------------------

Working in Linux is an exercise in understanding the concepts that Linux uses as its foundations such as ‘Everything is a file’ and the use of wildcardspipes and the directory structure.
While working at the command line there will very quickly come the realisation that there is a need to know how to edit a file. Linux being what it is, there are many ways that files can be edited.
An outstanding illustration of this is via the excellent cartoon work of the xkcd comic strip (Buy his stuff, it’s awesome!).
Real Programmers (courtesy xkcd)
For a taste of the possible options available Wikipedia has got our back. Inevitably where there is choice there are preferences and where there are preferences there is bias. Everyone will have a preference towards a particular editor and don’t let a particular bias influence you to go down a particular direction without considering your options. Speaking from personal experience I was encouraged to use ‘vi’ as it represented the preference of the group I was in, but because I was a late starter to the command line I struggled for the longest time to try and become familiar with it. I know I should have tried harder, but I failed. For a while I wandered in the editor wilderness trying desperately to cling to the GUI where I could use ‘gedit’ or ‘geany’ and then one day I was introduced to ‘nano’.
This has become my preference and I am therefore biased towards it. Don’t take my word for it. Try alternatives. I’ll describe ‘nano’ below, but take that as a possible path and realise that whatever editor works for you will be the right one. The trick is simply to find one that works for you.

The nano Editor

The nano editor can be started from the command line using just the command and the /path/name of the file.
If the file requires administrator permissions it can be executed with ‘sudo`.
When it opens it presents us with a working space and part of the file and some common shortcuts for use at the bottom of the console;
nano Interface
It includes some simple syntax highlighting for common file formats;
nano Syntax Highlighting
This can be improved if desired (cue Google).
There is a swag of shortcuts to make editing easier, but the simple ones are as follows;
  • CTRL-x - Exit the editor. If we are in the middle of editing a file we will be asked if we want to save our work
  • CTRL-r - Read a file into our current working file. This enables us to add text from another file while working from within a new file.
  • CTRL-k - Cut text.
  • CTRL-u - Uncut (or Paste) text.
  • CTRL-o - Save file name and continue working.
  • CTRL-t - Check the spelling of our text.
  • CTRL-w - Search the text.
  • CTRL-a - Go to the beginning of the current working line.
  • CTRL-e - Go to the end of the current working line.
  • CTRL-g - Get help with nano.

The post above (and heaps of other stuff) is in the book 'Just Enough Linux' that can be downloaded for free (or donate if you really want to :-)).

Thursday, 7 April 2016

Links in Linux

The following post is a section of the book 'Just Enough Linux'.  The entire book can be downloaded in pdf format for free from Leanpub or you can read it online here.
Since this post is a snapshot in time. I recommend that you download a copy of the book which is updated frequently to improve and expand the content.
---------------------------------------

A link in a Linux file system provides a mechanism for making a connection between files. While comparisons can be made to shortcuts in Windows, this is not terribly accurate as linking in Linux has a lot more utility.
To get the full benefit from the description of links we should be passingly familiar with the previous discussion on files and inodes. We should recall that a Linux file consists of three parts;
  1. The filename and it’s associated inode number
  2. An inode that describes the attributes of the file
  3. The data associated with the file.
The file name and inode number point to an inode that has all the information about the file and in turn points to the data stored on the hard drive.
Links can be identified when listing files with ls -l by a l that appears as the first character in the listing for th file and by the arrow (->) at the end that illustrates the link. For example by executing the following list command we can see (as only one amongst many files) that the file /etc/vtrgb is a link to the file /etc/alternatives/vtrgb.
Note the leading l and link sign for vtrgb in the output.
drwxr-xr-x  2 root root     4096 Apr 17  2014 vim
lrwxrwxrwx  1 root root       23 Jul 10 07:45 vtrgb -> /etc/alternatives/vtrgb
-rw-r--r--  1 root root     4812 Feb  8  2014 wgetrc
drwxr-xr-x  2 root root     4096 Jul 10 08:04 wildmidi

A soft link can also be referred to as a symbolic link, but is probably more frequently called a ‘symlink’. A symlink has a file name and inode number that points to it’s own inode, but in the inode there is a path that redirects to another inode that in turn points to a data block.
Soft Link
Because inodes are restricted to the partition they are created on, a symbolic link allows redirecting to a path that can cross partitions.
If we edit the linked file, the original file will be edited. If we delete the linked file, the original file will not be deleted. But if we delete the original file without deleting the link, we will be left with an orphaned link.
Soft links can link to directories and files but if the file is moved the link will no longer work.

A hard link is one where more than one file name links to an inode.
Hard Link
This means that two file names are essentially sharing the same inode and data block, however they will behave as independent files. For example if we then delete one of the files the link is broken, but the inode and data block will remain until all the file names that link to the inode are deleted.
Hard links only link to a file (no directories), cannot span partitions and will still link to a file even if it is moved.

Hard links
  • Will only link to a file (no directories)
  • Will not link to a file on a different hard drive / partition
  • Will link to a file even when it is moved
  • Links to an inode and a physical location on the disk
Soft links (or symbolic links or symlinks)
  • Will link to directories or files
  • Will link to a file or directory on a different hard drive / partition
  • Links will remain if the original file is deleted
  • Links will not connect to the file if it is moved
  • Links connect via abstract (hence symbolic) conventions, not physical locations on the disk. They have their own inode


The post above (and heaps of other stuff) is in the book 'Just Enough Linux' that can be downloaded for free (or donate if you really want to :-)).

Tuesday, 5 April 2016

Linux files and inodes

The following post is a section of the book 'Just Enough Linux'.  The entire book can be downloaded in pdf format for free from Leanpub or you can read it online here.
Since this post is a snapshot in time. I recommend that you download a copy of the book which is updated frequently to improve and expand the content.
---------------------------------------

Having already established that everything is a file in a Linux operating system we find ourselves in the situation where files are suddenly slightly more interesting and mysterious, and it’s useful to get a better understanding of how files are represented in a Linux operating system so that the context of other functions can be better understood (thinking of links).

Linux files can be best understood by thinking of them as being described by three parts;
  • The file name
  • An inode
  • The data

The file name

Individual file names are typically restricted to 255 characters and can be made up of almost any combination of characters. The only ‘special’ or ‘reserved’ character when assigning a file name is the forward slash (/). This character is used solely to separate directories and file names as appropriate.
The other component of a file name is an inode number. This is a number assigned by the operating system which acts as a pointer to an inode (Index Node) which is an object that can be thought of as an entry in a database that stores information specifically about that file.
We can list the inode numbers associated with files by using the -i option when listing files. For example;
The output (reproduced in edited form below) shows;
pi@raspberrypi ~ $ ls -i python_games/
45014 4row_arrow.png           63554 inkspillresetbutton.png
49906 4row_black.png           63556 inkspillsettingsbutton.png
54215 4row_red.png             63559 match0.wav
63530 cat.png                  63569 princess.png
63533 drawing.py               63488 RedSelector.png
63544 gem6.png                 63576 star_title.png
63545 gem7.png                 63578 tetrisb.mid
63546 gemgem.py                63579 tetrisc.mid
63553 inkspilllogo.png         63522 Wood_Block_Tall.png
63552 inkspill.py              63582 wormy.py
The numbers to the left of the file names are the associated inode numbers.
The file name and associated inode is in the directory that the file is contained in. Each directory (which is itself just a file) contains a table of the files it contains with the associated inode numbers.

The inode

Each file has an associated inode (index node). The inode is analogous to a database entry that describes a range of attributes of the file. These attributes include;
  • The unique inode number
  • Access Control List (ACL)
  • Extended attributes such as append only or immutability
  • Disk block location
  • Number of blocks
  • File access, change and modification time
  • File deletion time
  • File generation number
  • File size
  • File type
  • Group
  • Number of links
  • Owner
  • Permissions
  • Status flags
You may well ask what the point of having numbered inodes is. Good question. They have a range of uses including being used when there are files to be deleted with complex file names that cannot be easily reproduced at the command line or enabling linking.
Interestingly, space for inodes must be assigned when an operating system is installed. Within any file-system, the maximum number of inodes, and hence the maximum number of files, is set when the file-system is created.
Therefore there are two ways in which a file-system can run out of space;
  1. It can consume all the space for adding new data (i.e. run out of hard drive space), or
  2. It can use up all the inodes.
Running out of inodes will bring a computer to a halt because exhaustion of the inodes prohibits the creation of additional files. Even if sufficient hard drive space exists. It is particularly easy to run out of inodes if a file-system contains a very large number of very small files.

The post above (and heaps of other stuff) is in the book 'Just Enough Linux' that can be downloaded for free (or donate if you really want to :-)).