Part 1 - Navigating the filesystem using the terminal

You are probably already familiar with your computer's [filesystem]. If you've ever looked at files on your Desktop, or opened Finder/Explorer and navigated to your Documents folder, you've been using the file system. Data is stored on your hard drive as a continuous binary sequence (just 1s and 0s) and without a filesystem, it would be impossible to find anything!

Finder/Explorer are software programs used to interact with your filesystem. The filesystem contains a mix of files, which are blobs of information (eg. a .docx file containing a MS Word doc) and folders, which can contain files as well as other folders. Just to be confusing, what we colloquially call folders are technically called "directories." I will use the terms interchangeably.

You may not have thought much about how the filesystem is organized, but you probably have an intuitive sense of how this works. Your [home] directory contains the Desktop and Documents directories, each of which might contain many other subdirectories and/or files, and each subdirectory may have yet more subdirectories and files and so on.

Finder/Explorer are examples of a graphical user interface (GUI - pronounced "gooey"), that is a program that allows you to interact with it visually. However, GUIs are difficult to design and maintain, so most bioinformatics tools are built around [commandline] or text-based interfaces. The good news is that the organization of the filesystem is the same whether you're interacting with it in a GUI or in the terminal. And once you learn how to use the command line interface, you'll see that it is a powerful way to get a lot done quickly.

1.1: Open the terminal application

Mac Users:

1. Open Applications
2. Click Open the Terminal application

Windows Users:

1. Click Start -> Programs
2. Open Windows Subsystem for Linux

You will be greeted with a white or black box that contains some text, and a cursor next to the dollar symbol $. This is the command prompt.

1.2 Walk around your file system

Think of your filesystem as a branching network of roads that you can walk along. Each directory is a location that has a path back to its parent directory, may or may not contain objects (files), and may have one or many roads leading to new locations (subdirectories).

Your present location, referred to as the [workingdirectory], is your home folder when you first open the terminal. Let's see where that is.

In your terminal, type pwd and press enter. This is the "print working directory" command.

$ pwd

/home/kevin

If you're using a Mac, this probably displayed something like /Users/yourname.

Let's take a look around. the ls command is used to list the contents of the directory.

$ ls

bin      Documents     R       scratch
Desktop  Downloads     Public  repos

Your output will likely look a bit different from mine, but you'll probably see Documents and Desktop, as well as other directories you may recognize.

The ls command can also take a directory as an "[argument]." We'll talk more about what that means later - for now, just add Desktop to the command, separated by a space

$ ls Desktop

df.csv  dm.csv  dupes.csv  itp.svg

Again, your output should look different from mine, but (assuming you store anything on your Desktop), you should see a list of files and directories. If your Desktop is pristine, congratulations! You won't see any output.

The filesystem is organized hierarchically - At the very top of the hierarchy is the [root], which you can think of as a folder that contains all other files and folders. On a Mac (and other Unix systems), the root is referred to with a single forward slash (/), While on Windows machines, the root is probably C:\.

Moving around

Say you are sitting in your room, and you want to give someone directions to Lulu. One way to do this would be to give directions from where you are:

1. Go out the door,
2. Go down the hall to the elevator
3. ... etc

In this case, you're providing a [relativepath] - the directions only make sense if the person is starting in your room. If they're at the Science Center and they follow your instructions, they will become hopelessly lost.

Alternatively, you could provide directions from a shared point of reference.

1. Start at the campus gate on the corner of Weston Rd and Central St
2. Walk down the path and turn right just past the Child Study Center towards the Botanical Gardens
3. ... etc

In this case, you're providing an [absolutepath] - no matter where the person is at the time, the directions will make sense.

An absolute path is great, since the same instructions are useful in most situations, but in many cases, it would be tedious to always have to start instructions from the same location.

When providing a file path to a unix operating system, you can also provide relative or absolute paths. There are two shared points of reference, your home folder and root, which are specified by ~/ and / respectively when placed at the beginning of the path.

Up above, when we looked at the contents of the Desktop using ls, we provided the relative path from where we were in the home folder. You can tell, because the path did not start with ~/ or /.

We could have done the same thing using an absolute path:

$ ls ~/Desktop

df.csv  dm.csv  dupes.csv  itp.svg

The output here should be the same as what you got above. But what if we're in a different location?

In order to change the working directory, we use the cd command, giving a path as an argument. For example, to make Desktop the current working directory:

$ cd ~/Desktop

It's a good idea (in general, but especially while learning) to repeatedly check that the thing you think happened actually happened.

$ pwd

/home/kevin/Desktop

$ ls

df.csv  dm.csv  dupes.csv  itp.svg

Notice that with ls, we didn't have to give Desktop as an argument this time. By default, ls lists files in the current working directory.

Let's go back to the home folder. Again, we can provide a relative or an absolute path - the absolute path back to home is always ~/, but how do we do the relative path? To walk into subdirectories, we can just provide the name of the subdirectory, but how do we walk "backwards", into a parent directory?

In unix, parent directories are indicated with ../. So to go back home from /home/kevin/Desktop using a relative path, I would write:

$ cd ../
$ pwd

/home/kevin

Practice

Practice moving around your file system using cd, and checking where you are using pwd and ls. If you get lost, just enter cd ~/ to get yourself home.

1.3 Manipulating files and directories

Of course, looking around is not the only thing you can do from the terminal. Using the terminal allows you to rapidly view, edit, and otherwise manipulate stuff in your file system.

First, let's create a directory for use in this course. You can put it in your home directory, or in Documents, or if you have a directory where your course files usually go, that's fine too.

I'll assume you're making this in ~/Documents/, but if you want it somewhere else, just modify the path accordingly. the command to make a directory is mkdir <path>.

Create and explore directories and files

$ mkdir ~/Documents/bisc195
$ cd ~/Documents/bisc195
$ pwd

/home/kevin/Documents/bisc195

Now, copy and execute the following command - don't worry about what it does. (if you really want to know, you can read about it here: [whileloop])

bisc195 $ for i in {1..10}; do echo "This is file #${i}" > "file${i}.txt"; done

It should finish very quickly. Let's look at what it did

bisc195 $ ls

file1.txt  file2.txt  file4.txt  file6.txt  file8.txt
file10.txt file3.txt  file5.txt  file7.txt  file9.txt

The code you executed created 10 files, each of which contains a bit of text. Look at the contents of file1.txt using the head command.

bisc195 $ head file1.txt

This is file #1

The head command prints the first 10 lines of a file by default, but this file only has 1 line, so that's all that's shown.

We can look at multiple files at the same time using the concatenate command:

bisc195 $ cat file1.txt file2.txt file3.txt

This is file #1
This is file #2
This is file #3

cat takes any number of files (even just 1!) and prints their entire content to the screen one after another. Be careful - some files are really long, and will go on and on and on and on and...

Glob patterns

If we want to print the content of all 9 files, we could type them all out, but that would get tedious. There are many useful shortcuts in programming (programmers ~~are lazy~~ like efficiency), one of which is glob patterns.

The most common glob is *, which stands for any number of characters, including none.

For example,

bisc195 $ cat *.txt

This is file #1
This is file #10
This is file #2
This is file #3
This is file #4
This is file #5
This is file #6
This is file #7
This is file #8
This is file #9

Here, *.txt means "anything that ends with .txt".

Another glob is ?, which matches any single character.

bisc195 $ cat file?.txt

This is file #1
This is file #2
This is file #3
This is file #4
This is file #5
This is file #6
This is file #7
This is file #8
This is file #9

Notice that in this example, file10.txt is not included, since there are 2 characters between file and .txt.

Finally, you can use brackets to specify specific characters, or ranges of characters:

bisc195 $ cat file[2468].txt

This is file #2
This is file #4
This is file #6
This is file #8

bisc195 $ cat file[6-8].txt

This is file #6
This is file #7
This is file #8

Let's make a new file that contains the content of all of the others. The > character at the end of a command redirects the output of that command into a new file, so we can do:

bisc195 $ cat file*.txt > all_files.txt
bisc195 $ ls

all_files.txt file10.txt    file3.txt     file5.txt     file7.txt     file9.txt
file1.txt     file2.txt     file4.txt     file6.txt     file8.txt

Notice that the cat command in this case didn't print anything to the screen.

We can check that the right content went into all_files.txt using cat again.

bisc195 $ cat all_files.txt

This is file #1
This is file #10
This is file #2
This is file #3
This is file #4
This is file #5
This is file #6
This is file #7
This is file #8
This is file #9

move and copy files

We'll use this directory for the rest of this course, but it could use some more organization. First, let's make a directory called lesson1.

Can you remember how to make a new directory? I'm not going to provide the command for this step, but before moving on, you should be able to execute the following command without getting an error:

bisc195 $ ls lesson1/
bisc195 $

When you first create the directory, it will be empty, so ls won't return anything.

If you see ls: lesson1: No such file or directory, don't move on to the next step.

Once you've created the lesson1/ directory, let's move the text files we've created into it. The mv command take the form mv <source> <destination>. For example:

bisc195 $ mv file1.txt lesson1/file1.txt
bisc195 $

Now,

bisc195 $ ls lesson1

file1.txt

bisc195 $ ls

all_files.txt file2.txt     file4.txt     file6.txt     file8.txt     lesson1
file10.txt    file3.txt     file5.txt     file7.txt     file9.txt

As you can see, file1.txt has been moved to the lesson1/ subdirectory, and is no longer in the current directory.

If we don't want to change the name of the file, we can actually just provide a directory as the destination, rather than typing out the entire new path.

bisc195 $ mv file2.txt lesson1/
bisc195 $ ls lesson1

file1.txt file2.txt

In this way, we can move multiple files at the same time, separated by spaces, as long as the final argument is a directory:

bisc195 $ mv file3.txt file4.txt file5.txt lesson1/
bisc195 $ ls lesson1/

file1.txt file2.txt file3.txt file4.txt file5.txt

Remember, ls without an argument shows us the contents of the current directory.

bisc195 $ ls

all_files.txt file10.txt    file6.txt     file7.txt     file8.txt     file9.txt     lesson1

Once you've moved all the text files other than all_files.txt into the lesson1 folder, change the working directory to lesson1 using cd

The copy command works the same way as mv, except that the original stays where it is.

Check to make sure you're in the lesson1 directory:

lesson1 $ pwd

/home/kevin/Documents/bisc195/lesson1

lesson1 $ ls ./

file1.txt  file2.txt  file4.txt  file6.txt  file8.txt
file10.txt file3.txt  file5.txt  file7.txt  file9.txt

lesson1 $ ls ../

all_files.txt lesson1

Let's copy all_files.txt into the current directory using cp:

lesson1 $ cp ../all_files.txt ./
lesson1 $ ls

all_files.txt file10.txt    file3.txt     file5.txt     file7.txt     file9.txt
file1.txt     file2.txt     file4.txt     file6.txt     file8.txt

$ ls ../

all_files.txt lesson1

As you can see, all_files.txt is now present in both the parent bisc195/ directory and the lesson1/ directory.

Rename and remove files

We actually don't need 2 copies of the all_files.txt. Let's remove the one in the bisc195 directory.

lesson1 $ rm ../all_files.txt
lesson1 $ ls ../

lesson1

Come to think of it - we don't need any of these files. Let's remove them all.

Before we do anything, verify we're still in the lesson1 directory using pwd and ls.

Then, remove all of the .txt files in the current directory:

lesson1 $ rm *.txt
lesson1 $ ls
lesson1 $

We don't need the lesson1 directory either. First, move to the parent directory (bisc195/) using cd. rm doesn't work on directories:

bisc195 $ rm lesson1

rm: lesson1: is a directory

Instead, use the remove directory command:

bisc195 $ rmdir lesson1
bisc195 $ ls
bisc195 $

Summary

• cd changes the current working directory
• pwd prints (shows) the current working directory
• ls <path> lists the contents of a directory
  • ls without a path argument lists the contents of the current working directory
  • ls <glob> lists the files/directories matching a pattern (eg ls *.txt)
• use relative or absolute paths
  • aboslute paths start with / (root) or ~/ (home)
  • relative paths start with ./, ../ or file and directory names
• mkdir for creating directories
• mv <source(s)> <destination> to move files and directories
• cp <source(s)> <destination> to copy files and directories
• glob patterns can be used to act on many files at once
  • * = 0 or more characters (eg. *.txt)
  • ? = exactly 1 character (eg. file?.txt)
  • [] = specific characters or range of characters
    • eg file[246].txt = file2.txt file3.txt file6.txt
    • eg file[5-7].txt = file5.txt file6.txt file7.txt
• rm removes files
• rmdir removes (empty) directories

Key Terms

More info

Here's some more information for the curious, but at least for now, it's not necessary to understand it.