Linux Fundamentals

I am writing this blog after watching the Linux and Docker Fundamentals workshop by Saiyam Pathak. I am writing a blog for Linux. So, let's jump into the Linux fundamentals and feed our brains some Linux food.

What is an operating System?

An Operating System is an interface between a computer user and computer hardware. It is a software which performs all the basic tasks like:

• file management,
• memory management,
• process management,
• handling input and output, and
• controlling peripheral devices such as disk drives and printers.

It also allows you to communicate with the computer without knowing how to speak the computer's language. Without an operating system, a computer is useless.

Linux OS

First of all, Linux is not an OS. It is a kernel. So, when we say Linux OS, it means that the OS has Linux as its kernel.

Linux Distributions (Distro)

A Linux distribution is an operating system composed of the

• Linux kernel,
• GNU tools,
• additional software, and
• a package manager
• It may also include
  • display server, and
  • a desktop environment

The term is Linux distribution because an entity like Debian or Ubuntu ‘distributes’ the Linux kernel along with all the necessary software and utilities (like network manager, package manager, desktop environments, etc.).

In short, Linux -> Kernel; Linux Distro -> Operating System

To read more about distros, click here.

Some of the famous Linux distros are:

• Fedora
• Ubuntu
• Arch Linux
• Debian

File System

A file system is a logical collection of files on a partition or disk.

• / - This is the root of your filesystem, where everything begins.

• /etc - This directory contains system configuration files.

• /home - This is the default home directory for all users (except the root user).

• /root - This is the home directory for the root user.

• /dev - This is where your devices such as your hard disks, USB drives, and optical drives reside on your system.

• /opt - This is where you can install additional 3rd party software.

• /boot - All the files required for your system to boot are stored here.

• /bin - This is where essential binaries (programs) reside on your system. In other words, your Linux commands executables.

• /sbin - This is where system binaries (programs) that are typically used by the system administrator are stored. Commands here require sudo access.

• /opt - This is where you can install additional 3rd party software (not coming from your distribution's package manager).

• /tmp - This is where temporary files are stored; they are usually deleted after a system reboot, so never - store important files here!

• /var - This directory contains files that may change in size, such as mail spools and log files. Many sysadmins store their web services here.

• /usr - This directory contains files and utilities that are shared between users.

• /lib - This directory contains libraries needed by the essential binaries in the /bin and /sbin directories. A library is basically a set of precompiled functions that can be used by a program.

• /proc - This is where information about running processes is stored.

Workshop #1

Before moving on, I want you to read this blog in order to brush up some of the basic Linux commands that you may have already learnt.

1.1 OS Info

Aim: Find the info about the current distro and other system info such as CPU and RAM.

To do so, there are various approaches:

1. lsb_release -a → print distribution-specific information
2. cat /etc/os-release → contains data that is defined by the operating system vendor.
3. uname -a → Displays the operating system name as well as the system node name, operating system release, operating system version, hardware name, and processor type.
4. lscpu → Display information about the CPU architecture
5. lsmem → List the ranges of available memory with their online status

1.2 SysLog

Aim: Find where syslog is on a Linux system.

Before diving in, let's learn about a new command i.e. find command which search for files in a directory hierarchy. It's syntax is:

find / -name file1.txt → starts searching a file named `file1.txt` from the root directory(`/`)  and returns the location of the file.

Now, we want to find syslog files. Run this command find / -name syslog and you will get an output like this:

/var/log/syslog file stores general messages, as well as system-related information; this log stores all activity data across the global system. Now, we will display the contents of this file by executing cat /var/log/syslog command:

1.3 User

Aim: Create a new user named chris that will use the bash shell as their default shell.

It's new command time! And, the command is useradd.

useradd - create a new user or update default new user information.

First, we will display the default user information. After that we will change the shell from default to "bash" shell. Then, we will add a new user named chris.

• useradd -D → alllows us to output the info that is default for all users.

• Change the default shell from sh to bash. To do so execute this command: useradd -D -s /bin/bash→ -s flag is used to change the shell name of the default shell; /bin/bash is used to replace the default shell with ‘bash’. This means whenever we create a new user in our system it will have bash as it's default shell.

  

• You may also display what shells are available in your system; these are all the shells that you can potentially switch to if you want to. To do this run cat /etc/shells command.

  

Now, it's time to add a new user named chris.

• useradd -m chris → creates a new user named chris; -m flag creates the home directory for the newly added user. Use sudo in case you are not logged in as root. User chris created. Now, the home directory in root will look like this:

• passwd chris → to set a password for chris

• When you add a new user then, a group by the user name automatically gets created, and that user, by default, belongs to this newly created group.

  In this case, a group named “chris” gets created. This user chris also belongs to this chris group.

  How to add user to a group?

  New command time! And the commands are : groups and usermod.

• First of all let’s see to how many group our user chris belongs to. You can display this info by using groups userName command. As you can see that chris belongs to only one group i.e. chris

• If you want to add user to a group(s) run this command:

  usermod -aG group1Name, group2Name, group3Name userName

  - `usermod` → modify a user account
  - `-a` :  append; Add the user to the supplementary group(s). Use only with the -G option.
  - `group1Name` , `group2Name`, `group3Name` →these are the groups that we want userName to be added in.
  

• Now, let’s say we want to add user ‘chris’ to our sudo group so as to let him enjoy root user privileges. Use usermod -aG sudo chris commad to make it happen. User chris added to “sudo” group

• Check out if the command was successful by running the groups chriscommand.

  You can see that sudo group is now added in the list of groups to which chris belongs to.

• In case you don't add chris to the sudo group, and then when you try to access “sudo” command as chris then this will happen:

  Switch to a different user

  Swtich to user chris using the su command

  • su - chris
    • su : switch user
    • - : Start the shell as a login shell with an environment similar to a real login; changes to the target user’s home directory
    • chris : target user name

  • Here’s an example:

    • First of all let’s see who is the current user by executing the whoami command.

    • Switch to user chris by running the following command, and enter the password for the chris user:

    • As you can see below that the current user sanskriti has been switched to chris :

  • To exit out of the chris user use exit command:

List the groups to which users belong

• In order to do that, just display the content of /etc/group file.

  For e.g: Let’s take 5th entry from top:
  1.) adm → Group Name
  2.) x → Password
  3.) 4 → Group ID (GID)
  4.) syslog, sanskriti → Group List: list of user names of users who are members of the group. In this case, syslog and sanskriti are the members of adm group

How to list all users in a Linux system?

• cat /etc/passwd : The is the user database, with fields giving the username, real name, home directory, and other information about each user; this file stores essential information required during login. In other words, it stores user account information. For e.g, Let’s take 1st entry from top:
  1.) root → User Name
  2.) x → Password
  3.) 0 → User ID (UID)
  4.) 0 → Group ID (GID)
  5.) root → User ID Info ; The comment field; allow you to add extra information about the users such as user’s full name, phone number etc.
  6.) /root → Home directory; directory the user will be in when they log in
  7.) /bin/bash → Command shell; absolute path of a command or shell

1.4 Group

Aim: Create a new group named 'docker' and add the user chris to the group docker.

How to create a group?

New command time! And the commands are :

`groupadd`-> to create a new group. 

`getent`-> Allows users to fetch details from several text files called databases. It contains various databases like hosts, services, networks, group etc.

• groupadd docker : creates a new group named “docker ”

• getent group docker : to get info about the docker group; or to get entry named "docker" from the group database
  

  • docker : Group Name
  • x : Password
  • 1002 : Group ID (GID)

Creating docker group and adding chris to it

1. Create docker group : groupadd docker. docker group created. As you can see that no user belongs to this group.
2. Add chris to it: sudo usermod -aG docker chris.
3. Check whether chris was added to docker group or not. You can check it using several methods:
   1. Using groups chris command which will list all the groups to which chris belongs to. As you can see here, user chris belongs to groups named chris , sudo and docker.
   2. Use cat /etc/group | grep docker to get the info about docker group.
   3. Use getent group docker command.

Commands and Packages

Package manager

Package Managers are essentially software applications that help users to: Search, Download, Install, Remove and Update software applications on their computer operating system.

There are different package managers for different distros.

 - apt is a package manager for ubuntu
 - yum is a package manager for fedora

Why do we need a package manager?

• Package manager makes it easy to install software because it knows where all the files go, and which one should be executable, and how to reference that to your current path.

• Package manager automatically enables the service and load it in, so that we can start using it right away.

  Workshop #2

  2.1 Prompt

  Aim: Change the prompt for your user to MyNewPrompt$.

  Prompt

  • Command prompt is a short text at the start of the command line followed by prompt symbol on a command line interface.
  • On my system it is:

     sanskriti@Sanskriti-twts:~$
    

  • $PS1 is an environment variable which stores your normal "waiting for a command" prompt and it's value can be displayed using echo $PS1 command.

  How to change your prompt?

  • The current prompt is chris@Sanskriti-twts:~$
  • You can print your prompt by executing echo $PS1 command.

  • To change your prompt from chris@Sanskriti-twts:~$ to MyNewPrompt $ , simply change the value of the environment variable PS1 which stores stores your normal "waiting for a command" prompt.

    • Run PS1="MyNewPrompt $" command
      
    • But this change is not persistent. As soon as you log out the changes will be lost. This happens because the .bashrc file was not updated. In simple terms, the value of the env variable $PS1 was temporarily changed and it lost it’s temporary value once you logged out or exited.

  How to make variable persistent?

  • To make variables persistent edit the .bashrc file (if you are using korn shell then edit .kshrc file), or, you need to edit /etc/environment file and then logout, or, restart system to bring changes into effect.

        - .bashrc : Changes are done only for one user
        - /etc/environment : Changes are visible across all users
    

  • But as you can see below that there were no changes made in .bashrc file for the env var PS1 ,therefore, the command prompt is not permanently changed.
  • Two methods to edit .bashrc file :
    1. Append the variable and it’s value in .bashrc file by executing echo 'PS1="MyNewPrompt $ " ' >> .bashrc command. As you can see that the prompt has changed.
    2. Make changes directly to the .bashrc using vim or nano editor. Run source .bashrc command to make the changes effective. These changes are persistent. You may check it out by logging out the system and then logging in.

  2.2 Directory

  Aim: Create a directory named dir1 in the chris home folder. Copy that directory to "dir2". Delete "dir2". Let's do it.

    - To create directory named *dir1* use `mkdir -p dir1`. 
    - To copy *dir1* to *dir2*,  run `cp -r dir1 dir2` command.
    - To delete *dir2*, execute `rm -r dir2`.
  

  See all of this in one go:

  2.3 File

  Aim: Create a file named file1.txt in the dir1 directory. Copy that file to file2.txt, rename file2.txt to file3.tx. Let's go!

    - Change the directory to dir1. 
    - Run 'touch file1.txt' command to create file named *file1.txt* .
    - To copy *file1.txt* to *file2.txt*, execute `cp file1.txt file2.txt`.   
    - Rename *file2.txt* to *file3.txt* using 'mv file2.txt file3.txt' command.
  

  See above commands in action:

  2.4 Install

  Aim: List the sources in local repository & update. Install docker engine, containerd, and Docker compose.

  First, let's list the sources in local repo.

  • cat /etc/apt/sources.list → It is a configuration file for Linux's Advance Packaging Tool, that holds URLs and other information for remote repositories from where software packages and applications are installed.
  • cd /etc/apt/sources.list.d → It’s a directory. Using the directory you can easily add new repositories without the need to edit the central /etc/apt/sources.list file. i.e. you can just put a file with a unique name and the same format as /etc/apt/sources.list into this folder and it is used by apt. As you can see, if you cat the google-chrome.list file stored in /etc/apt/sources.list.d folder it displays the URL and other information for remote repository from where software packages and applications of google chrome are installed.

  How to install docker?

  1. Install ca-certificates package→ sudo apt install -y ca-certificates : It is a digital certificate that is used to verify the identity of 3rd parties, and encrypt data between you and said 3rd party.
  2. Install curl package → sudo apt install -y curl : curl is a command-line tool to transfer data to or from a server, using any of the supported protocols (HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP, or FILE).
  3. Install gnupg package →sudo apt install gnupg : GnuPG allows you to encrypt and sign your data and communications; getting the neccessary files that allow us to work with GPG keys which are just making sure that software you are pulling down is valind and it’s coming from trusted resources.
  4. sudo apt install -y lsb-release
  5. The docker that we want is not added to the /etc/apt/sources.list. Therefore we will do it: sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  6. echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  7. You can check that it’s now added to the sources.list by navigating to the /etc/apt/sources.list.d directory and then listing the files of that folder.
  8. Run sudo apt search docker-ce
  9. Update sudo apt update
  10. Run sudo apt search docker-ce
  11. Install Docker Engine, containerd, and Docker Compose: sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

File Permissions & Redirects

• UNIX is a multi-user system. Every file and directory in your account can be protected from or made accessible to other users by changing its access permissions. Every user has responsibility for controlling access to their files.
• Permissions for a file or directory may be restricted to by types.
• There are 3 type of permissions

  • r - read
  • w - write
  • x - execute = running a program
  

• Each permission (rwx) can be controlled at three levels:

  • u - user = yourself
  • g - group = can be people in the same project
  • o - other = everyone on the system
  

• File or Directory permission can be displayed by running ls –l command

• -rwxrwxrwx

  

  

  • - = First dash or bit identifies the file type;
    • - → file
    • d → directory
    • l → link to another file
  • --- = 2nd 3 bits defines the permission for user (file or dir owner)
  • --- = 3rd 3 bits defines the permission for group

  • --- = 4th 3 bits defines the permission for everyone else

    Command to change permission

  • chmod - This command is used to change file mode bits; in simple words this command changes permission of a file. Example:

    • chmod u-r file1 → Removes the user permission to read.

    • chmod u+r file1 → Grants user the permission to read.

      You can use "a" to define all users.

  • Permissions can also be changed through numerical method.
    • chmod 744 file1
    • Each of the permission types is represented by either a numeric equivalent:
      • read = 4
      • write = 2
      • execute = 1
    • Example:
      • A permission of 4 or r would specify read permissions. If the permissions desired are read and write, the 4 (representing read) and the 2 (representing write) are added together to make a permission of 6.

• Examples: If the permission desired for file1 is user: read, write, execute, group: read, execute, other: read,execute, the command to use would be: chmod 755 file1 or chmod u=rwx,go=rx file1

  Workshop #3

  3.1 Pipe

  Aim: View contents of /var/log/syslog and filter based on the word "ubuntu". Pipe to less.

  Pipe (|) is used to combine two or more commands, and in this, the output of one command acts as input to another command, and this command’s output may act as input to the next command and so on.

  Do it using sudo cat /var/log/syslog | grep ubuntu | less command. This command first redirects the output of the sudo cat /var/log/syslog command as an input to the next command i.e. grep ubuntu, which then redirects it's output to the next command. This is how piping works.
  

  

  3.2 Script

  Aim: Create a file with vim named "script.sh" in the "chris" home folder. Type the word "bash" inside of the file.

  First of all let's learn in brief about the Vim editor.

  Vim Editor

  • You can create a file and be in the file all at the same time by using the vim script.sh command.
  • Some of the modes in Vim and commands in those modes:
    1. Insert Mode
       1. Press i key to get into Insert mode form the Command mode.
       2. To switch to the Insert mode from Visual mode, first shift to Normal mode by pressing the Esc, then press i to get into the Insert mode.
       3. Write whatever you want.
    2. Command Mode / Normal mode
       1. Normal mode is the initial mode of the Vim editor.
       2. All the keystrokes you perform are interpreted as commands.
       3. To access normal mode from other modes, press Esc key.
       4. Some of the commands are:
          1. gg- Places the cursor at the start of the file
          2. Shift + g - Places the cursor at the last line of the file
          3. Shift + 4 - Places the cursor at starting of the line
          4. Shift + 6 - Places the cursor at the end of the line
    3. Visual Mode
       1. This mode allows you to select text so that you may perform certain operations (cut, copy, delete) on it.
       2. Press v when you are in Commad mode to enter into the Visual mode.
       3. To switch to the Visual mode from Insert mode, first shift to Normal mode by pressing the Esc, then press v to get into the Visual mode.
          1. Use navigation commands, such as the Arrow keys, to highlight multiple lines of text.
          2. Once the desired text is highlighted, use commands to manipulate it. Press d to delete, then move the cursor to the new location, and press p to paste the text.
          3. y (yank) can be used instead of d (delete) if you want to copy the task.
  • When using movement commands, you can put a number in front of them to make Vim complete a command multiple times. For e.g, 4gg will place the cursor at the start of 5th line of the file instead at the start of the file. It means running the gg command but for “four” times.

  • To save a file press Esc key to exit insert or visual mode and the type :wq to save the file and quit the editor. But before that insert the line #!/bin/bash in the script.sh file to achieve our aim.

    We have created script.sh file in the chris home folder.

  3.3 Chmod

  Aim: Change the ownership of this file to be executable for the user "chris".

  Use chmod u+x script.sh command to give user chris the permission to execute. If the file doesn't have the executable permission for a user then it can't be executed by that user.

  3.4 Insert

  Aim: Paste in text from the web into script.sh and use ":set paste" to take away formatting. Run the script.

  Pasting with and without formatting:

  • use :set paste command in Vim to keep the formatting while pasting.
  • To paste without formatting use :set nopaste command.
  • To run script using one of the following commands:
    1. ./filename.sh
    2. sh filename.sh
    3. bash filename.sh

  Services & Git

• Services or Daemons

  • A service is a process or group of processes (commonly known as daemons) running continuously in the background, waiting for requests to come in (especially from clients).
  • Daemons are essential; without them we couldn’t do anything. There is a daemon for every service like networking daemon, printing daemon, ssh daemon.

    How can we identify if a process is a daemon process?

    • A daemon service will have a d at the end.
    • Let’s display bluetooth daemon. To display running process or services run ps -aux command.
    • As you can see below that the bluetooth daemon process ends with d.

• systemd

  • The way we control daemons(or system services) is by dealing the master daemon.
  • System services or system processes or units or daemons are one and the same thing.

  • systemd ← Master Daemon → Boss of the daemons

    • It is in charge of the daemons.

      • It starts the daemons, It stops the daemons, It restarts the daemons.
      • If you wanna do anything with daemons, you gotta go through systemd .
      • systemd has two main jobs:
        1. Service manager
           1. Manages all the services or daemons.
           2. systemd call daemons or services as “units”.
        2. Initialization System (”init”)
           1. It is a massive job and vital to the boot process.
           2. systemd isn’t the only init system out there. There are others but systemd will be the main one you see on any modern linux distro.
           3. systemd daemon starts all the processes.

      • You can use pstree command to see that all the processes are started by the systemd daemon. pstree is a Linux command that shows the running processes as a tree.

        

      • Understanding the below image (I took this ss from this video):

        • Boot loader loads the kernel which in turn kisckstarts the systemd daemon or service or process (whatever you wanna call it).
        • Then systemd mounts the files system, and starts all other services or processes or units(as systemd calls it) by forking.

  Workshop #4

  4.1 Daemon

  Aim: View the services running on Linux for all users and also just for the "chris" user. List just the daemons on Linux.

• To view the services running on Linux for all users run ps -aux command.

• To view the services running just for the "chris" user execute ps -aux |grep chris command

  How to list services running in our system?

  • Run the sudo systemctl list-units -t service command. This command lists the unit files available in the system that are of type “service”. It gives a runtime snapshot of units.

  • systemctl is used to examine and control the state of “systemd” system and service manager.

  • You can also list the files in /lib/systemd/system or /etc/systemd/system to view the unit files.

  • There's a command which displays the status of units at startup, and that is the systemctl list-unit-files command.

  • Difference between systemctl list-units and systemctl list-unit-files command?

    The difference between the "systemctl" or "systemctl list-units" commands and the "systemctl list-unit-files" command is that the first two commands give a run-time snapshot of units while the later command displays the status of units at startup.

  4.2 Nginx

  It is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability.

  Aim: Install nginx and verify that services is listed and enabled.

  • To install nginx run sudo apt install -y nginx command.
  • Verify that Nginx service is listed and enabled by executing the sudo systemctl list-unit-files | grep nginx command. You can see that the Nginx service is enabled.

  4.3 Logs

  Aim: Stop and start the nginx service. Check the logs for the nginx service using journalctl.

  • To start, stop, restart or check the status of nginx service:

    1. To check the status of let’s say nginx service(or unit, as systemd calls it), use sudo systemctl status nginx or sudo systemctl status nginx.service: As you can see that it is showing active(running) for the Active column. It means the process is running.

       systemctl status command also shows the most recent log entries at the bottom. In case you want to view even more log entries, then use the journalctl command.

       

    2. To stop the unit: systemctl stop nginx

       Just observe the status of the nginx service. It is showing inactive(dead) in the Active column. It means the process is stopped.

    3. To start the unit: systemctl start nginx
    4. To restart the unit : systemctl restart nginx

  • Check the logs for the nginx service using journalctl.

    journalctl allows you to discover the logging info about the service while systemctl can allow you to detect if the services started or stopped, or info about the service itself.
    journalctl -xe -> allows us to go through logs of all the services.

4.4 Git

Aim: Create a new directory and initialise it as a git repository. Add a file to be checked in and commit.

How to initialise a directory as a git repository

• Use the git init command.

• Let’s start by making a new directory and then making it a git repo

  1. Create a new directory named “newrepo”
  2. cd into it
  3. Execute the git init command to create an empty git repository.
  4. Run ls -l command to see that you now you have a .gitrepo
     

  What does .git folder contains?

• It serves as the “staging area” between the files you have on your file system and your commit history.
• When you run git add . command , the files from your current working directory are hashed and stored as objects in the index, leading them to be “staged changes”.

• Git repository when initialised doesn’t contain index folder. Index folder is created once you add one or more than file to the staging area.

  Add a file to be checked in and commit

• Create a file named file1.txt and check the status of the file. To check status run git status command.
• Add file to the staging area using git add .

• Commit the file with a message by executing git commit -m "message" command.

  More about Git

• You can also create a separate branch and work on it using the command git checkout -b branchName. Now, all the changes will be committed to this branch until you switch to another branch.

That's it for this blog. I know that it's a bit lengthy, but if you practice side by side then you will find it even more interesting.