Docker Volume and Bind mounts

Introduction

Once you remove a container all the data that is present in the container's writable layer also gets deleted.

But what if we want container's data(all or only some of it) to persist even after the removal of the container? Also, what if I want two or more containers to share data among themselves?

Here comes into the picture volume and bind mounts. There's more to the story though. Let's dive into it.

Types of mounts

There are different types of mounts - volume, bind mounts, tmpfs mounts and named pipes.

But the volume and bind mounts allow containers to store files on the host machine, so that the files are persisted even after the container stops.

tmpfs mount is used to store non-persistent state data and the data doesn't persist once the container is removed.

I would like to ignore the existence of named pipes. It's not relevant to this blog.

The location where the data is stored in case of each mounts differs, as you can see in the image below.

1. Volumes

Documentation states that -

Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem. Volumes are the best way to persist data in Docker.

• Use cases: You can use volume in the following scenarios (from the documentation) -

  • When multiple running containers are sharing the same data.

  • Since the shared volume data is stored inside the volumes folder of the docker directory (i.e. /var/lib/docker/volumes) and not in the container's writable layer (i.e. /var/lib/docker/overlay2), the data is not lost even if the container is removed.

    So, if you want to persist your data even after the removal of container then go for volumes.

  • When you want to restore, backup and migrate your container's data from one Docker host to another.
  • When you want to store container's data to a remote host or a cloud service provider.
  • When your application requires high-performance I/O on Docker host. Volumes are stored in the Linux VM, which means that the reads and writes have much lower latency and higher throughput.

2. Bind Mounts

Let's see what the documentation has to say about it:

Bind mounts may be stored anywhere on the host system. They may even be important system files or directories. Non-Docker processes on the Docker host or a Docker container can modify them at any time.

• Use cases: Use bind mounts -

  • When you want to share the configuration files from the host machine to containers.
  • When the file or directory structure of the Docker host is consistent with the bind mounts the containers require.

  Read more about the use cases of bind mounts.

3. tmpfs mounts

Let's read what the documentation has to state:

tmpfs mounts are stored in the host system’s memory only, and are never written to the host system’s filesystem.

When the container stops, the tmpfs mount is removed, and files written there won’t be persisted.

• You can’t share tmpfs mounts between containers.

• Use cases: Use tmpfs mounts -

  • For cases when you do not want the data to persist either on the host machine or within the container.

  • This may be for security reasons or to protect the performance of the container when your application needs to write a large volume of non-persistent state data.

Difference between volume and bind mounts

1. You can use the Docker CLI to manage volumes, for example, retrieving the list of volumes or removing unused volumes. But, you can’t use Docker CLI commands to directly manage bind mounts. Try it out!

2. A bind mount can be accessed and modified by processes outside Docker. This can be a benefit when you want to integrate Docker with other processes, but as you can guess already that security gets compromised.

3. In case of bind mounts, you can change host file system via processes running in a container, including creating, modifying, or deleting important system files or directories. Again, security is at stake. Non-docker processes can be affected by this.

   Docker volume are somewhat secure in the sense that these volumes are isolated from the core functionality of the host machine.

4. Refer to below resources to learn more about their differences:

   • Resource 1
   • Resource 2
   • Resource 3

Create a volume

There are various ways using which you can create volume:

1. Using create command:

    docker volume create <volumeName>

• To create a volume named, let's say home just run the command:

  docker volume create home
  

  As you can see below that volume named home is created.

List Volumes

To list volume just execute this command:

docker volume ls

Remove Volumes

We have to run commands to delete a volume.

1. To remove one or more volume that is no longer in use, run this command:

   docker volume rm <volumeName>
   

   • Let's say I want to remove a volume named office then all I have to do is execute the below command:

     docker volume rm office
     

     

2. To remove all unused volumes just run below command:

   docker volume prune
   

   

Note:- If you try to remove a volume that is already in use(one or more container is using it) then it will display the following error: It is stating that the volume named 98b15559add3bb43cd02a9c9c757785e72db07ff1d344c1fa4695596a24010c4 can't be removed as it is being used by a container having container ID b66e5a2379cfbf339703b3631e7b8b6a3dd5ffb02038882007819290a3a58583

Inspect Volume

To display information on one or more volumes run this command:

docker volume inspect <volName>

• Let's say I want to display information of volume named home.

  docker volume inspect home
  

  

• Use -f flag to format the output. E.g., let's say I want to display only the mountpoint of volume home .

   docker volume inspect -f '{{ .Mountpoint }}' home
  

  

Start a container with a volume

There are various approaches:

1. using -v flag,
2. using --mount flag,
3. using DockerFile
4. using Docker compose

I'm going to cover only 1st and 3rd approach. Refer to the documentation to learn more about other approaches.

1. Using -v flag

 docker run -itd -v a:b:c --name <containerName> <imageName> bash
 - a : name of the volume
 - b : path where the file or directory is mounted in the container.
 - c : optional, and is a comma-separated list of options, such as "ro" (readonly)

• For anonymous volumes, the first field is omitted:

  i. Example : Here, I ran a container named test using ubuntu image. As we are not specifying the volume name here, therefore, Docker will create a volume and assign a unique random name to it. This volume will then be mounted into the directory named sharedVol within the container.

  docker run -itd -v sharedVol --name test ubuntu bash
  

  

  • test container created. Let's get into it.

  • cd into the sharedVol and make some changes in it. I created a file named names.txt and added some data into it.

• For named volumes:

  i. Example : Let's say I want to create a container named babyYoda and mount the galaxy volume into the /data directory within this new container.

  docker run -itd -v galaxy:/data --name babyYoda ubuntu bash
  
  - galaxy -> volume
  - Don't forget to use ':'
  - /data -> absolute path of the directory that you want to mount your volume on.
  

  In case volume (i.e. galaxy) doesn't already exist, Docker will create it.

• Inspect babyYoda container by running docker inspect babyYoda command, and you'll find out that the mount type is volume.
• You can also use --mount option. I'll not be covering it in here. Read the documentation to gain more insight on this one.

2. Using Dockerfile

• First create a file named Dockerfile.

• Edit Dockerfile. In the VOLUME columns mention the absolute path of a folder named volume1 where an anonymous volume will be mounted within the container. To mount a named volume, let's say namedVol , just write namedVol:/volume1. Save the file and come out of the file editor.

• Now, build the image and name it “test-image” using docker build -t test-image . command.

• Create a container named “test-container” using the “test-image”.

• Enter the container and list the contents. As you can see below that the volume1 folder is present.

Tip!

I would like you to read this really good article on "How to share data between docker containers". I am sharing a snippet of it below:

Note: The -v flag is very flexible. It can bind mount or name a volume with just a slight adjustment in syntax.

If the first argument begins with a / or ~/ you’re creating a bind mount. Remove that, and you’re naming the volume. For example:

• -v /path:/path/in/container mounts the host directory, /path at the /path/in/container
• -v path:/path/in/container creates a volume named path with no relationship to the host.

It is a minute thing but it can create a lot of confusion. Now that you know what is what (if that makes sense), you'll understand things better.

Start a container with a bind mount

There are various approaches just like volumes - using -v, --mount, Dockerfile and Docker compose. I'll only discuss about bind mount using -v flag.

Using -v flag

 docker run -itd -v a:b:c --name <containerName> <imageName> bash
 - a : path to the file or directory on the host machine.
 - b : path where the file or directory is mounted in the container.
 - c : optional, and is a comma-separated list of options, such as 
           1. ro (readonly), 
           2. z (content is shared among multiple containers), and 
           3. Z (content is private and unshared)

Example : Let's bind-mount hostTarget directory present in the host's current working directory into our container at /app. Let's name the container devTest. Run the command from within the source directory.

docker run -itd -v $(pwd)/hostTarget:/app --name devTest nginx 

  - Instead of using $(pwd) you can also specify the absolute path.

Let's have a look:

• I ran the container, entered into it, and created a hello.txt file inside the /app directory.
• As /app is bind mounted to /hostTarget folder present in the host OS, therefore the file hello.txt will also be visible in the hostTarget folder.
• As bind mounts are not directly managed by Docker, therefore, we can't display it using docker volume ls:
• Inspect the devTest container by executing docker inspect devTest and check for the Mounts. You'll see that the mount type is bind.

Location of created volumes and what's inside it

Location

Now, let's see where the volumes that we just created are located.

• They are stored in /var/lib/docker/volumes. Volumes that are managed by Docker will be present here, storing all the persistent data that the mounted folders of the containers have.
• home shared volume got created when we ran the docker volume create home command.
• 366991dafb9a70b1e07d7286b8c0136be4c88b6614512206fbc355f40d9468c0 volume was created when we ran docker run -itd -v sharedVol --name test ubuntu bash command. This is the volume that the test container make changes to via sharedVol folder i.e. present inside the container.
• Shared volume folder named 80615a4b202691e640f9ae900a926a0f6fe1e87226479173e6a5208d6a49519d was created using DockerFile.
• The above two volumes have these long weird alphanumeric names. Well, that's because we didn't specify the volume name while creating the container. We only mentioned the folder within the container on which this volume has to be mounted . Docker then creates volumes with random unique name and then mounts it to the container's volume folder that you specified.

Sneak Peek

• Now we will check the contents of the 366991dafb9a70b1e07d7286b8c0136be4c88b6614512206fbc355f40d9468c0 shared volume folder.

• It is showing us the _data folder which will store all the data. Let's see what's in it.

• See, that's what were talking about. We made changes in the sharedVol folder of the test container but these changes were made to the volume that we mounted. Even if we remove the test container the contents of the sharedVol aren't going to get lost. Everything is being saved in the /var/lib/docker/volumes/366991dafb9a70b1e07d7286b8c0136be4c88b6614512206fbc355f40d9468c0/_data folder.

  This way we can make our container data persist even if the container is removed.

Share volume between two containers

We already have a container named babyYoda and it also has a volume mounted into it's /data directory. Now, I want to share this volume with another container named lukeSkywalker. I can share it while creating the lukeSkywalker container only.

So, here we go.

docker run -it --volumes-from babyYoda --name lukeSkywalker ubuntu bash

  >  --volumes-from -> use this flag and then enter the container name whose

• You will be inside the container after running the above command. As you can see that the data folder is also present inside the container lukeSkywalker.
• The /data folder is empty as of now. Make some changes to it. I've create hello.txt file and entered some data to it.
• Now exit out of this container and enter into the babyYoda container's /data directory to check whether the changes made by lukeSkywalker are visible to babyYoda container or not. As you can see, changes are visible. Congrats! You just learnt how to share volume between containers.
• You can also share volumes from multiple containers.
• You can also use Dockerfile and Docker-compose to share volume between containers. Go and try it out.

Share volume between host and container

To share volume between the host and container use the bind-mount.

Overview

In this blog we learnt about volume and bind mounts. We got to know how can we use them to share data among multiple containers, or how to share volumes between host and a container.

We covered how to create a volume, inspect it and delete it. We learnt a little bit about tmpfs mount also. We also learnt when to use different types of mounts.

That's it for this blog. Thanks for reading it. :)