Useful Docker tips and tricks
Some handy Docker tips in this post, check it out
By: Ajdin Imsirovic 21 February 2021
This article lists some experiences from learning and working with the Docker platform.
Image by Kinsey on Unsplash
1. Resolving error: Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?
Due to the fact that an app I was working on had a large number of containers running, and also due to the fact that these Docker containers were starting on boot up, this was significantly slowing my boot times and my machineās general performance. Since the machineās RAM was already maxed out, the only thing I could do was use systemctl
to prevent the auto-bootup of the said Docker containers. Which worked great, except when I wanted to boot them up manually, I got the error referenced in the above title:
Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?
The solution was to simply run the following command:
systemctl start docker
Once I did that, my Docker containers obeyed all other commands, which makes perfect sense, because the above command is the one that actually starts the docker daemon.
2. What does docker run
do?
The docker run
command starts a container:
- we give it an image name out of which to make a container instance
- we give it a process to run (aka āthe main processā)
When we run a container we can name it, or if we choose not to, Docker itself will assign a random name to a container.
Once the main process exits, the container is finished. Even if we have other processes started in the said container, when the main process finishes, the container finishes.
Hereās an example of a docker run
with the ubuntu
image and the echo
process:
docker run ubuntu echo 'Hello'
The above command will output Hello
, than the main process will finish and so the container will finish.
3. What does docker run --rm
do?
When we want to just run something in a container, then delete it when the process is finished, we run the docker run --rm
command.
Itās a one-liner for the following workflow:
docker run <container-name>
- *the container does its work and the main process finishes`
docker rm <container-name>
The docker run --rm
command is handy because itās a shortcut for the above process.
4. What does docker ps
do?
The docker ps
is one of the basic commands in Docker. It checks for running containers. The ps
command is short for āProcess Statusā. This command was borrowed in Docker from Linux operating system with the same meaning: āProcess Statusā.
5. Install Docker on Ubuntu 20.04
To install Docker on Ubuntu 20.04, you need to run the following commands:
sudo apt update
sudo apt upgrade
sudo apt install docker.io
sudo usermod -aG docker awv
docker --version
sudo docker run hello world
If everything works as expected, youāll get a nice Hello World message in the bash, as well as some explanations of all the steps that Docker took to run the Hello World image on your machine.
Hereās a sample output:
sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0a04fdcd24d3: Pull complete
Digest: sha256:a451b...d
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64)
3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
Now we can run docker ps
command to list running containers.
With docker ps -a
we can list all the containers.
6. Restart multiple containers on the command line easily using the $()
syntax
Hereās a quick use case. We have three docker containers running. We check their ids using:
docker ps -q
We get the following sample container ids:
abc123123123
def789789789
cde258258258
Now weād like to restart all three. We could copy paste their idās and type out a command like this:
docker restart abc123123123 def789789789 cde258258258
But we can do it better, using the $()
syntax:
docker restart $(docker ps -q)
The end result is still the same, but the second approach is much better because it saves time and is more generalized (weāre skipping hard-coded values).
7. Whatās a docker image?
A Docker image is a file which contains a ārecipeā for building a container (a fully functional, self-contained application, which utilizes the host OS kernel).
When we run Docker image, it can become one or multiple instances of that container.
A docker image is:
- immutable (can never change)
- shareable (with other people, e.g on Docker hub)
As an analogy using JavaScript, an image is a constructor function, and the container is an object instance of that constructor function.
To list docker images, use:
docker images
We get the output similar to this:
REPOSITORY TAG IMAGE ID CREATED SIZE
... ... ... ... ...
The dots above represent actual image data. Hereās what each column means:
- REPOSITORY: where the image came from
- TAG: version number
- IMAGE ID: internal docker id for this specific image
- CREATED:
- SIZE:
To refer to a specific image, use the combination:
<repository>:<tag>
Or use the image id.
The second approach is better because images donāt have to be named (but they must have the image id
).
8. How to run the bash app from a Docker image of containerized Ubuntu?
You can do it if the image is of an OS, like Ubuntu.
The docker run
command uses a specific image to build a container instance from that image:
docker run -ti <image> bash
The -ti
is an abbreviation for āterminal interactiveā.
To exit this imageās bash, just type:
exit
9. Format the output of the docker ps
command
This section is inspired by the Learning Docker course by Arthur Ulfeldt, available from Linkedin Learning.
To format the output of the docker ps
command, we use the --format
flag:
docker ps --format=<how-to-format>
For example, if we can replace the <how-to-format>
section with a variable. We can, for example, name that variable like this: $FORMAT
.
But how do we get the variable into bash?
We just write a simple shell script, like this:
export FORMAT="\nID\t\nIMAGE\t\nCOMMAND\t\nCREATED\t\nSTATUS\t\nPORTS\t\nNAMES\t\n"
# this shell will only work on bash
We can name the shell script, for example, reformat.sh
.
Now we need to make this new script executable. Like this:
chmod +x reformat.sh
Finally, we also need to run this script, like this:
sh reformat.sh
Alternatively, we could have just ran the code from the reformat.sh
script directly in the terminal, like this:
export FORMAT="\nID\t\nIMAGE\t\nCOMMAND\t\nCREATED\t\nSTATUS\t\nPORTS\t\nNAMES\t\n"
Now we have the $FORMAT
script available on our command line:
$FORMAT
The above command returns:
\nID\t\nIMAGE\t\nCOMMAND\t\nCREATED\t\nSTATUS\t\nPORTS\t\nNAMES\t\n
Now we can go back to the first command in this section:
docker ps --format=$FORMAT
10. What happens in a container instance, stays in a container instance
If we spin up two container instances from the same image of, say an Ubuntu OS, and then add a new file to one of these Ubuntu containers, that file is going to only be present in that one Ubuntu OS instance. The file wonāt be magically added to the image, nor will it be available in the other container instance.
11. The basic anatomy of docker commands
The way that we can write the docker commands in bash follows a simple pattern:
docker <option> <command> <arguments>
For a list of all commands we can use, just type docker
:
docker
To see the detailed description of each command, type:
docker <command> --help
For example:
docker run --help
The above command will give detailed info on the Dockerās run
command.
To get an overview of whatās happening with Docker on our machine, we can run the info
command:
docker info
12. Get a list of the most recently stopped containers
The last exited container can be found with:
docker ps -l
The output from the above command returns the following columns:
- ID
- IMAGE
- COMMAND
- CREATED
- STATUS
- PORTS
- NAMES
The STATUS
column gives us the reason for exit in round brankets, for example Exited (0)
, or Exited(127)
.
13. Make an image out of a container
We can do this using the docker commit
command.
So, with docker run
we make a new container from a docker image. Then we might make some changes inside the container instance. We can then commit those changes to a new image, using the docker commit
command.
This way, the immutability of docker images is preserved.
To make an image from the most recently stopped container, we need to get that containerās id, like this:
docker ps -l
Then, weāll copy the ID
keyās value, for example: 1234567890ab
.
Then weāll run the commit
command with the above ID provided:
docker commit 1234567890ab
The above command will produce a huge SHA256 hash, which is a unique ID of our new image. To make this more human-readable, we can use tags, like this:
docker tag <the-entire-copy-pasted-sha256-string> <tag-name>
For example:
docker tag 123abc234bcd... my-own-image
Docker will also give each of our containers a custom name, listed in the NAMES column, such as:
fervent_sammet
determined_ishizaka
romantic_mendel
amazing_zhukovsky
etc...
Because committing and tagging docker images is commonplace, thereās a handy shortcut command that automates this process:
docker commit <name> <tag>
For example, letās say we got this back from running docker ps -l
:
CONTAINER ID 12cd7701db78
IMAGE my-own-image
COMMAND "/hello"
CREATED 26 minutes ago
STATUS Exited (0) 26 minutes ago
PORTS
NAMES fervent_sammet
Now we can commit the above container as a new image, like this:
docker commit fervent_sammet my-second-image
Now if we inspected the available images with docker image
, weāll get the listing of the available images, with the first listed image being the most recently-added one; in our case, the my-second-image
image.
14. Find a Docker image on the Docker hub
You donāt even need to visit the website; everything can be done from the terminal, like this:
docker search ubuntu
The above command will return a list of images we can download. There are plenty of images there. Weāll just go with the first result, the ubuntu
image.
To actually download an image, we can do:
docker pull <image-name>
Thus:
docker pull ubuntu
Hereās the output:
Using default tag: latest
latest: Pulling from library/ubuntu
83ee3a23efb7: Pull complete
db98fc6f11f0: Pull complete
f611acd52c6c: Pull complete
Digest: sha256:123abc123abc...cba321
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest
15. What is a detached Docker container?
Detached mode is when a Docker container runs in the background of our terminal. It does not receive input or display output.
For example, we can run our newly downloaded ubuntu image in detached mode, as follows.
First, weāll locate it using the images
command:
docker images
Then, we can use the ID to run it, for example, our ubuntu
image id is: a6318a119b2a
, and thus weāll run:
docker run -d a6318a119b2a
Alternatively, we could use the human-friendly NAME parameter:
docker run -d ubuntu
Either way, weāll get back that huge SHA string so that we can identify the running container.
Now we can see the most recent running container with docker ps -l
:
docker ps -l
As expected, we get back all the data about that most recently ran container, which, indeed, is the one weāve just ran with the -d
flag, including the container name.
Now we can attach the currently detached container:
docker attach <container-name>
This way, Iāve gotten into the container that was previously detached.
Additionally, we can convert an attached container into a detached one, using a sequence of two keyboard shortcuts:
- CTRL + p
- CTRL + q
This way, we can exit the container *but the container is still running, detached**. This means that later on, we can attach one more time.
16. Set the maximum memory a container can use
To prevent from too much resources being used, we can run the --memory
flag, with options, for example, like this:
docker run --memory <max-allowed-memory> <image-name> <command>
A great feature is to assign a portion of the cpu-shares
, like this:
docker run --cpu-shares
ā¦the above command is a value that takes into account all the running containers, then assigns the appropriate amount based on the cpu shares. This is more flexible, because if we have two containers and one doesnāt take up any resources, the other will then be able to take more resources because thatās how we specified it should work.
Otherwise, we can set the --cpu-quota
flag to, say 20%, no more, no less, and this will always be the hard-coded value that weāll be using.
I like to think of it as seats on a bus. In the above scenarios, the former is like saying āspread to as many empty seats availableā, and the latter is like saying, āwhatever happens, only use your seatā.
17. How to run a process in a Docker container, then stop the running container
Hereās an example of running bash in our ubuntu
container.
docker run -d -ti ubuntu bash
The -ti
is an abbreviation for terminal interactive
.
We can then run the docker ps
command to get the info on the running container(s), then pick the container that was built using the ubuntu
image, and copy that containerās id. Once we have the id in the clipboard, we can run the following command to stop the currently running container:
docker stop a1b2c3d4e5a6b7
With the a1b2c3d4e5a6b7
SHA stub being the previously copied ID of the then-running container.
18. How to attach to a detached Docker container?
To see all the containers running (including detached ones), we run:
docker ps
Then, to attach a specific container, we run:
docker attach <container-name>
19. How to add another process to a running container in Docker?
We use this command:
docker exec
Itās useful for debugging and DB administration.
20. Giving a name to a running container, on-the-fly
If we want to specify a name to a container, at the time we run it, we can use the following command:
docker run --name <our-custom-name> -d <container-image>
For example, we can run our ubuntu image in a container instance we named testname
:
docker run --name testname -d ubuntu
21. Inspecting Docker logs to find data about errors in a container
Sometimes we can start a container but it doesnāt do what we expect. We can use the docker logs
command to inspect the logs of that container by passing the container NAME, like this:
docker logs <container-name>
An example of passing a custom name and a wrong command (so that we can inspect the logs):
docker run --name testname -d ubuntu bash -c "ech osomething"
In the above command, weāre running the ubuntu
image:
- with the custom name of
testname
- detached (specified with
-d
) so that it just runs in the background
Additionally, weāve passed it the process to run, bash
, and weāre running this command: ech osomething
. The command was āannouncedā using the -c
flag.
Obviously, the above command is wrong: we typed ech osomething
but we should have typed echo something
. The above command wonāt run, because the ech
command, contrary to the echo
command, doesnāt exist.
Letās inspect what happend to make sure:
docker logs testname
Indeed, the ech
command was the problem; itās confirmed by the output:
bash: ech: command not found
Hereās an important note: we need to keep our docker logs slim. If thereās too much stuff being output to the logs, we can slow down docker, even so much that itās simply unusable.
22. Stop a container with the kill
command
Using the kill
command moves the container to a stopped state.
Any stopped container can be removed with the rm
command.
Thus:
docker kill <container-name>
docker rm <container-name>
For example:
docker run ubuntu
We get the containerās NAME, in our case it was lucid_newton, so:
docker kill lucid_newton
That stopped our container.
Now, if we inspect the most recent container that was stopped, using:
docker ps -l
⦠weād get back that indeed, it was lucid_newton, plus the STATUS is: Exited (137) 26 seconds ago.
Now we can remove it:
docker rm lucid_newton
This will just output the affected containerās name:
lucid_newton
Thatās it, the container is removed, and the luci_newton name is now again freely available. Otherwise, if we wanted to use the same name sometimes in the future, weād get a notification saying āthat name is already in useā>
23. Make sure your container includes your dependencies (not downloads them at run time)
For example, if weāre using containerized Node.js, and we start running it, and itās set up so that it downloads its dependecies on start, this can work fine, untilā¦
For example, a library gets removed from the Node.js repositories, and then our container is broken, and canāt run.
24. The importance of naming Docker containers
We can have containers without NAME value specified.
We can also keep some important stuff in such containers.
Donāt do it!
Because, during container clean-up, itās easier to erase a container that is āunlabelledā, and thus accidentally erase some work that you shouldāve kept.
25. Using container ports
Container ports have to do with container networking.
Containers, by default, are set up initially so that they canāt access the internet.
We can group multiple containers into separate local networks, and we can specify exactly how they get connected to each other.
To connect to the internet, we can also explicitly expose a port (aka āpublishā a port).
We can expose a port by setting up:
- the internal port that a containerized software is listening on
- the āoutsideā port that a container is listening on
- the protocol to use (optional)
Hereās an example of exposing the exact same port on the inside and the outside of a container:
-p 34567:34567
We can expose more than one port, by simply specifying another one, like this:
-p 34567:34567 -p 34566:34566 -p 34565:34565
Above, weāve just specified three inner ports matching three outter ports.
Hereās a command sending data between three different bash instances on the same machine. The first one is the āserverā:
docker run --rm -ti -p 34567:34567 -p 34568:L34568 --name the-server ubuntu bash
If we ran docker run --help
, weād find in the output instructions that:
- the
--rm
flag automatically removes the container when it exits, - the
-ti
:-t : Allocate a pseudo-tty
-i : Keep STDIN open even if not attached
- the
-p
, as just described:-p, --publish list : Publish a container's port(s) to the host
- the
--name
:--name string : Assign a name to the container
Now we can run in the same bash instance, the netcat program, with the nc
command:
nc -lp 34567 | nc -lp 34568
So weāre listening with one nc
instance on port 34567, and weāre piping that to another nc
instance on port 34568.
In the second bash instance, weāll run nc localhost 34567
, and in the third, weāll run nc localhost 34568
, so that, when we type a āmessageā in the second bash instance, weāll get that āmessageā in the third bash instance. The āmessageā is just some text that we type and that gets passed through by the netcat utility.
To reiterate, weāll have 3 separate bash instances open. The first one is a server, so we type:
docker run --rm -ti -p 34567:34567 -p 34568:34568 --name the-server ubuntu
Then the the-server
containerās bash will open:
nc -lp 34567 | nc -lp 34568
But, the nc
is not installed on the system, so:
bash: nc: command not found
bash: nc: command not found
This is an opportunity to containerize an app (or a utility in our case).
To do that with our netcat, weāll just run a docker container that has it installed.
26. How to expose ports dynamically?
The port inside the container is fixed.
The port on the host is chosen from the unused ports.
This allows many containers running programs with fixed ports.
This is often used with a service discovery program.
For example, I can specify only the port inside the container, and let docker choose the port on the outside:
docker run --rm -ti -p 34567 -p 34568 --name the-server ubuntu:14.04 bash
Now weāre inside the the-server
container, and we can run nc
:
nc -lp 34567 | nc -lp 34568
Now, in another bash instance, we can check for the the-server
containerās external port:
docker port the-server
Hereās a sample output:
34567/tcp -> 0.0.0.0:32777
34568/tcp -> 0.0.0.0:32776
Now in the other two bash instances, I can run the exernal ports that Docker picked dynamically:
nc localhost 32777
ā¦and in the other instance:
nc localhost 32776
27. Itās enough to run the docker rm
command with the first 4 characters of the id
Itās as simple as, for example, this:
docker rm 76f9
This is great for a quick cleanup.
28. Containers shouldnāt directly refer to a container by an IP address
So how can containers āaddressā the container that is hosting them?
By using the containerās host name: host.docker.internal
.
29. In Docker, can we use ports with udp protocol instead of the tcp protocol?
Yes.
We specify them with a slash and the port on the end of the command:
docker run -p <outside-port>:<inside-port>/<protocol-tcp-or-udp)>
For example:
docker run -p 23456:23456/udp
Practically, in one bash instance, weāll run this:
docker run --rm -ti 34567/udp --name the-server ubuntu:14.04 bash
And then in another container we check the port:
docker port the-server # returns, e.g: 34567/udp -> 0.0.0.0:32774
Back in the first one, once inside the container, weāll run:
nc -ulp 34567
Back in the second:
Hello from second
The message from the second is now passed in to first:
nc -ulp 34567
Hello from second
30. Networking between containers and inspecting Docker networks defaults
There are a number of ways and options of how our Docker containers connect to one another.
For example, once we exposed a port of a container, this opens a network path from the outside of that computer to that container.
Other containers are able to connect to it by going out to the host and then back in along the said network path.
To inspect Docker networks, run:
docker network ls
This is a sample output:
NETWORK ID NAME DRIVER SCOPE
abdcabc5fd59 bridge bridge local
1dca4d941c2f host host local
f450fbf5a6c4 none null local
Bridge is the network used by containers that donāt specify a preference to put into any other network.
Host is when you want a container to have no isolation at all. This does have some security concerns.
And none is for when a container should have no networking.
31. Build a new Docker network
Hereās the command: docker network create <network-name>
, e.g:
docker network create my-network
To run a machine on our Docker network, weāll run:
docker run --rm -ti --net my-network
We can also give our machine a name, by appending the --name <name-we-choose>
to the above command, like this:
docker run --rm -ti --net my-network --name my-server
Names are very useful when using private networks in Docker because different containers inside the network can refer to each other by those names, so it makes it very easy for them to find each other.
Now that we have the name, we can run the rest, namely ubuntu:14.04 bash
, so that the entire command now looks like this:
docker run --rm -ti --net my-network --name my-server ubuntu:14.04 bash
Now we can verify that we can send traffic to ourselves:
ping my-server
Indeed, we can, which is proven by the output we get:
PING my-server (172.18.0.2) 56(84) bytes of data.
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=3 ttl=64 time=0.064 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=4 ttl=64 time=0.063 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=5 ttl=64 time=0.062 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=6 ttl=64 time=0.063 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=7 ttl=64 time=0.064 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=8 ttl=64 time=0.063 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=9 ttl=64 time=0.036 ms
64 bytes from 31a3b94a4ee1 (172.18.0.2): icmp_seq=10 ttl=64 time=0.064 ms
^C
--- my-server ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9224ms
rtt min/avg/max/mdev = 0.036/0.058/0.064/0.011 ms
If we try to ping a non-existing server, nothing will happen. So letā open another bash instance and build another custom server:
docker run --rm -ti --net my-network --name my-second-server ubuntu:14.04 bash
Iāve put my-second-server
on the same network so that my-server
and my-second-server
can find each other.
Now in my second server, I can write:
nc -lp 1234
And now I can send a message to my-server
:
hello from my-second-server
Back in my-server
, I can received the messages from my-second-server
, and I can send a message to my-second-server
:
my-second-server 1234
hi from my-server
Our two containers communicate both ways, and are on the same network.
32. Communicating between Docker networks
Iām going to start up a new terminal here.
Letās make another network.
docker network create catsonly
This is a network only for cats.
Okay, now letās go ahead and put the cat machine onto the cat network.
docker network connect catsonly catserver
Okay, now, over here, letās make things a little more interesting. Iāll start up another terminal, and Iām going to start the new member of our network.
docker run --rm -ti --net catsonly --name bobcatserver bash
Itās like a catserver, only moreso, running bash.
Ok, so from here, I can do pink catserver
, and get data. If I do ping dogeserver
, traffic is not allowed.
From the catserver in the middle, I can ping dogserver, hit CTRL + c, and it works.
And bobcatserver.
Now over here, on the dogserver, we can still only make connections to catserver, but we cannot connect to babcatserver, because the dogserver is not on the catsonly network. There are many, many more options to this that give us fine-grain control over how things connect here. But what was covered here is about 90% of the user cases. The rest will be left up to experience.
33. What are Dockerfiles?
Itās just a file used to build a Docker image.
We run it with docker build
and the -t
tag for ātagā. The below command builds a new Docker image and tags it with the <name-tag>
, in the current folder.
docker build -t <name-tag> .