diff --git a/slides/intro-fullday.yml b/slides/intro-fullday.yml index 8aebbc93..4a10d87e 100644 --- a/slides/intro-fullday.yml +++ b/slides/intro-fullday.yml @@ -49,7 +49,8 @@ chapters: - intro/Advanced_Dockerfiles.md - intro/Application_Configuration.md - intro/Logging.md -- - intro/Namespaces_Cgroups.md +- - intro/Resource_Limits.md + - intro/Namespaces_Cgroups.md - intro/Copy_On_Write.md #- intro/Containers_From_Scratch.md - intro/Container_Engines.md diff --git a/slides/intro-selfpaced.yml b/slides/intro-selfpaced.yml index 5637cdc8..06ac2353 100644 --- a/slides/intro-selfpaced.yml +++ b/slides/intro-selfpaced.yml @@ -49,7 +49,8 @@ chapters: - intro/Advanced_Dockerfiles.md - intro/Application_Configuration.md - intro/Logging.md -- - intro/Namespaces_Cgroups.md +- - intro/Resource_Limits.md + - intro/Namespaces_Cgroups.md - intro/Copy_On_Write.md #- intro/Containers_From_Scratch.md - intro/Container_Engines.md diff --git a/slides/intro/Resource_Limits.md b/slides/intro/Resource_Limits.md new file mode 100644 index 00000000..c74d2ebe --- /dev/null +++ b/slides/intro/Resource_Limits.md @@ -0,0 +1,229 @@ +# Limiting resources + +- So far, we have used containers as convenient units of deployment. + +- What happens when a container tries to use more resources than available? + + (RAM, CPU, disk usage, disk and network I/O...) + +- What happens when multiple containers compete for the same resource? + +- Can we limit resources available to a container? + + (Spoiler alert: yes!) + +--- + +## Container processes are normal processes + +- Containers are closer to "fancy processes" than to "lightweight VMs". + +- A process running in a container is, in fact, a process running on the host. + +- Let's look at the output of `ps` on a container host running 3 containers : + + ``` + 0 2662 0.2 0.3 /usr/bin/dockerd -H fd:// + 0 2766 0.1 0.1 \_ docker-containerd --config /var/run/docker/containe + 0 23479 0.0 0.0 \_ docker-containerd-shim -namespace moby -workdir + 0 23497 0.0 0.0 | \_ `nginx`: master process nginx -g daemon off; + 101 23543 0.0 0.0 | \_ `nginx`: worker process + 0 23565 0.0 0.0 \_ docker-containerd-shim -namespace moby -workdir + 102 23584 9.4 11.3 | \_ `/docker-java-home/jre/bin/java` -Xms2g -Xmx2 + 0 23707 0.0 0.0 \_ docker-containerd-shim -namespace moby -workdir + 0 23725 0.0 0.0 \_ `/bin/sh` + ``` + +- The highlighted processes are containerized processes. +
+ (That host is running nginx, elasticsearch, and alpine.) + +--- + +## By default: nothing changes + +- What happens when a process uses too much memory on a Linux system? + +-- + +- Simplified answer: + + - swap is used (if available); + + - if there is not enough swap space, eventually, the out-of-memory killer is invoked; + + - the OOM killer uses heuristics to kill processes; + + - sometimes, it kills an unrelated process. + +-- + +- What happens when a container uses too much memory? + +- The same thing! + + (i.e., a process eventually gets killed, possibly in another container.) + +--- + +## Limiting container resources + +- The Linux kernel offers rich mechanisms to limit container resources. + +- For memory usage, the mechanism is part of the *cgroup* subsystem. + +- This subsystem allows to limit the memory for a process or a group of processes. + +- A container engine leverages these mechanisms to limit memory for a container. + +- The out-of-memory killer has a new behavior: + + - it runs when a container exceeds its allowed memory usage, + + - in that case, it only kills processes in that container. + +--- + +## Limiting memory in practice + +- The Docker Engine offers multiple flags to limit memory usage. + +- The two most useful ones are `--memory` and `--memory-swap`. + +- `--memory` limits the amount of physical RAM used by a container. + +- `--memory-swap` limits the total amount (RAM+swap) used by a container. + +- The memory limit can be expressed in bytes, or with a unit suffix. + + (e.g.: `--memory 100m` = 100 megabytes.) + +- We will see two strategies: limiting RAM usage, or limiting both + +--- + +## Limiting RAM usage + +Example: + +```bash +docker run -ti --memory 100m python +``` + +If the container tries to use more than 100 MB of RAM, *and* swap is available: + +- the container will not be killed, + +- memory above 100 MB will be swapped out, + +- in most cases, the app in the container will be slowed down (a lot). + +If we run out of swap, the global OOM killer still intervenes. + +--- + +## Limiting both RAM and swap usage + +Example: + +```bash +docker run -ti --memory 100m --memory-swap 100m python +``` + +If the container tries to use more than 100 MB of memory, it is killed. + +On the other hand, the application will never be slowed down because of swap. + +--- + +## When to pick which strategy? + +- Stateful services (like databases) will lose or corrupt data when killed + +- Allow them to use swap space, but monitor swap usage + +- Stateless services can usually be killed with little impact + +- Limit their mem+swap usage, but monitor if they get killed + +- Ultimately, this is no different from "do I want swap, and how much?" + +--- + +## Limiting CPU usage + +- There are no less than 3 ways to limit CPU usage: + + - setting a relative priority with `--cpu-shares`, + + - setting a CPU% limit with `--cpus`, + + - pinning a container to specific CPUs with `--cpuset-cpus`. + +- They can be used separately or together. + +--- + +## Setting relative priority + +- Each container has a relative priority used by the Linux scheduler. + +- By default, this priority is 1024. + +- As long as CPU usage is not maxed out, this has no effect. + +- When CPU usage is maxed out, each container receives CPU cycles in proportion of its relative priority. + +- In other words: a container with `--cpu-shares 2048` will receive twice as much than the default. + +--- + +## Setting a CPU% limit + +- This setting will make sure that a container doesn't use more than a given % of CPU. + +- The value is expressed in CPUs; therefore: + + `--cpus 0.1` means 10% of one CPU, + + `--cpus 1.0` means 100% of one whole CPU, + + `--cpus 10.0` means 10 entire CPUs. + +--- + +## Pinning containers to CPUs + +- On multi-core machines, it is possible to restrict the execution on a set of CPUs. + +- Examples: + + `--cpuset-cpus 0` forces the container to run on CPU 0; + + `--cpuset-cpus 3,5,7` restricts the container to CPUs 3, 5, 7; + + `--cpuset-cpus 0-3,8-11` restricts the container to CPUs 0, 1, 2, 3, 8, 9, 10, 11. + +- This will not reserve the corresponding CPUs! + + (They might still be used by other containers, or uncontainerized processes.) + +--- + +## Limiting disk usage + +- Most storage drivers do not support limiting the disk usage of containers. + + (With the exception of devicemapper, but the limit cannot be set easily.) + +- This means that a single container could exhaust disk space for everyone. + +- In practice, however, this is not a concern, because: + + - data files (for stateful services) should reside on volumes, + + - assets (e.g. images, user-generated content...) should reside on object stores or on volume, + + - logs are written on standard output and gathered by the container engine. + +- Container disk usage can be audited with `docker ps -s` and `docker diff`.