From 7281ca3ca013762413703a48a276807ac64a4a58 Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Fri, 6 Dec 2019 14:16:48 -0600 Subject: [PATCH] Tweak content for inclusion in master branch --- slides/containers/Init_Systems.md | 137 ++++++++++++++++++++++++++++++ slides/containers/Init_systems.md | 79 ----------------- slides/intro-fullday.yml | 2 +- slides/intro-selfpaced.yml | 1 + 4 files changed, 139 insertions(+), 80 deletions(-) create mode 100644 slides/containers/Init_Systems.md delete mode 100644 slides/containers/Init_systems.md diff --git a/slides/containers/Init_Systems.md b/slides/containers/Init_Systems.md new file mode 100644 index 00000000..510cdbe8 --- /dev/null +++ b/slides/containers/Init_Systems.md @@ -0,0 +1,137 @@ +# Init systems and PID 1 + +In this chapter, we will consider: + +- the role of PID 1 in the world of Docker, + +- how to avoid some common pitfalls due to the misuse of init systems. + +--- + +## What's an init system? + +- On UNIX, the "init system" (or "init" in short) is PID 1. + +- It is the first process started by the kernel when the system starts. + +- It has multiple responsibilities: + + - start every other process on the machine, + + - reap orphaned zombie processes. + +--- + +class: extra-details + +## Orphaned zombie processes ?!? + +- When a process exits (or "dies"), it becomes a "zombie". + + (Zombie processes show up in `ps` or `top` with the status code `Z`.) + +- Its parent process must *reap* the zombie process. + + (This is done by calling `waitpid()` to retrieve the process' exit status.) + +- When a process exits, if it has child processes, these processes are "orphaned." + +- They are then re-parented to PID 1, init. + +- Init therefore needs to take care of these orphaned processes when they exit. + +--- + +## Don't use init systems in containers + +- It's often tempting to use an init system or a process manager. + + (Examples: *systemd*, *supervisord*...) + +- Our containers are then called "system containers". + + (By contrast with "application containers".) + +- "System containers" are similar to lightweight virtual machines. + +- They have multiple downsides: + + - when starting multiple processes, their logs get mixed on stdout, + + - if the application process dies, the container engine doesn't see it. + +- Overall, they make it harder to operate troubleshoot containerized apps. + +--- + +## Exceptions and workarounds + +- Sometimes, it's convenient to run a real init system like *systemd*. + + (Example: a CI system whose goal is precisely to test an init script or unit file.) + +- If we need to run multiple processes: can we use multiple containers? + + (Example: [this Compose file](https://github.com/jpetazzo/container.training/blob/master/compose/simple-k8s-control-plane/docker-compose.yaml) runs multiple processes together.) + +- When deploying with Kubernetes: + + - a container belong to a pod, + + - a pod can have multiple containers. + +--- + +## What about these zombie processes? + +- Our application runs as PID 1 in the container. + +- Our application may or may not be designed to reap zombie processes. + +- If our application uses subprocesses and doesn't reap them ... + + ... this can lead to PID exhaustion! + + (Or, more realistically, to a confusing herd of zombie processes.) + +- How can we solve this? + +--- + +## Tini to the rescue + +- Docker can automatically provide a minimal `init` process. + +- This is enabled with `docker run --init ...` + +- It uses a small init system ([tini](https://github.com/krallin/tini)) as PID 1: + + - it reaps zombies, + + - it forwards signals, + + - it exits when the child exits. + +- It is totally transparent to our application. + +- We should use it if our application creates subprocess but doesn't reap them. + +--- + +class: extra-details + +## What about Kubernetes? + +- Kubernetes does not expose that `--init` option. + +- However, we can achieve the same result with [Process Namespace Sharing](https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/). + +- When Process Namespace Sharing is enabled, PID 1 will be `pause`. + +- That `pause` process takes care of reaping zombies. + +- Process Namespace Sharing is available since Kubernetes 1.16. + +- If you're using an older version of Kubernetes ... + + ... you might have to add `tini` explicitly to your Docker image. diff --git a/slides/containers/Init_systems.md b/slides/containers/Init_systems.md deleted file mode 100644 index 60c9da22..00000000 --- a/slides/containers/Init_systems.md +++ /dev/null @@ -1,79 +0,0 @@ -# Init-systems and PID 1 - -In this chapter, we will consider the role of PID 1 in the world of Docker, - -and how to avoid some common pitfalls due to the misuse of init-systems. - ---- -## Don't use init-systems - -- It's often tempting to use init-systems (*systemd*, *supervisord*) - - and use docker as a "lightweight VM" - -- This often a bad idea, as it make things harder to debug: - - - *example 1*: if you start a container changing it's entrypoint to a shell, - - how to easily start the original process ? - - - *example 2*: if you run multiple process, logs are mixed to stdout - - - *example 3*: you're process is dying but you're init process is not - - => the container is running for nothing - ---- -## Don't use init-systems, but ... - -- In UNIX, a dead child process still use a PID till it's parent read it's status - -- In the meantime of being read by it's parent, - - those process are called `Zombie` or `defunct` process - -- If not being ripped off, zombie processes could crash a server (PID exhaution) - -- If the parent also dies before reading it's child container the zombie are attach to the PID 1 in some cases. - -- On a VM or real system, one of the role of the PID 1(Init-systems) is to rip zombies. - - *This also apply to containers* - ---- -## Use an init - -- You're application is running as PID 1 in the docker container - -- You're application is surely not designed to read status of random attaching child - -- Then everything is blowing up due to PID exhaution - - => Docker now has a built-in init you can enable `docker run --init` - -- This is a small init-system([tini](https://github.com/krallin/tini)) that takes the role of PID 1 - -- Only rips zombies, completly transparent otherwise - - (forwards signals, exit when child exit, etc). - -- Orchestrators like kubernetes has no option to turn `--init` when running container, - - so you might want to add explicitly to you're docker image, and use it as entrypoint - ---- -## Use it or not ? - -- Sometimes it's also handy to run a full init-system like *systemd*: - - - In CI when you're goal is exactly to test an init-script or a unit-file. - -- You might think, if it's ok for *systemd*, this is surely ok for *supervisord* - - especially running multiple times the same process (then, mixed logs is not a big deal) - - => I would strongly *NOT* recommand to do so. - -- It's often design to restart unhealthy process automatically - - and thus masquerade things to the operator or to the orchestrator (whose role is identical) diff --git a/slides/intro-fullday.yml b/slides/intro-fullday.yml index 65b6beb6..7daec807 100644 --- a/slides/intro-fullday.yml +++ b/slides/intro-fullday.yml @@ -60,7 +60,7 @@ chapters: - - containers/Docker_Machine.md - containers/Advanced_Dockerfiles.md - - containers/Init_systems.md + - containers/Init_Systems.md - containers/Application_Configuration.md - containers/Logging.md - diff --git a/slides/intro-selfpaced.yml b/slides/intro-selfpaced.yml index 14c494b3..07a5bca6 100644 --- a/slides/intro-selfpaced.yml +++ b/slides/intro-selfpaced.yml @@ -52,6 +52,7 @@ chapters: - containers/Exercise_Composefile.md - containers/Docker_Machine.md - - containers/Advanced_Dockerfiles.md + - containers/Init_Systems.md - containers/Application_Configuration.md - containers/Logging.md - containers/Resource_Limits.md