mirror of
https://github.com/jpetazzo/container.training.git
synced 2026-09-06 03:07:15 +00:00
Merge branch 'master' of https://github.com/jpetazzo/container.training
This commit is contained in:
Bridget Kromhout
@@ -292,15 +292,31 @@ If there is a bug and you can't even reproduce it:
|
||||
sorry. It is probably an Heisenbug. We can't act on it
|
||||
until it's reproducible, alas.
|
||||
|
||||
If you have attended this workshop and have feedback,
|
||||
or if you want somebody to deliver that workshop at your
|
||||
conference or for your company: you can contact one of us!
|
||||
|
||||
# “Please teach us!”
|
||||
|
||||
If you have attended one of these workshops, and want
|
||||
your team or organization to attend a similar one, you
|
||||
can look at the list of upcoming events on
|
||||
http://container.training/.
|
||||
|
||||
You are also welcome to reuse these materials to run
|
||||
your own workshop, for your team or even at a meetup
|
||||
or conference. In that case, you might enjoy watching
|
||||
[Bridget Kromhout's talk at KubeCon 2018 Europe](
|
||||
https://www.youtube.com/watch?v=mYsp_cGY2O0), explaining
|
||||
precisely how to run such a workshop yourself.
|
||||
|
||||
Finally, you can also contact the following persons,
|
||||
who are experienced speakers, are familiar with the
|
||||
material, and are available to deliver these workshops
|
||||
at your conference or for your company:
|
||||
|
||||
- jerome dot petazzoni at gmail dot com
|
||||
- bret at bretfisher dot com
|
||||
|
||||
If you are willing and able to deliver such workshops,
|
||||
feel free to submit a PR to add your name to that list!
|
||||
(If you are willing and able to deliver such workshops,
|
||||
feel free to submit a PR to add your name to that list!)
|
||||
|
||||
**Thank you!**
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ wrap Run this program in a container
|
||||
- Run `./workshopctl deploy TAG settings/somefile.yaml` to run `lib/postprep.py` via parallel-ssh
|
||||
- If it errors or times out, you should be able to rerun
|
||||
- Requires good connection to run all the parallel SSH connections, up to 100 parallel (ProTip: create dedicated management instance in same AWS region where you run all these utils from)
|
||||
- Run `./workshopctl pull-images TAG` to pre-pull a bunch of Docker images to the instances
|
||||
- Run `./workshopctl pull_images TAG` to pre-pull a bunch of Docker images to the instances
|
||||
- Run `./workshopctl cards TAG settings/somefile.yaml` generates PDF/HTML files to print and cut and hand out to students
|
||||
- *Have a great workshop*
|
||||
- Run `./workshopctl stop TAG` to terminate instances.
|
||||
@@ -210,7 +210,7 @@ The `postprep.py` file will be copied via parallel-ssh to all of the VMs and exe
|
||||
|
||||
#### Pre-pull images
|
||||
|
||||
$ ./workshopctl pull-images TAG
|
||||
$ ./workshopctl pull_images TAG
|
||||
|
||||
#### Generate cards
|
||||
|
||||
|
||||
@@ -108,7 +108,7 @@ system("sudo chmod +x /usr/local/bin/docker-machine")
|
||||
system("docker-machine version")
|
||||
|
||||
system("sudo apt-get remove -y --purge dnsmasq-base")
|
||||
system("sudo apt-get -qy install python-setuptools pssh apache2-utils httping htop unzip mosh")
|
||||
system("sudo apt-get -qy install python-setuptools pssh apache2-utils httping htop unzip mosh tree")
|
||||
|
||||
### Wait for Docker to be up.
|
||||
### (If we don't do this, Docker will not be responsive during the next step.)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
- All the content is available in a public GitHub repository:
|
||||
|
||||
https://github.com/jpetazzo/container.training
|
||||
https://@@GITREPO@@
|
||||
|
||||
- You can get updated "builds" of the slides there:
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
<!--
|
||||
.exercise[
|
||||
```open https://github.com/jpetazzo/container.training```
|
||||
```open https://@@GITREPO@@```
|
||||
```open http://container.training/```
|
||||
]
|
||||
-->
|
||||
@@ -23,7 +23,7 @@
|
||||
|
||||
<!--
|
||||
.exercise[
|
||||
```open https://github.com/jpetazzo/container.training/tree/master/slides/common/about-slides.md```
|
||||
```open https://@@GITREPO@@/tree/master/slides/common/about-slides.md```
|
||||
]
|
||||
-->
|
||||
|
||||
|
||||
@@ -48,11 +48,11 @@ Misattributed to Benjamin Franklin
|
||||
|
||||
- This is the stuff you're supposed to do!
|
||||
|
||||
- Go to [container.training](http://container.training/) to view these slides
|
||||
- Go to @@SLIDES@@ to view these slides
|
||||
|
||||
- Join the chat room: @@CHAT@@
|
||||
|
||||
<!-- ```open http://container.training/``` -->
|
||||
<!-- ```open @@SLIDES@@``` -->
|
||||
|
||||
]
|
||||
|
||||
@@ -222,7 +222,7 @@ If anything goes wrong — ask for help!
|
||||
Small setup effort; small cost; flexible environments
|
||||
|
||||
- Create a bunch of clusters for you and your friends
|
||||
([instructions](https://github.com/jpetazzo/container.training/tree/master/prepare-vms))
|
||||
([instructions](https://@@GITREPO@@/tree/master/prepare-vms))
|
||||
|
||||
Bigger setup effort; ideal for group training
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ fi
|
||||
|
||||
- Clone the repository on `node1`:
|
||||
```bash
|
||||
git clone git://github.com/jpetazzo/container.training
|
||||
git clone git://@@GITREPO@@
|
||||
```
|
||||
|
||||
]
|
||||
@@ -56,16 +56,16 @@ and displays aggregated logs.
|
||||
## More detail on our sample application
|
||||
|
||||
- Visit the GitHub repository with all the materials of this workshop:
|
||||
<br/>https://github.com/jpetazzo/container.training
|
||||
<br/>https://@@GITREPO@@
|
||||
|
||||
- The application is in the [dockercoins](
|
||||
https://github.com/jpetazzo/container.training/tree/master/dockercoins)
|
||||
https://@@GITREPO@@/tree/master/dockercoins)
|
||||
subdirectory
|
||||
|
||||
- Let's look at the general layout of the source code:
|
||||
|
||||
there is a Compose file [docker-compose.yml](
|
||||
https://github.com/jpetazzo/container.training/blob/master/dockercoins/docker-compose.yml) ...
|
||||
https://@@GITREPO@@/blob/master/dockercoins/docker-compose.yml) ...
|
||||
|
||||
... and 4 other services, each in its own directory:
|
||||
|
||||
@@ -124,7 +124,7 @@ def hash_bytes(data):
|
||||
```
|
||||
|
||||
(Full source code available [here](
|
||||
https://github.com/jpetazzo/container.training/blob/8279a3bce9398f7c1a53bdd95187c53eda4e6435/dockercoins/worker/worker.py#L17
|
||||
https://@@GITREPO@@/blob/8279a3bce9398f7c1a53bdd95187c53eda4e6435/dockercoins/worker/worker.py#L17
|
||||
))
|
||||
|
||||
---
|
||||
|
||||
@@ -17,5 +17,5 @@ class: title, in-person
|
||||
*Don't stream videos or download big files during the workshop.*<br/>
|
||||
*Thank you!*
|
||||
|
||||
**Slides: http://container.training/**
|
||||
]
|
||||
**Slides: @@SLIDES@@**
|
||||
]
|
||||
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python
|
||||
import re
|
||||
import sys
|
||||
|
||||
PREFIX = "name: toc-"
|
||||
EXCLUDED = ["in-person"]
|
||||
|
||||
class State(object):
|
||||
def __init__(self):
|
||||
self.current_slide = 1
|
||||
self.section_title = None
|
||||
self.section_start = 0
|
||||
self.section_slides = 0
|
||||
self.chapters = {}
|
||||
self.sections = {}
|
||||
def show(self):
|
||||
if self.section_title.startswith("chapter-"):
|
||||
return
|
||||
print("{0.section_title}\t{0.section_start}\t{0.section_slides}".format(self))
|
||||
self.sections[self.section_title] = self.section_slides
|
||||
|
||||
state = State()
|
||||
|
||||
title = None
|
||||
for line in open(sys.argv[1]):
|
||||
line = line.rstrip()
|
||||
if line.startswith(PREFIX):
|
||||
if state.section_title is None:
|
||||
print("{}\t{}\t{}".format("title", "index", "size"))
|
||||
else:
|
||||
state.show()
|
||||
state.section_title = line[len(PREFIX):].strip()
|
||||
state.section_start = state.current_slide
|
||||
state.section_slides = 0
|
||||
if line == "---":
|
||||
state.current_slide += 1
|
||||
state.section_slides += 1
|
||||
if line == "--":
|
||||
state.current_slide += 1
|
||||
toc_links = re.findall("\(#toc-(.*)\)", line)
|
||||
if toc_links and state.section_title.startswith("chapter-"):
|
||||
if state.section_title not in state.chapters:
|
||||
state.chapters[state.section_title] = []
|
||||
state.chapters[state.section_title].append(toc_links[0])
|
||||
# This is really hackish
|
||||
if line.startswith("class:"):
|
||||
for klass in EXCLUDED:
|
||||
if klass in line:
|
||||
state.section_slides -= 1
|
||||
state.current_slide -= 1
|
||||
|
||||
state.show()
|
||||
|
||||
for chapter in sorted(state.chapters):
|
||||
chapter_size = sum(state.sections[s] for s in state.chapters[chapter])
|
||||
print("{}\t{}\t{}".format("total size for", chapter, chapter_size))
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
#!/bin/sh
|
||||
INPUT=$1
|
||||
|
||||
{
|
||||
echo "# Front matter"
|
||||
cat "$INPUT"
|
||||
} |
|
||||
grep -e "^# " -e ^---$ | uniq -c |
|
||||
sed "s/^ *//" | sed s/---// |
|
||||
paste -d "\t" - -
|
||||
@@ -1,11 +1,14 @@
|
||||
title: |
|
||||
Introduction
|
||||
to Docker and
|
||||
Containers
|
||||
to Containers
|
||||
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/workshop-yyyymmdd-city)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- self-paced
|
||||
|
||||
@@ -27,13 +30,13 @@ chapters:
|
||||
- intro/Building_Images_With_Dockerfiles.md
|
||||
- intro/Cmd_And_Entrypoint.md
|
||||
- intro/Copying_Files_During_Build.md
|
||||
- intro/Multi_Stage_Builds.md
|
||||
- - intro/Multi_Stage_Builds.md
|
||||
- intro/Publishing_To_Docker_Hub.md
|
||||
- intro/Dockerfile_Tips.md
|
||||
- - intro/Naming_And_Inspecting.md
|
||||
- intro/Labels.md
|
||||
- intro/Getting_Inside.md
|
||||
- intro/Container_Networking_Basics.md
|
||||
- - intro/Container_Networking_Basics.md
|
||||
- intro/Network_Drivers.md
|
||||
- intro/Container_Network_Model.md
|
||||
#- intro/Connecting_Containers_With_Links.md
|
||||
@@ -42,13 +45,14 @@ chapters:
|
||||
- intro/Working_With_Volumes.md
|
||||
- intro/Compose_For_Dev_Stacks.md
|
||||
- intro/Docker_Machine.md
|
||||
- intro/Advanced_Dockerfiles.md
|
||||
- - intro/Advanced_Dockerfiles.md
|
||||
- intro/Application_Configuration.md
|
||||
- intro/Logging.md
|
||||
- intro/Resource_Limits.md
|
||||
- - intro/Namespaces_Cgroups.md
|
||||
- intro/Copy_On_Write.md
|
||||
#- intro/Containers_From_Scratch.md
|
||||
- intro/Container_Engines.md
|
||||
- - intro/Container_Engines.md
|
||||
- intro/Ecosystem.md
|
||||
- intro/Orchestration_Overview.md
|
||||
- common/thankyou.md
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
title: |
|
||||
Introduction
|
||||
to Docker and
|
||||
Containers
|
||||
to Containers
|
||||
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/workshop-yyyymmdd-city)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- in-person
|
||||
|
||||
@@ -27,13 +30,13 @@ chapters:
|
||||
- intro/Building_Images_With_Dockerfiles.md
|
||||
- intro/Cmd_And_Entrypoint.md
|
||||
- intro/Copying_Files_During_Build.md
|
||||
- intro/Multi_Stage_Builds.md
|
||||
- - intro/Multi_Stage_Builds.md
|
||||
- intro/Publishing_To_Docker_Hub.md
|
||||
- intro/Dockerfile_Tips.md
|
||||
- - intro/Naming_And_Inspecting.md
|
||||
- intro/Labels.md
|
||||
- intro/Getting_Inside.md
|
||||
- intro/Container_Networking_Basics.md
|
||||
- - intro/Container_Networking_Basics.md
|
||||
- intro/Network_Drivers.md
|
||||
- intro/Container_Network_Model.md
|
||||
#- intro/Connecting_Containers_With_Links.md
|
||||
@@ -42,13 +45,14 @@ chapters:
|
||||
- intro/Working_With_Volumes.md
|
||||
- intro/Compose_For_Dev_Stacks.md
|
||||
- intro/Docker_Machine.md
|
||||
- intro/Advanced_Dockerfiles.md
|
||||
- - intro/Advanced_Dockerfiles.md
|
||||
- intro/Application_Configuration.md
|
||||
- intro/Logging.md
|
||||
- intro/Resource_Limits.md
|
||||
- - intro/Namespaces_Cgroups.md
|
||||
- intro/Copy_On_Write.md
|
||||
#- intro/Containers_From_Scratch.md
|
||||
- intro/Container_Engines.md
|
||||
- - intro/Container_Engines.md
|
||||
- intro/Ecosystem.md
|
||||
- intro/Orchestration_Overview.md
|
||||
- common/thankyou.md
|
||||
|
||||
@@ -40,6 +40,8 @@ ambassador containers.
|
||||
|
||||
---
|
||||
|
||||
class: pic
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
@@ -49,7 +49,7 @@ Before diving in, let's see a small example of Compose in action.
|
||||
|
||||
---
|
||||
|
||||
## Compose in action
|
||||
class: pic
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ Containers also exist (sometimes with other names) on Windows, macOS, Solaris, F
|
||||
|
||||
## LXC
|
||||
|
||||
* The venerable ancestor (first realeased in 2008).
|
||||
* The venerable ancestor (first released in 2008).
|
||||
|
||||
* Docker initially relied on it to execute containers.
|
||||
|
||||
|
||||
@@ -498,12 +498,12 @@ b2887adeb5578a01fd9c55c435cad56bbbe802350711d2743691f95743680b09
|
||||
|
||||
* If containers span multiple hosts, we need an *overlay* network to connect them together.
|
||||
|
||||
* Docker ships with a default network plugin, `overlay`, implementing an overlay network leveraging
|
||||
* Docker ships with a default network plugin, `overlay`, implementing an overlay network leveraging
|
||||
VXLAN, *enabled with Swarm Mode*.
|
||||
|
||||
* Other plugins (Weave, Calico...) can provide overlay networks as well.
|
||||
|
||||
* Once you have an overlay network, *all the features that we've used in this chapter work identically
|
||||
* Once you have an overlay network, *all the features that we've used in this chapter work identically
|
||||
across multiple hosts.*
|
||||
|
||||
---
|
||||
@@ -542,13 +542,174 @@ General idea:
|
||||
|
||||
---
|
||||
|
||||
## Section summary
|
||||
## Connecting and disconnecting dynamically
|
||||
|
||||
We've learned how to:
|
||||
* So far, we have specified which network to use when starting the container.
|
||||
|
||||
* Create private networks for groups of containers.
|
||||
* The Docker Engine also allows to connect and disconnect while the container runs.
|
||||
|
||||
* Assign IP addresses to containers.
|
||||
* This feature is exposed through the Docker API, and through two Docker CLI commands:
|
||||
|
||||
* Use container naming to implement service discovery.
|
||||
* `docker network connect <network> <container>`
|
||||
|
||||
* `docker network disconnect <network> <container>`
|
||||
|
||||
---
|
||||
|
||||
## Dynamically connecting to a network
|
||||
|
||||
* We have a container named `es` connected to a network named `dev`.
|
||||
|
||||
* Let's start a simple alpine container on the default network:
|
||||
|
||||
```bash
|
||||
$ docker run -ti alpine sh
|
||||
/ #
|
||||
```
|
||||
|
||||
* In this container, try to ping the `es` container:
|
||||
|
||||
```bash
|
||||
/ # ping es
|
||||
ping: bad address 'es'
|
||||
```
|
||||
|
||||
This doesn't work, but we will change that by connecting the container.
|
||||
|
||||
---
|
||||
|
||||
## Finding the container ID and connecting it
|
||||
|
||||
* Figure out the ID of our alpine container; here are two methods:
|
||||
|
||||
* looking at `/etc/hostname` in the container,
|
||||
|
||||
* running `docker ps -lq` on the host.
|
||||
|
||||
* Run the following command on the host:
|
||||
|
||||
```bash
|
||||
$ docker network connect dev `<container_id>`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checking what we did
|
||||
|
||||
* Try again to `ping es` from the container.
|
||||
|
||||
* It should now work correctly:
|
||||
|
||||
```bash
|
||||
/ # ping es
|
||||
PING es (172.20.0.3): 56 data bytes
|
||||
64 bytes from 172.20.0.3: seq=0 ttl=64 time=0.376 ms
|
||||
64 bytes from 172.20.0.3: seq=1 ttl=64 time=0.130 ms
|
||||
^C
|
||||
```
|
||||
|
||||
* Interrupt it with Ctrl-C.
|
||||
|
||||
---
|
||||
|
||||
## Looking at the network setup in the container
|
||||
|
||||
We can look at the list of network interfaces with `ifconfig`, `ip a`, or `ip l`:
|
||||
|
||||
.small[
|
||||
```bash
|
||||
/ # ip a
|
||||
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
|
||||
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
|
||||
inet 127.0.0.1/8 scope host lo
|
||||
valid_lft forever preferred_lft forever
|
||||
18: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
|
||||
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
|
||||
valid_lft forever preferred_lft forever
|
||||
20: eth1@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
|
||||
link/ether 02:42:ac:14:00:04 brd ff:ff:ff:ff:ff:ff
|
||||
inet 172.20.0.4/16 brd 172.20.255.255 scope global eth1
|
||||
valid_lft forever preferred_lft forever
|
||||
/ #
|
||||
```
|
||||
]
|
||||
|
||||
Each network connection is materialized with a virtual network interface.
|
||||
|
||||
As we can see, we can be connected to multiple networks at the same time.
|
||||
|
||||
---
|
||||
|
||||
## Disconnecting from a network
|
||||
|
||||
* Let's try the symmetrical command to disconnect the container:
|
||||
```bash
|
||||
$ docker network disconnect dev <container_id>
|
||||
```
|
||||
|
||||
* From now on, if we try to ping `es`, it will not resolve:
|
||||
```bash
|
||||
/ # ping es
|
||||
ping: bad address 'es'
|
||||
```
|
||||
|
||||
* Trying to ping the IP address directly won't work either:
|
||||
```bash
|
||||
/ # ping 172.20.0.3
|
||||
... (nothing happens until we interrupt it with Ctrl-C)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
class: extra-details
|
||||
|
||||
## Network aliases are scoped per network
|
||||
|
||||
* Each network has its own set of network aliases.
|
||||
|
||||
* We saw this earlier: `es` resolves to different addresses in `dev` and `prod`.
|
||||
|
||||
* If we are connected to multiple networks, the resolver looks up names in each of them
|
||||
(as of Docker Engine 18.03, it is the connection order) and stops as soon as the name
|
||||
is found.
|
||||
|
||||
* Therefore, if we are connected to both `dev` and `prod`, resolving `es` will **not**
|
||||
give us the addresses of all the `es` services; but only the ones in `dev` or `prod`.
|
||||
|
||||
* However, we can lookup `es.dev` or `es.prod` if we need to.
|
||||
|
||||
---
|
||||
|
||||
class: extra-details
|
||||
|
||||
## Finding out about our networks and names
|
||||
|
||||
* We can do reverse DNS lookups on containers' IP addresses.
|
||||
|
||||
* If the IP address belongs to a network (other than the default bridge), the result will be:
|
||||
|
||||
```
|
||||
name-or-first-alias-or-container-id.network-name
|
||||
```
|
||||
|
||||
* Example:
|
||||
|
||||
.small[
|
||||
```bash
|
||||
$ docker run -ti --net prod --net-alias hello alpine
|
||||
/ # apk add --no-cache drill
|
||||
...
|
||||
OK: 5 MiB in 13 packages
|
||||
/ # ifconfig
|
||||
eth0 Link encap:Ethernet HWaddr 02:42:AC:15:00:03
|
||||
inet addr:`172.21.0.3` Bcast:172.21.255.255 Mask:255.255.0.0
|
||||
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
|
||||
...
|
||||
/ # drill -t ptr `3.0.21.172`.in-addr.arpa
|
||||
...
|
||||
;; ANSWER SECTION:
|
||||
3.0.21.172.in-addr.arpa. 600 IN PTR `hello.prod`.
|
||||
...
|
||||
```
|
||||
]
|
||||
|
||||
@@ -10,10 +10,12 @@
|
||||
|
||||
* [Solaris Containers (2004)](https://en.wikipedia.org/wiki/Solaris_Containers)
|
||||
|
||||
* [FreeBSD jails (1999)](https://www.freebsd.org/cgi/man.cgi?query=jail&sektion=8&manpath=FreeBSD+4.0-RELEASE)
|
||||
* [FreeBSD jails (1999-2000)](https://www.freebsd.org/cgi/man.cgi?query=jail&sektion=8&manpath=FreeBSD+4.0-RELEASE)
|
||||
|
||||
Containers have been around for a *very long time* indeed.
|
||||
|
||||
(See [this excellent blog post by Serge Hallyn](https://s3hh.wordpress.com/2018/03/22/history-of-containers/) for more historic details.)
|
||||
|
||||
---
|
||||
|
||||
class: pic
|
||||
|
||||
@@ -96,3 +96,266 @@ CMD, EXPOSE ...
|
||||
* The build fails as soon as an instruction fails
|
||||
* If `RUN <unit tests>` fails, the build doesn't produce an image
|
||||
* If it succeeds, it produces a clean image (without test libraries and data)
|
||||
|
||||
---
|
||||
|
||||
# Dockerfile examples
|
||||
|
||||
There are a number of tips, tricks, and techniques that we can use in Dockerfiles.
|
||||
|
||||
But sometimes, we have to use different (and even opposed) practices depending on:
|
||||
|
||||
- the complexity of our project,
|
||||
|
||||
- the programming language or framework that we are using,
|
||||
|
||||
- the stage of our project (early MVP vs. super-stable production),
|
||||
|
||||
- whether we're building a final image or a base for further images,
|
||||
|
||||
- etc.
|
||||
|
||||
We are going to show a few examples using very different techniques.
|
||||
|
||||
---
|
||||
|
||||
## When to optimize an image
|
||||
|
||||
When authoring official images, it is a good idea to reduce as much as possible:
|
||||
|
||||
- the number of layers,
|
||||
|
||||
- the size of the final image.
|
||||
|
||||
This is often done at the expense of build time and convenience for the image maintainer;
|
||||
but when an image is downloaded millions of time, saving even a few seconds of pull time
|
||||
can be worth it.
|
||||
|
||||
.small[
|
||||
```dockerfile
|
||||
RUN apt-get update && apt-get install -y libpng12-dev libjpeg-dev && rm -rf /var/lib/apt/lists/* \
|
||||
&& docker-php-ext-configure gd --with-png-dir=/usr --with-jpeg-dir=/usr \
|
||||
&& docker-php-ext-install gd
|
||||
...
|
||||
RUN curl -o wordpress.tar.gz -SL https://wordpress.org/wordpress-${WORDPRESS_UPSTREAM_VERSION}.tar.gz \
|
||||
&& echo "$WORDPRESS_SHA1 *wordpress.tar.gz" | sha1sum -c - \
|
||||
&& tar -xzf wordpress.tar.gz -C /usr/src/ \
|
||||
&& rm wordpress.tar.gz \
|
||||
&& chown -R www-data:www-data /usr/src/wordpress
|
||||
```
|
||||
]
|
||||
|
||||
(Source: [Wordpress official image](https://github.com/docker-library/wordpress/blob/618490d4bdff6c5774b84b717979bfe3d6ba8ad1/apache/Dockerfile))
|
||||
|
||||
---
|
||||
|
||||
## When to *not* optimize an image
|
||||
|
||||
Sometimes, it is better to prioritize *maintainer convenience*.
|
||||
|
||||
In particular, if:
|
||||
|
||||
- the image changes a lot,
|
||||
|
||||
- the image has very few users (e.g. only 1, the maintainer!),
|
||||
|
||||
- the image is built and run on the same machine,
|
||||
|
||||
- the image is built and run on machines with a very fast link ...
|
||||
|
||||
In these cases, just keep things simple!
|
||||
|
||||
(Next slide: a Dockerfile that can be used to preview a Jekyll / github pages site.)
|
||||
|
||||
---
|
||||
|
||||
```dockerfile
|
||||
FROM debian:sid
|
||||
|
||||
RUN apt-get update -q
|
||||
RUN apt-get install -yq build-essential make
|
||||
RUN apt-get install -yq zlib1g-dev
|
||||
RUN apt-get install -yq ruby ruby-dev
|
||||
RUN apt-get install -yq python-pygments
|
||||
RUN apt-get install -yq nodejs
|
||||
RUN apt-get install -yq cmake
|
||||
RUN gem install --no-rdoc --no-ri github-pages
|
||||
|
||||
COPY . /blog
|
||||
WORKDIR /blog
|
||||
|
||||
VOLUME /blog/_site
|
||||
|
||||
EXPOSE 4000
|
||||
CMD ["jekyll", "serve", "--host", "0.0.0.0", "--incremental"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Multi-dimensional versioning systems
|
||||
|
||||
Images can have a tag, indicating the version of the image.
|
||||
|
||||
But sometimes, there are multiple important components, and we need to indicate the versions
|
||||
for all of them.
|
||||
|
||||
This can be done with environment variables:
|
||||
|
||||
```dockerfile
|
||||
ENV PIP=9.0.3 \
|
||||
ZC_BUILDOUT=2.11.2 \
|
||||
SETUPTOOLS=38.7.0 \
|
||||
PLONE_MAJOR=5.1 \
|
||||
PLONE_VERSION=5.1.0 \
|
||||
PLONE_MD5=76dc6cfc1c749d763c32fff3a9870d8d
|
||||
```
|
||||
|
||||
(Source: [Plone official image](https://github.com/plone/plone.docker/blob/master/5.1/5.1.0/alpine/Dockerfile))
|
||||
|
||||
---
|
||||
|
||||
## Entrypoints and wrappers
|
||||
|
||||
It is very common to define a custom entrypoint.
|
||||
|
||||
That entrypoint will generally be a script, performing any combination of:
|
||||
|
||||
- pre-flights checks (if a required dependency is not available, display
|
||||
a nice error message early instead of an obscure one in a deep log file),
|
||||
|
||||
- generation or validation of configuration files,
|
||||
|
||||
- dropping privileges (with e.g. `su` or `gosu`, sometimes combined with `chown`),
|
||||
|
||||
- and more.
|
||||
|
||||
---
|
||||
|
||||
## A typical entrypoint script
|
||||
|
||||
```dockerfile
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# first arg is '-f' or '--some-option'
|
||||
# or first arg is 'something.conf'
|
||||
if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
|
||||
set -- redis-server "$@"
|
||||
fi
|
||||
|
||||
# allow the container to be started with '--user'
|
||||
if [ "$1" = 'redis-server' -a "$(id -u)" = '0' ]; then
|
||||
chown -R redis .
|
||||
exec su-exec redis "$0" "$@"
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
```
|
||||
|
||||
(Source: [Redis official image](https://github.com/docker-library/redis/blob/d24f2be82673ccef6957210cc985e392ebdc65e4/4.0/alpine/docker-entrypoint.sh))
|
||||
|
||||
---
|
||||
|
||||
## Factoring information
|
||||
|
||||
To facilitate maintenance (and avoid human errors), avoid to repeat information like:
|
||||
|
||||
- version numbers,
|
||||
|
||||
- remote asset URLs (e.g. source tarballs) ...
|
||||
|
||||
Instead, use environment variables.
|
||||
|
||||
.small[
|
||||
```dockerfile
|
||||
ENV NODE_VERSION 10.2.1
|
||||
...
|
||||
RUN ...
|
||||
&& curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION.tar.xz" \
|
||||
&& curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
|
||||
&& gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
|
||||
&& grep " node-v$NODE_VERSION.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
|
||||
&& tar -xf "node-v$NODE_VERSION.tar.xz" \
|
||||
&& cd "node-v$NODE_VERSION" \
|
||||
...
|
||||
```
|
||||
]
|
||||
|
||||
(Source: [Nodejs official image](https://github.com/nodejs/docker-node/blob/master/10/alpine/Dockerfile))
|
||||
|
||||
---
|
||||
|
||||
## Overrides
|
||||
|
||||
In theory, development and production images should be the same.
|
||||
|
||||
In practice, we often need to enable specific behaviors in development (e.g. debug statements).
|
||||
|
||||
One way to reconcile both needs is to use Compose to enable these behaviors.
|
||||
|
||||
Let's look at the [trainingwheels](https://github.com/jpetazzo/trainingwheels) demo app for an example.
|
||||
|
||||
---
|
||||
|
||||
## Production image
|
||||
|
||||
This Dockerfile builds an image leveraging gunicorn:
|
||||
|
||||
```dockerfile
|
||||
FROM python
|
||||
RUN pip install flask
|
||||
RUN pip install gunicorn
|
||||
RUN pip install redis
|
||||
COPY . /src
|
||||
WORKDIR /src
|
||||
CMD gunicorn --bind 0.0.0.0:5000 --workers 10 counter:app
|
||||
EXPOSE 5000
|
||||
```
|
||||
|
||||
(Source: [traininghweels Dockerfile](https://github.com/jpetazzo/trainingwheels/blob/master/www/Dockerfile))
|
||||
|
||||
---
|
||||
|
||||
## Development Compose file
|
||||
|
||||
This Compose file uses the same image, but with a few overrides for development:
|
||||
|
||||
- the Flask development server is used (overriding `CMD`),
|
||||
|
||||
- the `DEBUG` environment variable is set,
|
||||
|
||||
- a volume is used to provide a faster local development workflow.
|
||||
|
||||
.small[
|
||||
```yaml
|
||||
services:
|
||||
www:
|
||||
build: www
|
||||
ports:
|
||||
- 8000:5000
|
||||
user: nobody
|
||||
environment:
|
||||
DEBUG: 1
|
||||
command: python counter.py
|
||||
volumes:
|
||||
- ./www:/src
|
||||
```
|
||||
]
|
||||
|
||||
(Source: [trainingwheels Compose file](https://github.com/jpetazzo/trainingwheels/blob/master/docker-compose.yml))
|
||||
|
||||
---
|
||||
|
||||
## How to know which best practices are better?
|
||||
|
||||
- The main goal of containers is to make our lives easier.
|
||||
|
||||
- In this chapter, we showed many ways to write Dockerfiles.
|
||||
|
||||
- These Dockerfiles use sometimes diametrally opposed techniques.
|
||||
|
||||
- Yet, they were the "right" ones *for a specific situation.*
|
||||
|
||||
- It's OK (and even encouraged) to start simple and evolve as needed.
|
||||
|
||||
- Feel free to review this chapter later (after writing a few Dockerfiles) for inspiration!
|
||||
|
||||
@@ -85,7 +85,7 @@ class: extra-details
|
||||
|
||||
https://docs.docker.com/docker-for-mac/install/
|
||||
|
||||
* On Windows 10 Pro, Enterprise, and Eduction, you can use Docker for Windows:
|
||||
* On Windows 10 Pro, Enterprise, and Education, you can use Docker for Windows:
|
||||
|
||||
https://docs.docker.com/docker-for-windows/install/
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ At the end of this section, you will be able to:
|
||||
|
||||
---
|
||||
|
||||
## Containerized local development environments
|
||||
## Local development in a container
|
||||
|
||||
We want to solve the following issues:
|
||||
|
||||
|
||||
+22
-1
@@ -131,6 +131,27 @@ We will then show one particular method in action, using ELK and Docker's loggin
|
||||
|
||||
---
|
||||
|
||||
## A word of warning about `json-file`
|
||||
|
||||
- By default, log file size is unlimited.
|
||||
|
||||
- This means that a very verbose container *will* use up all your disk space.
|
||||
|
||||
(Or a less verbose container, but running for a very long time.)
|
||||
|
||||
- Log rotation can be enabled by setting a `max-size` option.
|
||||
|
||||
- Older log files can be removed by setting a `max-file` option.
|
||||
|
||||
- Just like other logging options, these can be set per container, or globally.
|
||||
|
||||
Example:
|
||||
```bash
|
||||
$ docker run --log-opt max-size=10m --log-opt max-file=3 elasticsearch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Demo: sending logs to ELK
|
||||
|
||||
- We are going to deploy an ELK stack.
|
||||
@@ -192,7 +213,7 @@ $ docker-compose -f elk.yml up -d
|
||||
|
||||
- it is set with the `ELASTICSEARCH_URL` environment variable,
|
||||
|
||||
- by default it is `localhost:9200`, we change it to `elastichsearch:9200`.
|
||||
- by default it is `localhost:9200`, we change it to `elasticsearch:9200`.
|
||||
|
||||
- We need to configure Logstash:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Multi-stage builds
|
||||
# Reducing image size
|
||||
|
||||
* In the previous example, our final image contained:
|
||||
|
||||
@@ -14,7 +14,196 @@
|
||||
|
||||
---
|
||||
|
||||
## Multi-stage builds principles
|
||||
## Can't we remove superfluous files with `RUN`?
|
||||
|
||||
What happens if we do one of the following commands?
|
||||
|
||||
- `RUN rm -rf ...`
|
||||
|
||||
- `RUN apt-get remove ...`
|
||||
|
||||
- `RUN make clean ...`
|
||||
|
||||
--
|
||||
|
||||
This adds a layer which removes a bunch of files.
|
||||
|
||||
But the previous layers (which added the files) still exist.
|
||||
|
||||
---
|
||||
|
||||
## Removing files with an extra layer
|
||||
|
||||
When downloading an image, all the layers must be downloaded.
|
||||
|
||||
| Dockerfile instruction | Layer size | Image size |
|
||||
| ---------------------- | ---------- | ---------- |
|
||||
| `FROM ubuntu` | Size of base image | Size of base image |
|
||||
| `...` | ... | Sum of this layer <br/>+ all previous ones |
|
||||
| `RUN apt-get install somepackage` | Size of files added <br/>(e.g. a few MB) | Sum of this layer <br/>+ all previous ones |
|
||||
| `...` | ... | Sum of this layer <br/>+ all previous ones |
|
||||
| `RUN apt-get remove somepackage` | Almost zero <br/>(just metadata) | Same as previous one |
|
||||
|
||||
Therefore, `RUN rm` does not reduce the size of the image or free up disk space.
|
||||
|
||||
---
|
||||
|
||||
## Removing unnecessary files
|
||||
|
||||
Various techniques are available to obtain smaller images:
|
||||
|
||||
- collapsing layers,
|
||||
|
||||
- adding binaries that are built outside of the Dockerfile,
|
||||
|
||||
- squashing the final image,
|
||||
|
||||
- multi-stage builds.
|
||||
|
||||
Let's review them quickly.
|
||||
|
||||
---
|
||||
|
||||
## Collapsing layers
|
||||
|
||||
You will frequently see Dockerfiles like this:
|
||||
|
||||
```dockerfile
|
||||
FROM ubuntu
|
||||
RUN apt-get update && apt-get install xxx && ... && apt-get remove xxx && ...
|
||||
```
|
||||
|
||||
Or the (more readable) variant:
|
||||
|
||||
```dockerfile
|
||||
FROM ubuntu
|
||||
RUN apt-get update \
|
||||
&& apt-get install xxx \
|
||||
&& ... \
|
||||
&& apt-get remove xxx \
|
||||
&& ...
|
||||
```
|
||||
|
||||
This `RUN` command gives us a single layer.
|
||||
|
||||
The files that are added, then removed in the same layer, do not grow the layer size.
|
||||
|
||||
---
|
||||
|
||||
## Collapsing layers: pros and cons
|
||||
|
||||
Pros:
|
||||
|
||||
- works on all versions of Docker
|
||||
|
||||
- doesn't require extra tools
|
||||
|
||||
Cons:
|
||||
|
||||
- not very readable
|
||||
|
||||
- some unnecessary files might still remain if the cleanup is not torough
|
||||
|
||||
- that layer is expensive (slow to build)
|
||||
|
||||
---
|
||||
|
||||
## Building binaries outside of the Dockerfile
|
||||
|
||||
This results in a Dockerfile looking like this:
|
||||
|
||||
```dockerfile
|
||||
FROM ubuntu
|
||||
COPY xxx /usr/local/bin
|
||||
```
|
||||
|
||||
Of course, this implies that the file `xxx` exists in the build context.
|
||||
|
||||
That file has to exist before you can run `docker build`.
|
||||
|
||||
For instance, it can:
|
||||
|
||||
- exist in the code repository,
|
||||
- be created by another tool (script, Makefile...),
|
||||
- be created by another container image and extracted from the image.
|
||||
|
||||
See for instance the [busybox official image](https://github.com/docker-library/busybox/blob/fe634680e32659aaf0ee0594805f74f332619a90/musl/Dockerfile) or this [older busybox image](https://github.com/jpetazzo/docker-busybox).
|
||||
|
||||
---
|
||||
|
||||
## Building binaries outside: pros and cons
|
||||
|
||||
Pros:
|
||||
|
||||
- final image can be very small
|
||||
|
||||
Cons:
|
||||
|
||||
- requires an extra build tool
|
||||
|
||||
- we're back in dependency hell and "works on my machine"
|
||||
|
||||
Cons, if binary is added to code repository:
|
||||
|
||||
- breaks portability across different platforms
|
||||
|
||||
- grows repository size a lot if the binary is updated frequently
|
||||
|
||||
---
|
||||
|
||||
## Squashing the final image
|
||||
|
||||
The idea is to transform the final image into a single-layer image.
|
||||
|
||||
This can be done in (at least) two ways.
|
||||
|
||||
- Activate experimental features and squash the final image:
|
||||
```bash
|
||||
docker image build --squash ...
|
||||
```
|
||||
|
||||
- Export/import the final image.
|
||||
```bash
|
||||
docker build -t temp-image .
|
||||
docker run --entrypoint true --name temp-container temp-image
|
||||
docker export temp-container | docker import - final-image
|
||||
docker rm temp-container
|
||||
docker rmi temp-image
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Squashing the image: pros and cons
|
||||
|
||||
Pros:
|
||||
|
||||
- single-layer images are smaller and faster to download
|
||||
|
||||
- removed files no longer take up storage and network resources
|
||||
|
||||
Cons:
|
||||
|
||||
- we still need to actively remove unnecessary files
|
||||
|
||||
- squash operation can take a lot of time (on big images)
|
||||
|
||||
- squash operation does not benefit from cache
|
||||
<br/>
|
||||
(even if we change just a tiny file, the whole image needs to be re-squashed)
|
||||
|
||||
---
|
||||
|
||||
## Multi-stage builds
|
||||
|
||||
Multi-stage builds allow us to have multiple *stages*.
|
||||
|
||||
Each stage is a separate image, and can copy files from previous stages.
|
||||
|
||||
We're going to see how they work in more detail.
|
||||
|
||||
---
|
||||
|
||||
# Multi-stage builds
|
||||
|
||||
* At any point in our `Dockerfile`, we can add a new `FROM` line.
|
||||
|
||||
|
||||
@@ -76,6 +76,8 @@ The last item should be done for educational purposes only!
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Manipulating namespaces
|
||||
|
||||
- Namespaces are created with two methods:
|
||||
@@ -94,6 +96,8 @@ The last item should be done for educational purposes only!
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Namespaces lifecycle
|
||||
|
||||
- When the last process of a namespace exits, the namespace is destroyed.
|
||||
@@ -114,6 +118,8 @@ The last item should be done for educational purposes only!
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Namespaces can be used independently
|
||||
|
||||
- As mentioned in the previous slides:
|
||||
@@ -150,6 +156,8 @@ The last item should be done for educational purposes only!
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Creating our first namespace
|
||||
|
||||
Let's use `unshare` to create a new process that will have its own UTS namespace:
|
||||
@@ -166,6 +174,8 @@ $ sudo unshare --uts
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Demonstrating our uts namespace
|
||||
|
||||
In our new "container", check the hostname, change it, and check it:
|
||||
@@ -398,6 +408,8 @@ class: extra-details
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Setting up a private `/tmp`
|
||||
|
||||
Create a new mount namespace:
|
||||
@@ -435,6 +447,8 @@ The mount is automatically cleaned up when you exit the process.
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## PID namespace in action
|
||||
|
||||
Create a new PID namespace:
|
||||
@@ -453,10 +467,14 @@ Check the process tree in the new namespace:
|
||||
|
||||
--
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
🤔 Why do we see all the processes?!?
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## PID namespaces and `/proc`
|
||||
|
||||
- Tools like `ps` rely on the `/proc` pseudo-filesystem.
|
||||
@@ -471,6 +489,8 @@ Check the process tree in the new namespace:
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## PID namespaces, take 2
|
||||
|
||||
- This can be solved by mounting `/proc` in the namespace.
|
||||
@@ -570,6 +590,8 @@ Check `man 2 unshare` and `man pid_namespaces` if you want more details.
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## User namespace challenges
|
||||
|
||||
- UID needs to be mapped when passed between processes or kernel subsystems.
|
||||
@@ -686,6 +708,8 @@ cpu memory
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Cgroups v1 vs v2
|
||||
|
||||
- Cgroups v1 are available on all systems (and widely used).
|
||||
@@ -759,6 +783,8 @@ cpu memory
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Avoiding the OOM killer
|
||||
|
||||
- For some workloads (databases and stateful systems), killing
|
||||
@@ -778,6 +804,8 @@ cpu memory
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Overhead of the memory cgroup
|
||||
|
||||
- Each time a process grabs or releases a page, the kernel update counters.
|
||||
@@ -796,6 +824,8 @@ cpu memory
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Setting up a limit with the memory cgroup
|
||||
|
||||
Create a new memory cgroup:
|
||||
@@ -808,7 +838,7 @@ $ sudo mkdir $CG
|
||||
Limit it to approximately 100MB of memory usage:
|
||||
|
||||
```bash
|
||||
$ sudo tee $CG/memory.memsw.limit_in_bytes <<<100000000
|
||||
$ sudo tee $CG/memory.memsw.limit_in_bytes <<< 100000000
|
||||
```
|
||||
|
||||
Move the current process to that cgroup:
|
||||
@@ -819,8 +849,67 @@ $ sudo tee $CG/tasks <<< $$
|
||||
|
||||
The current process *and all its future children* are now limited.
|
||||
|
||||
(Confused about `<<<`? Look at the next slide!)
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## What's `<<<`?
|
||||
|
||||
- This is a "here string". (It is a non-POSIX shell extension.)
|
||||
|
||||
- The following commands are equivalent:
|
||||
|
||||
```bash
|
||||
foo <<< hello
|
||||
```
|
||||
|
||||
```bash
|
||||
echo hello | foo
|
||||
```
|
||||
|
||||
```bash
|
||||
foo <<EOF
|
||||
hello
|
||||
EOF
|
||||
```
|
||||
|
||||
- Why did we use that?
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Writing to cgroups pseudo-files requires root
|
||||
|
||||
Instead of:
|
||||
|
||||
```bash
|
||||
sudo tee $CG/tasks <<< $$
|
||||
```
|
||||
|
||||
We could have done:
|
||||
|
||||
```bash
|
||||
sudo sh -c "echo $$ > $CG/tasks"
|
||||
```
|
||||
|
||||
The following commands, however, would be invalid:
|
||||
|
||||
```bash
|
||||
sudo echo $$ > $CG/tasks
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo -i # (or su)
|
||||
echo $$ > $CG/tasks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
class: extra-details, deep-dive
|
||||
|
||||
## Testing the memory limit
|
||||
|
||||
Start the Python interpreter:
|
||||
@@ -860,8 +949,6 @@ Killed
|
||||
|
||||
- Allows to set relative weights used by the scheduler.
|
||||
|
||||
- We cannot set CPU limits (like, "don't use more than 10% of CPU").
|
||||
|
||||
---
|
||||
|
||||
## Cpuset cgroup
|
||||
|
||||
@@ -420,8 +420,3 @@ It depends on:
|
||||
|
||||
- false, if we focus on what matters.
|
||||
|
||||
---
|
||||
|
||||
## Kubernetes in action
|
||||
|
||||
.center[]
|
||||
@@ -0,0 +1,229 @@
|
||||
# Limiting resources
|
||||
|
||||
- So far, we have used containers as convenient units of deployment.
|
||||
|
||||
- What happens when a container tries to use more resources than available?
|
||||
|
||||
(RAM, CPU, disk usage, disk and network I/O...)
|
||||
|
||||
- What happens when multiple containers compete for the same resource?
|
||||
|
||||
- Can we limit resources available to a container?
|
||||
|
||||
(Spoiler alert: yes!)
|
||||
|
||||
---
|
||||
|
||||
## Container processes are normal processes
|
||||
|
||||
- Containers are closer to "fancy processes" than to "lightweight VMs".
|
||||
|
||||
- A process running in a container is, in fact, a process running on the host.
|
||||
|
||||
- Let's look at the output of `ps` on a container host running 3 containers :
|
||||
|
||||
```
|
||||
0 2662 0.2 0.3 /usr/bin/dockerd -H fd://
|
||||
0 2766 0.1 0.1 \_ docker-containerd --config /var/run/docker/containe
|
||||
0 23479 0.0 0.0 \_ docker-containerd-shim -namespace moby -workdir
|
||||
0 23497 0.0 0.0 | \_ `nginx`: master process nginx -g daemon off;
|
||||
101 23543 0.0 0.0 | \_ `nginx`: worker process
|
||||
0 23565 0.0 0.0 \_ docker-containerd-shim -namespace moby -workdir
|
||||
102 23584 9.4 11.3 | \_ `/docker-java-home/jre/bin/java` -Xms2g -Xmx2
|
||||
0 23707 0.0 0.0 \_ docker-containerd-shim -namespace moby -workdir
|
||||
0 23725 0.0 0.0 \_ `/bin/sh`
|
||||
```
|
||||
|
||||
- The highlighted processes are containerized processes.
|
||||
<br/>
|
||||
(That host is running nginx, elasticsearch, and alpine.)
|
||||
|
||||
---
|
||||
|
||||
## By default: nothing changes
|
||||
|
||||
- What happens when a process uses too much memory on a Linux system?
|
||||
|
||||
--
|
||||
|
||||
- Simplified answer:
|
||||
|
||||
- swap is used (if available);
|
||||
|
||||
- if there is not enough swap space, eventually, the out-of-memory killer is invoked;
|
||||
|
||||
- the OOM killer uses heuristics to kill processes;
|
||||
|
||||
- sometimes, it kills an unrelated process.
|
||||
|
||||
--
|
||||
|
||||
- What happens when a container uses too much memory?
|
||||
|
||||
- The same thing!
|
||||
|
||||
(i.e., a process eventually gets killed, possibly in another container.)
|
||||
|
||||
---
|
||||
|
||||
## Limiting container resources
|
||||
|
||||
- The Linux kernel offers rich mechanisms to limit container resources.
|
||||
|
||||
- For memory usage, the mechanism is part of the *cgroup* subsystem.
|
||||
|
||||
- This subsystem allows to limit the memory for a process or a group of processes.
|
||||
|
||||
- A container engine leverages these mechanisms to limit memory for a container.
|
||||
|
||||
- The out-of-memory killer has a new behavior:
|
||||
|
||||
- it runs when a container exceeds its allowed memory usage,
|
||||
|
||||
- in that case, it only kills processes in that container.
|
||||
|
||||
---
|
||||
|
||||
## Limiting memory in practice
|
||||
|
||||
- The Docker Engine offers multiple flags to limit memory usage.
|
||||
|
||||
- The two most useful ones are `--memory` and `--memory-swap`.
|
||||
|
||||
- `--memory` limits the amount of physical RAM used by a container.
|
||||
|
||||
- `--memory-swap` limits the total amount (RAM+swap) used by a container.
|
||||
|
||||
- The memory limit can be expressed in bytes, or with a unit suffix.
|
||||
|
||||
(e.g.: `--memory 100m` = 100 megabytes.)
|
||||
|
||||
- We will see two strategies: limiting RAM usage, or limiting both
|
||||
|
||||
---
|
||||
|
||||
## Limiting RAM usage
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
docker run -ti --memory 100m python
|
||||
```
|
||||
|
||||
If the container tries to use more than 100 MB of RAM, *and* swap is available:
|
||||
|
||||
- the container will not be killed,
|
||||
|
||||
- memory above 100 MB will be swapped out,
|
||||
|
||||
- in most cases, the app in the container will be slowed down (a lot).
|
||||
|
||||
If we run out of swap, the global OOM killer still intervenes.
|
||||
|
||||
---
|
||||
|
||||
## Limiting both RAM and swap usage
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
docker run -ti --memory 100m --memory-swap 100m python
|
||||
```
|
||||
|
||||
If the container tries to use more than 100 MB of memory, it is killed.
|
||||
|
||||
On the other hand, the application will never be slowed down because of swap.
|
||||
|
||||
---
|
||||
|
||||
## When to pick which strategy?
|
||||
|
||||
- Stateful services (like databases) will lose or corrupt data when killed
|
||||
|
||||
- Allow them to use swap space, but monitor swap usage
|
||||
|
||||
- Stateless services can usually be killed with little impact
|
||||
|
||||
- Limit their mem+swap usage, but monitor if they get killed
|
||||
|
||||
- Ultimately, this is no different from "do I want swap, and how much?"
|
||||
|
||||
---
|
||||
|
||||
## Limiting CPU usage
|
||||
|
||||
- There are no less than 3 ways to limit CPU usage:
|
||||
|
||||
- setting a relative priority with `--cpu-shares`,
|
||||
|
||||
- setting a CPU% limit with `--cpus`,
|
||||
|
||||
- pinning a container to specific CPUs with `--cpuset-cpus`.
|
||||
|
||||
- They can be used separately or together.
|
||||
|
||||
---
|
||||
|
||||
## Setting relative priority
|
||||
|
||||
- Each container has a relative priority used by the Linux scheduler.
|
||||
|
||||
- By default, this priority is 1024.
|
||||
|
||||
- As long as CPU usage is not maxed out, this has no effect.
|
||||
|
||||
- When CPU usage is maxed out, each container receives CPU cycles in proportion of its relative priority.
|
||||
|
||||
- In other words: a container with `--cpu-shares 2048` will receive twice as much than the default.
|
||||
|
||||
---
|
||||
|
||||
## Setting a CPU% limit
|
||||
|
||||
- This setting will make sure that a container doesn't use more than a given % of CPU.
|
||||
|
||||
- The value is expressed in CPUs; therefore:
|
||||
|
||||
`--cpus 0.1` means 10% of one CPU,
|
||||
|
||||
`--cpus 1.0` means 100% of one whole CPU,
|
||||
|
||||
`--cpus 10.0` means 10 entire CPUs.
|
||||
|
||||
---
|
||||
|
||||
## Pinning containers to CPUs
|
||||
|
||||
- On multi-core machines, it is possible to restrict the execution on a set of CPUs.
|
||||
|
||||
- Examples:
|
||||
|
||||
`--cpuset-cpus 0` forces the container to run on CPU 0;
|
||||
|
||||
`--cpuset-cpus 3,5,7` restricts the container to CPUs 3, 5, 7;
|
||||
|
||||
`--cpuset-cpus 0-3,8-11` restricts the container to CPUs 0, 1, 2, 3, 8, 9, 10, 11.
|
||||
|
||||
- This will not reserve the corresponding CPUs!
|
||||
|
||||
(They might still be used by other containers, or uncontainerized processes.)
|
||||
|
||||
---
|
||||
|
||||
## Limiting disk usage
|
||||
|
||||
- Most storage drivers do not support limiting the disk usage of containers.
|
||||
|
||||
(With the exception of devicemapper, but the limit cannot be set easily.)
|
||||
|
||||
- This means that a single container could exhaust disk space for everyone.
|
||||
|
||||
- In practice, however, this is not a concern, because:
|
||||
|
||||
- data files (for stateful services) should reside on volumes,
|
||||
|
||||
- assets (e.g. images, user-generated content...) should reside on object stores or on volume,
|
||||
|
||||
- logs are written on standard output and gathered by the container engine.
|
||||
|
||||
- Container disk usage can be audited with `docker ps -s` and `docker diff`.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
- This was initially written to support in-person, instructor-led workshops and tutorials
|
||||
|
||||
- These materials are maintained by [Jérôme Petazzoni](https://twitter.com/jpetazzo) and [multiple contributors](https://github.com/jpetazzo/container.training/graphs/contributors)
|
||||
- These materials are maintained by [Jérôme Petazzoni](https://twitter.com/jpetazzo) and [multiple contributors](https://@@GITREPO@@/graphs/contributors)
|
||||
|
||||
- You can also follow along on your own, at your own pace
|
||||
|
||||
|
||||
@@ -6,6 +6,10 @@ title: |
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/workshop-yyyymmdd-city)"
|
||||
chat: "In person!"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- self-paced
|
||||
|
||||
@@ -29,6 +33,7 @@ chapters:
|
||||
- kube/kubectlrun.md
|
||||
- - kube/kubectlexpose.md
|
||||
- kube/ourapponkube.md
|
||||
- kube/kubectlproxy.md
|
||||
- kube/dashboard.md
|
||||
- - kube/kubectlscale.md
|
||||
- kube/daemonset.md
|
||||
|
||||
@@ -5,6 +5,10 @@ title: |
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/training-20180413-paris)"
|
||||
chat: "In person!"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- self-paced
|
||||
|
||||
@@ -31,6 +35,7 @@ chapters:
|
||||
- - kube/kubectlrun.md
|
||||
- kube/kubectlexpose.md
|
||||
- kube/ourapponkube.md
|
||||
#- kube/kubectlproxy.md
|
||||
- - kube/dashboard.md
|
||||
- kube/kubectlscale.md
|
||||
- kube/daemonset.md
|
||||
|
||||
@@ -5,6 +5,10 @@ title: |
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/workshop-yyyymmdd-city)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- in-person
|
||||
|
||||
@@ -28,6 +32,7 @@ chapters:
|
||||
- kube/kubectlrun.md
|
||||
- - kube/kubectlexpose.md
|
||||
- kube/ourapponkube.md
|
||||
- kube/kubectlproxy.md
|
||||
- kube/dashboard.md
|
||||
- - kube/kubectlscale.md
|
||||
- kube/daemonset.md
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
- This was initially written by [Jérôme Petazzoni](https://twitter.com/jpetazzo) to support in-person,
|
||||
instructor-led workshops and tutorials
|
||||
|
||||
- Credit is also due to [multiple contributors](https://github.com/jpetazzo/container.training/graphs/contributors) — thank you!
|
||||
- Credit is also due to [multiple contributors](https://@@GITREPO@@/graphs/contributors) — thank you!
|
||||
|
||||
- You can also follow along on your own, at your own pace
|
||||
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# Accessing internal services with `kubectl proxy`
|
||||
|
||||
- `kubectl proxy` runs a proxy in the foreground
|
||||
|
||||
- This proxy lets us access the Kubernetes API without authentication
|
||||
|
||||
(`kubectl proxy` adds our credentials on the fly to the requests)
|
||||
|
||||
- This proxy lets us access the Kubernetes API over plain HTTP
|
||||
|
||||
- This is a great tool to learn and experiment with the Kubernetes API
|
||||
|
||||
- The Kubernetes API also gives us a proxy to HTTP and HTTPS services
|
||||
|
||||
- Therefore, we can use `kubectl proxy` to access internal services
|
||||
|
||||
(Without using a `NodePort` or similar service)
|
||||
|
||||
---
|
||||
|
||||
## Secure by default
|
||||
|
||||
- By default, the proxy listens on port 8001
|
||||
|
||||
(But this can be changed, or we can tell `kubectl proxy` to pick a port)
|
||||
|
||||
- By default, the proxy binds to `127.0.0.1`
|
||||
|
||||
(Making it unreachable from other machines, for security reasons)
|
||||
|
||||
- By default, the proxy only accepts connections from:
|
||||
|
||||
`^localhost$,^127\.0\.0\.1$,^\[::1\]$`
|
||||
|
||||
- This is great when running `kubectl proxy` locally
|
||||
|
||||
- Not-so-great when running it on a remote machine
|
||||
|
||||
---
|
||||
|
||||
## Running `kubectl proxy` on a remote machine
|
||||
|
||||
- We are going to bind to `INADDR_ANY` instead of `127.0.0.1`
|
||||
|
||||
- We are going to accept connections from any address
|
||||
|
||||
.exercise[
|
||||
|
||||
- Run an open proxy to the Kubernetes API:
|
||||
```bash
|
||||
kubectl proxy --port=8888 --address=0.0.0.0 --accept-hosts=.*
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
.warning[Anyone can now do whatever they want with our Kubernetes cluster!
|
||||
<br/>
|
||||
(Don't do this on a real cluster!)]
|
||||
|
||||
---
|
||||
|
||||
## Viewing available API routes
|
||||
|
||||
- The default route (i.e. `/`) shows a list of available API endpoints
|
||||
|
||||
.exercise[
|
||||
|
||||
- Point your browser to the IP address of the node running `kubectl proxy`, port 8888
|
||||
|
||||
]
|
||||
|
||||
The result should look like this:
|
||||
```json
|
||||
{
|
||||
"paths": [
|
||||
"/api",
|
||||
"/api/v1",
|
||||
"/apis",
|
||||
"/apis/",
|
||||
"/apis/admissionregistration.k8s.io",
|
||||
…
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Connecting to a service through the proxy
|
||||
|
||||
- The API can proxy HTTP and HTTPS requests by accessing a special route:
|
||||
```
|
||||
/api/v1/namespaces/`name_of_namespace`/services/`name_of_service`/proxy
|
||||
```
|
||||
|
||||
- Since we now have access to the API, we can use this special route
|
||||
|
||||
.exercise[
|
||||
|
||||
- Access the `hasher` service through the special proxy route:
|
||||
```open
|
||||
http://`X.X.X.X`:8888/api/v1/namespaces/default/services/hasher/proxy
|
||||
```
|
||||
|
||||
]
|
||||
|
||||
You should see the banner of the hasher service: `HASHER running on ...`
|
||||
|
||||
---
|
||||
|
||||
## Stopping the proxy
|
||||
|
||||
- Remember: as it is running right now, `kubectl proxy` gives open access to our cluster
|
||||
|
||||
.exercise[
|
||||
|
||||
- Stop the `kubectl proxy` process with Ctrl-C
|
||||
|
||||
]
|
||||
|
||||
@@ -40,7 +40,12 @@
|
||||
|
||||
## Creating namespaces
|
||||
|
||||
- We can create namespaces with a very minimal YAML, e.g.:
|
||||
- Creating a namespace is done with the `kubectl create namespace` command:
|
||||
```bash
|
||||
kubectl create namespace blue
|
||||
```
|
||||
|
||||
- We can also get fancy and use a very minimal YAML snippet, e.g.:
|
||||
```bash
|
||||
kubectl apply -f- <<EOF
|
||||
apiVersion: v1
|
||||
@@ -50,6 +55,8 @@
|
||||
EOF
|
||||
```
|
||||
|
||||
- The two methods above are identical
|
||||
|
||||
- If we are using a tool like Helm, it will create namespaces automatically
|
||||
|
||||
---
|
||||
|
||||
@@ -133,7 +133,7 @@ That rollout should be pretty quick. What shows in the web UI?
|
||||
|
||||
---
|
||||
|
||||
## Rolling out a boo-boo
|
||||
## Rolling out something invalid
|
||||
|
||||
- What happens if we make a mistake?
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
|
||||
6. Copy the configuration file generated by `kubeadm init`
|
||||
|
||||
- Check the [prepare VMs README](https://github.com/jpetazzo/container.training/blob/master/prepare-vms/README.md) for more details
|
||||
- Check the [prepare VMs README](https://@@GITREPO@@/blob/master/prepare-vms/README.md) for more details
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
## Versions installed
|
||||
|
||||
- Kubernetes 1.10.1
|
||||
- Kubernetes 1.10.3
|
||||
- Docker Engine 18.03.0-ce
|
||||
- Docker Compose 1.20.1
|
||||
- Docker Compose 1.21.1
|
||||
|
||||
|
||||
.exercise[
|
||||
|
||||
@@ -114,6 +114,8 @@ def generatefromyaml(manifest, filename):
|
||||
html = html.replace("@@MARKDOWN@@", markdown)
|
||||
html = html.replace("@@EXCLUDE@@", exclude)
|
||||
html = html.replace("@@CHAT@@", manifest["chat"])
|
||||
html = html.replace("@@GITREPO@@", manifest["gitrepo"])
|
||||
html = html.replace("@@SLIDES@@", manifest["slides"])
|
||||
html = html.replace("@@TITLE@@", manifest["title"].replace("\n", " "))
|
||||
return html
|
||||
|
||||
|
||||
@@ -5,6 +5,10 @@ title: |
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/workshop-yyyymmdd-city)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- self-paced
|
||||
- snap
|
||||
|
||||
@@ -5,6 +5,10 @@ title: |
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
#chat: "[Gitter](https://gitter.im/jpetazzo/workshop-yyyymmdd-city)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- self-paced
|
||||
- snap
|
||||
|
||||
@@ -4,6 +4,10 @@ title: |
|
||||
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- in-person
|
||||
- btp-auto
|
||||
|
||||
@@ -4,6 +4,10 @@ title: |
|
||||
|
||||
chat: "[Slack](https://dockercommunity.slack.com/messages/C7GKACWDV)"
|
||||
|
||||
gitrepo: github.com/jpetazzo/container.training
|
||||
|
||||
slides: http://container.training/
|
||||
|
||||
exclude:
|
||||
- in-person
|
||||
- btp-auto
|
||||
|
||||
@@ -49,7 +49,7 @@ This will display the unlock key. Copy-paste it somewhere safe.
|
||||
]
|
||||
|
||||
Note: if you are doing the workshop on your own, using nodes
|
||||
that you [provisioned yourself](https://github.com/jpetazzo/container.training/tree/master/prepare-machine) or with [Play-With-Docker](http://play-with-docker.com/), you might have to use a different method to restart the Engine.
|
||||
that you [provisioned yourself](https://@@GITREPO@@/tree/master/prepare-machine) or with [Play-With-Docker](http://play-with-docker.com/), you might have to use a different method to restart the Engine.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
- This was initially written by [Jérôme Petazzoni](https://twitter.com/jpetazzo) to support in-person,
|
||||
instructor-led workshops and tutorials
|
||||
|
||||
- Over time, [multiple contributors](https://github.com/jpetazzo/container.training/graphs/contributors) also helped to improve these materials — thank you!
|
||||
- Over time, [multiple contributors](https://@@GITREPO@@/graphs/contributors) also helped to improve these materials — thank you!
|
||||
|
||||
- You can also follow along on your own, at your own pace
|
||||
|
||||
|
||||
@@ -113,7 +113,7 @@ class: elk-manual
|
||||
|
||||
- We could author a custom image bundling this configuration
|
||||
|
||||
- We can also pass the [configuration](https://github.com/jpetazzo/container.training/blob/master/elk/logstash.conf) on the command line
|
||||
- We can also pass the [configuration](https://@@GITREPO@@/blob/master/elk/logstash.conf) on the command line
|
||||
|
||||
.exercise[
|
||||
|
||||
@@ -195,7 +195,7 @@ class: elk-auto
|
||||
Note: the *build* and *push* steps are not strictly necessary, but they don't hurt!
|
||||
|
||||
Let's have a look at the [Compose file](
|
||||
https://github.com/jpetazzo/container.training/blob/master/stacks/elk.yml).
|
||||
https://@@GITREPO@@/blob/master/stacks/elk.yml).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -169,7 +169,7 @@ class: in-person
|
||||
This should tell us that we are talking to `node3`.
|
||||
|
||||
Note: it can be useful to use a [custom shell prompt](
|
||||
https://github.com/jpetazzo/container.training/blob/master/prepare-vms/scripts/postprep.rc#L68)
|
||||
https://@@GITREPO@@/blob/master/prepare-vms/scripts/postprep.rc#L68)
|
||||
reflecting the `DOCKER_HOST` variable.
|
||||
|
||||
---
|
||||
|
||||
@@ -33,7 +33,7 @@ TOKEN=$(docker swarm join-token -q manager)
|
||||
for N in $(seq 2 5); do
|
||||
DOCKER_HOST=tcp://node$N:2375 docker swarm join --token $TOKEN node1:2377
|
||||
done
|
||||
git clone git://github.com/jpetazzo/container.training
|
||||
git clone git://@@GITREPO@@
|
||||
cd container.training/stacks
|
||||
docker stack deploy --compose-file registry.yml registry
|
||||
docker-compose -f dockercoins.yml build
|
||||
|
||||
Reference in New Issue
Block a user