github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-18 04:19:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
aabbc17d97a6422959e79d747fc3ef5f97a903bc
container.training/www/htdocs/index.html
Tom Mortimer-Jones b87ece9acd Typo
2016-04-27 09:37:51 +01:00

5764 lines
107 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html>
<head>
<base target="_blank">
<title>Docker Orchestration Workshop</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<style type="text/css">
@import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
@import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
@import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
body { font-family: 'Droid Serif'; }
h1, h2, h3 {
font-family: 'Yanone Kaffeesatz';
font-weight: normal;
margin-top: 0.5em;
}
a {
text-decoration: none;
color: blue;
}
.remark-slide-content { padding: 1em 2.5em 1em 2.5em; }
.remark-slide-content { font-size: 25px; }
.remark-slide-content h1 { font-size: 50px; }
.remark-slide-content h2 { font-size: 50px; }
.remark-slide-content h3 { font-size: 25px; }
.remark-code { font-size: 25px; }
.small .remark-code { font-size: 16px; }
.remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
.red { color: #fa0000; }
.gray { color: #ccc; }
.small { font-size: 70%; }
.big { font-size: 140%; }
.underline { text-decoration: underline; }
.pic {
vertical-align: middle;
text-align: center;
padding: 0 0 0 0 !important;
}
img {
max-width: 100%;
max-height: 550px;
}
.title {
vertical-align: middle;
text-align: center;
}
.title h1 { font-size: 100px; }
.title p { font-size: 100px; }
.quote {
background: #eee;
border-left: 10px solid #ccc;
margin: 1.5em 10px;
padding: 0.5em 10px;
quotes: "\201C""\201D""\2018""\2019";
font-style: italic;
}
.quote:before {
color: #ccc;
content: open-quote;
font-size: 4em;
line-height: 0.1em;
margin-right: 0.25em;
vertical-align: -0.4em;
}
.quote p {
display: inline;
}
.warning {
background-image: url("warning.png");
background-size: 1.5em;
background-repeat: no-repeat;
padding-left: 2em;
}
.exercise, .notexercise {
background-color: #eee;
background-image: url("keyboard.png");
background-size: 1.4em;
background-repeat: no-repeat;
background-position: 0.2em 0.2em;
border: 2px dotted black;
}
.exercise::before {
content: "Exercise";
margin-left: 1.8em;
}
.notexercise::before {
content: "Actually not an exercise";
color: red;
margin-left: 1.8em;
}
li p { line-height: 1.25em; }
</style>
</head>
<body>
<textarea id="source">
class: title
# Docker <br/> Orchestration <br/> Workshop
---
## Logistics
- Hello! We're `jerome at docker dot com` and `aj at soulshake dot net`
<!--
Reminder, when updating the agenda: when people are told to show
up at 9am, they usually trickle in until 9:30am (except for paid
training sessions). If you're not sure that people will be there
on time, it's a good idea to have a breakfast with the attendees
at e.g. 9am, and start at 9:30.
-->
- Agenda:
.small[
- 08:00-09:00 hello and breakfast
- 09:00:10:25 part 1
- 10:25-10:35 coffee break
- 10:35-12:00 part 2
- 12:00-13:00 lunch break
- 13:00-14:25 part 3
- 14:25-14:35 coffee break
- 14:35-16:00 part 4
]
<!-- - This will be FAST PACED, but DON'T PANIC! -->
- All the content is publicly available (slides, code samples, scripts)
<!--
Remember to change:
- the Gitter link below
- the "tweet my speed" hashtag in DockerCoins HTML
-->
- Experimental chat support on
[Gitter](http://container.training/chat)
---
<!--
grep '^# ' index.html | grep -v '<br' | tr '#' '-'^C
-->
## Outline (1/4)
- Pre-requirements
- VM environment
- Our sample application
- Running the application
- Container port mapping
- Identifying bottlenecks
- Scaling HTTP on a single node
- Put a load balancer on it
- Connecting to containers on other hosts
- Abstracting remote services with ambassadors
- Various considerations about ambassadors
---
## Outline (2/4)
- Docker for ops
- Backups
- Starting more containers from your container
- Docker events stream
- Attaching labels
- Logs
- Storing container logs in an ELK stack
- Security upgrades
- Network traffic analysis
---
## Outline (3/4)
- Dynamic orchestration
- Hands-on Swarm
- Deploying Swarm
- Cluster discovery
- Resource allocation
- Connecting containers with ambassadors
- Setting up Consul and overlay networks
- Multi-host networking
- Building images with Swarm
- Deploying a local registry
- Scaling workers
---
## Outline (4/4)
- Distributing Machine credentials
- Highly available Swarm managers
- Highly available containers
- Conclusions
---
# Pre-requirements
- Computer with network connection and SSH client
- on Linux, OS X, FreeBSD... you are probably all set
- on Windows, get [putty](http://www.putty.org/),
[Git BASH](https://msysgit.github.io/), or
[MobaXterm](http://mobaxterm.mobatek.net/)
- Basic Docker knowledge
<br/>(but that's OK if you're not a Docker expert!)
---
## Nice-to-haves
- [GitHub](https://github.com/join) account
<br/>(if you want to fork the repo; also used to join Gitter)
- [Gitter](https://gitter.im/) account
<br/>(to join the conversation during the workshop)
- [Docker Hub](https://hub.docker.com) account
<br/>(it's one way to distribute images on your Swarm cluster)
---
## Hands-on sections
- The whole workshop is hands-on
- I will show Docker in action
- I invite you to reproduce what I do
- All hands-on sections are clearly identified, like the gray rectangle below
.exercise[
- This is the stuff you're supposed to do!
- Go to [container.training](http://container.training/) to view these slides
- Join the chat room on
[Gitter](http://container.training/chat)
]
---
# VM environment
- Each person gets 5 private VMs (not shared with anybody else)
- They'll be up until tomorrow
- You have a little card with login+password+IP addresses
- You can automatically SSH from one VM to another
.exercise[
<!--
```bash
for N in $(seq 1 5); do
ssh -o StrictHostKeyChecking=no node$N true
done
for N in $(seq 1 5); do
(.
docker-machine rm -f node$N
ssh node$N "docker ps -aq | xargs -r docker rm -f"
ssh node$N sudo rm -f /etc/systemd/system/docker.service
ssh node$N sudo systemctl daemon-reload
echo Restarting node$N.
ssh node$N sudo systemctl restart docker
echo Restarted node$N.
) &
done
wait
```
-->
- Log into the first VM (`node1`)
- Check that you can SSH (without password) to `node2`:
```bash
ssh node2
```
- Type `exit` or `^D` to come back to node1
<!--
```meta
^D
```
-->
]
---
## We will (mostly) interact with node1 only
- Unless instructed, **all commands must be run from the first VM, `node1`**
- We will only checkout/copy the code on `node1`
- When we will use the other nodes, we will do it mostly through the Docker API
- We will use SSH only for a few "out of band" operations (mass-removing containers...)
---
## Terminals
Once in a while, the instructions will say:
<br/>"Open a new terminal."
There are multiple ways to do this:
- create a new window or tab on your machine, and SSH into the VM;
- use screen or tmux on the VM and open a new window from there.
You are welcome to use the method that you feel the most comfortable with.
---
## Tmux cheatsheet
- Ctrl-b c → creates a new window
- Ctrl-b n → go to next window
- Ctrl-b p → go to previous window
- Ctrl-b " → split window top/bottom
- Ctrl-b % → split window left/right
- Ctrl-b Alt-1 → rearrange windows in columns
- Ctrl-b Alt-2 → rearrange windows in rows
- Ctrl-b arrows → navigate to other windows
- Ctrl-b d → detach session
- tmux attach → reattach to session
---
## Brand new versions!
- Engine 1.12
- Compose 1.7
- Swarm 1.2
- Machine 0.6
.exercise[
- Check all installed versions:
```bash
docker version
docker-compose -v
docker run --rm swarm -version
docker-machine -v
```
]
---
## Why are we not using the latest version of Machine?
- The latest version of Machine is 0.7
- The way it deploys Swarm is different from 0.6
- This causes a regression in the strategy that we will use later
- More details later!
---
# Our sample application
- Visit the GitHub repository with all the materials of this workshop:
<br/>https://github.com/jpetazzo/orchestration-workshop
- The application is in the [dockercoins](
https://github.com/jpetazzo/orchestration-workshop/tree/master/dockercoins)
subdirectory
- Let's look at the general layout of the source code:
there is a Compose file [docker-compose.yml](
https://github.com/jpetazzo/orchestration-workshop/blob/master/dockercoins/docker-compose.yml) ...
... and 4 other services, each in its own directory:
- `rng` = web service generating random bytes
- `hasher` = web service computing hash of POSTed data
- `worker` = background process using `rng` and `hasher`
- `webui` = web interface to watch progress
---
## What's this application?
---
class: pic
![DockerCoins logo](dockercoins.png)
(DockerCoins logo courtesy of @jonasrosland. Thanks!)
---
## What's this application?
- It is a DockerCoin miner! 💰🐳📦🚢
- No, you can't buy coffee with DockerCoins
- How DockerCoins works:
- `worker` asks to `rng` to give it random bytes
- `worker` feeds those random bytes into `hasher`
- each hash starting with `0` is a DockerCoin
- DockerCoins are stored in `redis`
- `redis` is also updated every second to track speed
- you can see the progress with the `webui`
---
## Getting the application source code
- We will clone the GitHub repository
- The repository also contains scripts and tools that we will use through the workshop
.exercise[
<!--
```bash
[ -d orchestration-workshop ] && mv orchestration-workshop orchestration-workshop.$$
```
-->
- Clone the repository on `node1`:
```bash
git clone git://github.com/jpetazzo/orchestration-workshop
```
]
(You can also fork the repository on GitHub and clone your fork if you prefer that.)
---
# Running the application
Without further ado, let's start our application.
.exercise[
- Go to the `dockercoins` directory, in the cloned repo:
```bash
cd ~/orchestration-workshop/dockercoins
```
- Use Compose to build and run all containers:
```bash
docker-compose up
```
]
Compose tells Docker to build all container images (pulling
the corresponding base images), then starts all containers,
and displays aggregated logs.
---
## Lots of logs
- The application continuously generates logs
- We can see the `worker` service making requests to `rng` and `hasher`
- Let's put that in the background
.exercise[
- Stop the application by hitting `^C`
<!--
```meta
^C
```
-->
]
- `^C` stops all containers by sending them the `TERM` signal
- Some containers exit immediately, others take longer
<br/>(because they don't handle `SIGTERM` and end up being killed after a 10s timeout)
---
## Restarting in the background
- Many flags and commands of Compose are modeled after those of `docker`
.exercise[
- Start the app in the background with the `-d` option:
```bash
docker-compose up -d
```
- Check that our app is running with the `ps` command:
```bash
docker-compose ps
```
]
`docker-compose ps` also shows the ports exposed by the application.
---
## Connecting to the web UI
- The `webui` container exposes a web dashboard; let's view it
.exercise[
- Open http://[yourVMaddr]:8000/ (from a browser)
]
- The app actually has a constant, steady speed (3.33 coins/second)
- The speed seems not-so-steady because:
- the worker doesn't update the counter after every loop, but up to once per second
- the speed is computed by the browser, checking the counter about once per second
- between two consecutive updates, the counter will increase either by 4, or by 0
---
## Viewing logs
- The `docker-compose logs` command works like `docker logs`
.exercise[
- View all logs since container creation and exit when done:
```bash
docker-compose logs
```
- Stream container logs, starting at the last 10 lines for each container:
```bash
docker-compose logs --tail 10 --follow
```
<!--
```meta
^C
```
-->
]
Tip: use `^S` and `^Q` to pause/resume log output.
---
## Upgrading from Compose 1.6
.warning[The `logs` command has changed between Compose 1.6 and 1.7!]
- Up to 1.6
- `docker-compose logs` is the equivalent of `logs --follow`
- `docker-compose logs` must be restarted if containers are added
- Since 1.7
- `--follow` must be specified explicitly
- new containers are automatically picked up by `docker-compose logs`
---
## Compose file format version
*Particularly relevant if you have used Compose before...*
- Compose 1.6 introduced support for a new Compose file format (aka "v2")
- Services are no longer at the top level, but under a `services` section
- There has to be a `version` key at the top level, with value `"2"` (as a string, not an integer)
- Containers are placed on a dedicated network, making links unnecessary
- There are other minor differences, but upgrade is easy and straightforward
---
## Links, naming, and service discovery
- Containers can have network aliases (resolvable through DNS)
- Compose file version 2 makes each container reachable through its service name
- Compose file version 1 requires "links" sections
- Our code can connect to services using their short name
(instead of e.g. IP address or FQDN)
---
## Example in `worker/worker.py`
![Service discovery](service-discovery.png)
---
## Testing services in isolation
- We will stop the `worker` service, and test `rng` and `hasher` alone
.exercise[
- Stop the `worker` service:
```bash
docker-compose stop worker
```
- Look at the logs of `rng` and `hasher`:
```bash
docker-compose logs --tail 10 --follow
```
<!--
```meta
^Z
```
-->
]
We will let Compose stream the logs,
and open a new terminal.
---
## Compose file location
- What happens if you try to run Compose in `$HOME`?
--
- Compose complains that it cannot find `docker-compose.yml`
- You need to go to the directory containing `docker-compose.yml`
<br/>...or use the `-f` option to give the path to the Compose file
<br/>...or set the `COMPOSE_FILE` environment variable
.exercise[
- Go to the application directory:
```bash
cd ~/orchestration-workshop/dockercoins
```
]
---
## Locating port numbers for `rng` and `hasher`
- We can see the port mapping for our services with:
```bash
docker-compose ps
```
- Both services run on port 80 *in their respective container*
- On the Docker host, they are mapped to ports 8001 and 8002
- They are not exposed on port 80, because the Docker host has only one port 80
- The mapping is done in `docker-compose.yml`
---
# Container port mapping
- `node1`, the Docker host, has only one port 80
- If we give the one and only port 80 to the first
container who asks for it, we are in trouble when
another container needs it
- Default behavior: containers are not "exposed"
<br/>(only reachable by the Docker host and other containers,
through their private address)
- Container network services can be exposed:
- statically (you decide which host port to use)
- dynamically (Docker allocates a host port)
---
## Declaring port mapping
- Directly with the Docker Engine:
```bash
docker run -d -p 8000:80 nginx
docker run -d -p 80 nginx
docker run -d -P nginx
```
- With Docker Compose, in the `docker-compose.yml` file:
```yaml
rng:
…
ports:
- "8001:80"
```
→ port 8001 *on the host* maps to
port 80 *in the container*
---
## Testing the `rng` service
Let's get random bytes of data!
.exercise[
- Check that the service is alive:
<br/>`curl localhost:8001`
- Get 10 bytes of random data:
<br/>`curl localhost:8001/10`
]
If the binary data output messed up your terminal, fix it with `reset`.
---
## Testing the `hasher` service
.exercise[
- Check that the `hasher` service is alive:
```bash
curl localhost:8002
```
- Posting binary data requires some extra flags:
```bash
curl -H "Content-type: application/octet-stream" \
--data-binary hello localhost:8002
```
]
---
## Verifying the computed hash
- This web service is essentially implementing `sha256sum` over HTTP
.exercise[
- Check that `sha256sum` yields the same result:
```bash
echo -n hello | sha256sum
```
]
- We specify `-n` to prevent `echo` from adding a trailing `\n`
<br/>(which would change the SHA digest)
---
## Checking logs
- The tests that we made should show up in the first window
<br/>(where `docker-compose logs` is still running)
- We can now restart the `worker` service
.exercise[
- Start the `worker` container:
```bash
docker-compose start worker
```
]
In the web UI, the graph should go up again.
How can we get that graph to go further up?
---
## Looking at resource usage
- Let's look at CPU, memory, and I/O usage
.exercise[
- run `top` to see CPU and memory usage (you should see idle cycles)
- run `vmstat 3` to see I/O usage (si/so/bi/bo)
<br/>(the 4 numbers should be almost zero, except `bo` for logging)
]
We have available resources.
- Why?
- How can we use them?
---
## Scaling workers on a single node
- Docker Compose supports scaling.red[*]
- Let's scale `worker` and see what happens!
.exercise[
- Start 9 more `worker` containers:
```bash
docker-compose scale worker=10
```
- Check the aggregated logs of those containers
- See the impact on CPU load (with top/htop), and on compute speed (with web UI)
<!--
```bash
sleep 5
killall docker-compose
```
-->
]
.red[*]With some limitations, as we'll see later.
---
# Identifying bottlenecks
- You should have seen a 3x speed bump (not 10x)
- Adding workers didn't result in linear improvement
- *Something else* is slowing us down
--
- ... But what?
--
- The code doesn't have instrumentation
- Let's use state-of-the-art HTTP performance analysis!
<br/>(i.e. good old tools like `ab`, `httping`...)
---
## Measuring latency under load
We will use `httping`.
.exercise[
- Check the latency of `rng`:
```bash
httping -c 10 localhost:8001
```
- Check the latency of `hasher`:
```bash
httping -c 10 localhost:8002
```
]
`rng` has a much higher latency than `hasher`.
---
## Benchmarking in isolation
We will now use `ab`.
.exercise[
- Stop the `worker` containers:
```bash
docker-compose kill worker
```
]
---
## Benchmarking `rng`
We will send 50 requests, but with various levels of concurrency.
.exercise[
- Send 50 requests, with a single sequential client:
```bash
ab -c 1 -n 50 localhost:8001/10
```
- Send 50 requests, with ten parallel clients:
```bash
ab -c 10 -n 50 localhost:8001/10
```
]
---
## Benchmark results for `rng`
- In both cases, the benchmark takes ~5 seconds to complete
- When serving requests sequentially, they each take 100ms
- In the parallel scenario, the latency increased dramatically:
- one request is served in 100ms
- another is served in 200ms
- another is served in 300ms
- ...
- another is served in 1000ms
- What about `hasher`?
---
## Benchmarking `hasher`
We will do the same tests for `hasher`.
The command is slightly more complex, since we need to post random data.
First, we need to put the POST payload in a temporary file.
.exercise[
- Generate 10 bytes of random data:
```bash
curl localhost:8001/10 >/tmp/random
```
]
---
## Benchmarking `hasher`
Once again, we will send 50 requests, with different levels of concurrency.
.exercise[
- Send 50 requests with a sequential client:
```bash
ab -c 1 -n 50 -T application/octet-stream \
-p /tmp/random localhost:8002/
```
- Send 50 requests with 10 parallel clients:
```bash
ab -c 10 -n 50 -T application/octet-stream \
-p /tmp/random localhost:8002/
```
]
---
## Benchmark results for `hasher`
- The sequential benchmarks takes ~5 seconds to complete
- The parallel benchmark takes less than 1 second to complete
- In both cases, each request takes a bit more than 100ms to complete
- Requests are a bit slower in the parallel benchmark
- It looks like `hasher` is better equiped to deal with concurrency than `rng`
---
class: title
Why?
---
## Why does everything take (at least) 100ms?
--
`rng` code:
![RNG code screenshot](delay-rng.png)
--
`hasher` code:
![HASHER code screenshot](delay-hasher.png)
---
class: title
But ...
WHY?!?
---
## Why did we sprinkle this sample app with sleeps?
- Deterministic performance
<br/>(regardless of instance speed, CPUs, I/O...)
--
- Actual code sleeps all the time anyway
--
- When your code makes a remote API call:
- it sends a request;
- it sleeps until it gets the response;
- it processes the response.
---
## Why do `rng` and `hasher` behave differently?
![Equations on a blackboard](equations.png)
--
(Synchronous vs. asynchronous event processing)
---
## How to make `rng` go faster
- Obvious solution: comment out the `sleep` instruction
--
- Unfortunately, in the real world, network latency exists
--
- More realistic solution: use an asynchronous framework
<br/>(e.g. use gunicorn with gevent)
--
- New rule: we can't change the code!
--
- Solution: scale out `rng`
<br/>(dispatch `rng` requests on multiple instances)
---
# Scaling HTTP on a single node
- We can try to scale with Compose:
```bash
docker-compose scale rng=3
```
- This will result into an error, because our Compose file uses an explicit port
- We cannot have multiple containers bound to the same port (here, 8001)
- Compose *tries* to scale anyway
(because on a cluster, you *can* have multiple containers on the same public port)
- Let's remove the explicit port mapping, and see what happens when we scale!
---
## Removing explicit port from the Compose file
- We will replace `8001:80` with just `80`
- This will continue to make the service publicly available,
<br/>but on a port dynamically allocated by Docker
.exercise[
- Edit the `docker-compose.yml` file to change the ports section for `rng`:
```yaml
rng:
...
ports:
- "80"
```
<!--
```edit
cp docker-compose.yml-scaled docker-compose.yml
```
-->
]
Shortcut: `docker-compose.yml-portmap`
---
## Scaling up a network service with Compose
- We changed the definition of the service (by removing the static port mapping),
<br/>so we must execute `docker-compose up` before scaling
.exercise[
- Refresh the `rng` service:
```bash
docker-compose up -d
```
- Scale the `rng` service:
```bash
docker-compose scale rng=3
```
]
---
## Results
- In the web UI, you might see a performance increase ... or maybe not
--
- Since Engine 1.11, we get round-robin DNS records
(i.e. resolving `rng` will yield the IP addresses of all 3 containers)
- Docker randomizes the records it sends
- But many resolvers will sort them in unexpected ways
- Depending on various factors, you could get:
- all traffic on a single container
- traffic perfectly balanced on all containers
- traffic unevenly balanced across containers
---
## Assessing DNS randomness
- Let's see how our containers resolve DNS requests
.exercise[
- On each of our 10 scaled workers, execute 5 ping requests:
```bash
for N in $(seq 1 10); do
echo PING__________$N
for I in $(seq 1 5); do
docker exec -ti dockercoins_worker_$N ping -c1 rng
done
done | grep PING
```
]
(The 7th Might Surprise You!)
---
## DNS randomness
- Other programs can yield different results
- Same program on another distro can yield different results
- Same source code with another libc or resolver can yield different results
- Running the same test at different times can yield different results
- Did I mention that Your Results May Vary?
---
## Implementing fair load balancing
- Instead of relying on DNS round robin, let's use a proper load balancer
- Create multiple identical `rng` services
- Put a load balancer in front of them
- Point other services to the load balancer
---
## Scaling `rng`
.exercise[
- Replace the `rng` service with multiple copies of it:
```yaml
rng1:
build: rng
rng2:
build: rng
rng3:
build: rng
```
]
That's all!
Shortcut: `docker-compose.yml-scaled-rng`
---
## Introduction to `jpetazzo/hamba`
- General purpose load balancer and traffic director
- [Source code is available on GitHub](
https://github.com/jpetazzo/hamba)
- [Public image is available on the Docker Hub](
https://hub.docker.com/r/jpetazzo/hamba/)
- Generates a configuration file for HAProxy, then starts HAProxy
- Parameters are provided on the command line; for instance:
```bash
docker run -d -p 80 jpetazzo/hamba 80 www1:1234 www2:2345
docker run -d -p 80 jpetazzo/hamba 80 www1 1234 www2 2345
```
Those two commands do the same thing: they start a load balancer
listening on port 80, and balancing traffic across www1:1234 and www2:2345
---
# Put a load balancer on it
Let's add our load balancer to the Compose file.
.exercise[
- Add the following section to the Compose file:
```yaml
rng:
image: jpetazzo/hamba
command: 80 rng1 80 rng2 80 rng3 80
ports:
- "8001:80"
```
<!--
```edit
cp docker-compose.yml-scaled-rng docker-compose.yml
```
-->
]
Shortcut: `docker-compose.yml-scaled-rng`
---
## Start the whole stack
.exercise[
- Scale back `rng` to a single instance:
<br/>`docker-compose scale rng=1`
- Start the new services:
<br/>`docker-compose up -d`
- Check worker logs:
<br/>`docker-compose logs worker`
- Check load balancer logs:
<br/>`docker-compose logs rng`
]
.warning[If you get errors, see next slide.]
---
## Recovering from errors
- If you scale a service that has an explicit port mapping,
you can end up with extra containers that are *created*,
but not *started*, and will tie up resources
- Those containers can prevent other containers from starting
- If that happens, just remove those containers; for instance:
```bash
docker-compose kill rng && docker-compose rm -f rng
```
- After removing the containers, you can `docker-compose up -d`
again and everything should work fine.
---
## Results
- Check the latency of `rng`
<br/>(it should have improved significantly!)
- Check the application performance in the web UI
<br/>(it should improve if you have enough workers)
---
## The good, the bad, the ugly
--
- The good
We scaled a service, added a load balancer - without changing a single line of code.
--
- The bad
We manually copy-pasted sections in `docker-compose.yml`.
Improvement: write scripts to transform the YAML file.
--
- The ugly
If we scale up/down, we have to restart everything.
Improvement: reconfigure the load balancer dynamically.
---
## Cleaning up
- Before moving forward, let's clean up all those containers
.exercise[
- Use the `down` command to stop and remove containers:
```bash
docker-compose down
```
]
---
# Connecting to containers on other hosts
- So far, our whole stack is on a single machine
- We want to scale out (across multiple nodes)
- We will deploy the same stack multiple times
- But we want every stack to use the same Redis
<br/>(in other words: Redis is our only *stateful* service here)
--
- And remember: we're not allowed to change the code!
- the code connects to host `redis`
- `redis` must resolve to the address of our Redis service
- the Redis service must listen on the default port (6379)
---
# Using custom DNS mapping
- We could setup a Redis server on its default port
- And add a DNS entry mapping `redis` to this server
.exercise[
- See what happens if we run:
```bash
docker run --add-host redis:1.2.3.4 alpine ping redis
```
<!--
```meta
^C
```
-->
]
There is a Compose file option for that: `extra_hosts`.
---
# Abstracting remote services with ambassadors
- What if we can't/won't run Redis on its default port?
- What if we want to be able to move it easily?
--
- We will use an ambassador
- Redis will be started independently of our stack
- It will run at an arbitrary location (host+port)
- In our stack, we replace `redis` with an ambassador
- The ambassador will connect to Redis
- The ambassador will "act as" Redis in the stack
---
class: pic
![Ambassador principle](ambassadors/principle-1.png)
---
class: pic
![Ambassador principle](ambassadors/principle-2.png)
---
## Start redis
- Start a standalone Redis container
- Let Docker expose it on a random port
.exercise[
- Run redis with a random public port:
<br/>`docker run -d -P --name myredis redis`
- Check which port was allocated:
<br/>`docker port myredis 6379`
]
- Note the IP address of the machine, and this port
---
## Update `docker-compose.yml`
.exercise[
- Replace `redis` with an ambassador using `jpetazzo/hamba`:
```yaml
redis:
image: jpetazzo/hamba
command: 6379 `AA.BB.CC.DD:EEEEE`
```
<!--
```edit
cat docker-compose.yml-ambassador | sed "s/AA.BB.CC.DD/$(curl myip.enix.org/REMOTE_ADDR)/" | sed "s/EEEEE/$(docker port myredis 6379 | cut -d: -f2)/" > docker-compose.yml
```
-->
]
Shortcut: `docker-compose.yml-ambassador`
<br/>(But you still have to update `AA.BB.CC.DD:EEEEE`!)
---
## Start the stack on the first machine
- Compose will detect the change in the `redis` service
- It will replace `redis` with a `jpetazzo/hamba` instance
.exercise[
- Just tell Compose to do its thing:
<br/>`docker-compose up -d`
- Check that the stack is up and running:
<br/>`docker-compose ps`
- Look at the web UI to make sure that it works fine
]
---
## Controlling other Docker Engines
- Many tools in the ecosystem will honor the `DOCKER_HOST` environment variable
- Those tools include (obviously!) the Docker CLI and Docker Compose
- Our training VMs have been setup to accept API requests on port 55555
<br/>(without authentication - this is very insecure, by the way!)
- We will see later how to setup mutual authentication with certificates
---
## Setting the `DOCKER_HOST` environment variable
.exercise[
- Check how many containers are running on `node1`:
```bash
docker ps
```
- Set the `DOCKER_HOST` variable to control `node2`, and compare:
```bash
export DOCKER_HOST=tcp://node2:55555
docker ps
```
]
You shouldn't see any container running on `node2` at this point.
---
## Start the stack on another machine
- We will tell Compose to bring up our stack on the other node
- It will use the local code (we don't need to checkout the code on `node2`)
.exercise[
- Start the stack:
```bash
docker-compose up -d
```
- Check that it's running:
```bash
docker-compose ps
```
]
---
## Run the application on every node
- We will repeat the previous step with a little shell loop
... but introduce parallelism to save some time
.exercise[
- Deploy one instance of the stack on each node:
```bash
for N in 3 4 5; do
DOCKER_HOST=tcp://node$N:55555 docker-compose up -d &
done
wait
```
]
Note: building the stack everywhere is not optimal. We will see later
how to build once, and deploy the same build everywhere.
---
## Scale!
- The app is built (and running!) everywhere
- Scaling can be done very quickly
.exercise[
- Add a bunch of workers all over the place:
```bash
for N in 1 2 3 4 5; do
DOCKER_HOST=tcp://node$N:55555 docker-compose scale worker=10
done
```
- Admire the result in the web UI!
]
---
## Social Media Moment
Let's celebrate our success!
(And the fact that we're just 2498349893849283948982 DockerCoins away from being able to afford a cup of coffee!)
.exercise[
- If you have a Twitter account, tweet your mining speed!
</br>(use the "Tweet this!" link below the graph☺)
]
---
## A few words about development volumes
- Try to access the web UI on another node
--
- It doesn't work! Why?
--
- Static assets are masked by an empty volume
--
- We need to comment out the `volumes` section
---
## Why must we comment out the `volumes` section?
- Volumes have multiple uses:
- storing persistent stuff (database files...)
- sharing files between containers (logs, configuration...)
- sharing files between host and containers (source...)
- The `volumes` directive expands to an host path:
`/home/docker/orchestration-workshop/dockercoins/webui/files`
- This host path exists on the local machine (not on the others)
- This specific volume is used in development (not in production)
---
## Stop the app (but leave Redis running)
- Let's use `docker-compose down`
- It will stop and remove the DockerCoins app (but leave other containers running)
.exercise[
- We can do another simple parallel shell loop:
```bash
for N in $(seq 1 5); do
export DOCKER_HOST=tcp://node$N:55555
docker-compose down &
done
wait
```
]
(We need to keep the `myredis` container for
our next section, which will be about backups!)
---
# Various considerations about ambassadors
- "But, ambassadors are adding an extra hop!"
--
- Yes, but if you need load balancing, you need that hop
- Ambassadors actually *save* one hop
<br/>(they act as local load balancers)
- traditional load balancer:
<br/>client ⇒ external LB ⇒ server (2 physical hops)
- ambassadors:
<br/>client → ambassador ⇒ server (1 physical hop)
--
- Ambassadors are more reliable than traditional LBs
<br/>(they are colocated with their clients)
---
## Inconvenients of ambassadors
- Generic issues
<br/>(shared with any kind of load balancing / HA setup)
- extra logical hop (not transparent to the client)
- must assess backend health
- one more thing to worry about (!)
- Specific issues
- load balancing fairness
High-end load balancing solutions will rely on back pressure
from the backends. This addresses the fairness issue.
---
## There are many ways to deploy ambassadors
"Ambassador" is a design pattern.
There are many ways to implement it.
We will present three increasingly complex (but also powerful)
ways to deploy ambassadors.
---
## Single-tier ambassador deployment
- One-shot configuration process
- Must be executed manually after each scaling operation
- Scans current state, updates load balancer configuration
- Pros:
<br/>- simple, robust, no extra moving part
<br/>- easy to customize (thanks to simple design)
<br/>- can deal efficiently with large changes
- Cons:
<br/>- must be executed after each scaling operation
<br/>- harder to compose different strategies
- Example: this workshop
---
## Two-tier ambassador deployment
- Daemon listens to Docker events API
- Reacts to container start/stop events
- Adds/removes back-ends to load balancers configuration
- Pros:
<br/>- no extra step required when scaling up/down
- Cons:
<br/>- extra process to run and maintain
<br/>- deals with one event at a time (ordering matters)
- Hidden gotcha: load balancer creation
- Example: interlock
---
## Three-tier ambassador deployment
- Daemon listens to Docker events API
- Reacts to container start/stop events
- Adds/removes scaled services in distributed config DB (zookeeper, etcd, consul…)
- Another daemon listens to config DB events,
<br/>adds/removes backends to load balancers configuration
- Pros:
<br/>- more flexibility
- Cons:
<br/>- three extra services to run and maintain
- Example: registrator
---
## Ambassadors and overlay networks
- Overlay networks allow direct multi-host communication
- Ambassadors are still useful to implement other tasks:
- load balancing;
- credentials injection;
- instrumentation;
- fail-over;
- etc.
---
class: title
# Interlude <br/>
# Docker for ops
---
# Backups
- Redis is still running (with name `myredis`)
- We want to enable backups without touching it
- We will use a special backup container:
- sharing the same volumes
- linked to it (to connect to it easily)
- possibly containing our backup tools
- This works because the `redis` container image stores its data on a volume
---
## Starting the backup container
.exercise[
- Make sure you're talking to the initial host:
```bash
unset DOCKER_HOST
```
<!--
```meta
^{
```
-->
- Start the container:
```bash
docker run --link myredis:redis \
--volumes-from myredis \
-v /tmp/myredis:/output \
-ti alpine sh
```
- Look in `/data` in the container (that's where Redis puts its data dumps)
]
---
## Connecting to Redis
- We need to tell Redis to perform a data dump *now*
.exercise[
- Connect to Redis:
```bash
telnet redis 6379
```
- Issue commands `SAVE` then `QUIT`
- Look at `/data` again (notice the time stamps)
]
- There should be a recent dump file now!
---
## Getting the dump out of the container
- We could use many things:
- s3cmd to copy to S3
- SSH to copy to a remote host
- gzip/bzip/etc before copying
- We'll just copy it to the Docker host
.exercise[
- Copy the file from `/data` to `/output`
- Exit the container
- Look into `/tmp/myredis` (on the host)
<!--
```meta
^}
```
-->
]
---
## Scheduling backups
In the "old world," we (generally) use cron.
With containers, what are our options?
--
- run `cron` on the Docker host, and put `docker run` in the crontab
--
- run `cron` in the backup container, and make sure it keeps running
<br/>(e.g. with `docker run --restart=…`)
--
- run `cron` in a container, and start backup containers from there
--
- listen to the Docker events stream, automatically scheduling backups
<br/>when database containers are started
---
# Starting more containers from your container
- In a local environment, just bind-mount the Docker control socket:
```bash
docker run -ti -v /var/run/docker.sock:/var/run/docker.sock docker
```
- Otherwise, you have to:
- set `DOCKER_HOST`,
- set `DOCKER_TLS_VERIFY` and `DOCKER_CERT_PATH` (if you use TLS),
- copy certificates to the container that will need API access.
More resources on this topic:
- [Do not use Docker-in-Docker for CI](
http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/)
- [One container to rule them all](
http://jpetazzo.github.io/2016/04/03/one-container-to-rule-them-all/)
---
# Docker events stream
- Using the Docker API, we can get real-time
notifications of everything happening in the Engine:
- container creation/destruction
- container start/stop
- container exit/signal/out of memory
- container attach/detach
- volume creation/destruction
- network creation/destruction
- connection/disconnection of containers
(Networks will be covered a bit later!)
---
## Subscribing to the events stream
- This is done with `docker events`
.exercise[
- Get a stream of events:
```bash
docker events
```
<!--
```meta
^Z
```
-->
- In a new terminal, do *anything*:
```bash
docker run --rm alpine sleep 10
```
]
You should see events for the lifecycle of the
container, as well as its connection/disconnection
to the default `bridge` network.
---
# Attaching labels
- You can attach arbitrary labels to engines and containers
- You can read the value of those labels
- You can use those labels as filters in some commands
.exercise[
- Start two containers, with and without a `backup` label:
```bash
docker run -d --name leweb nginx
docker run -d --name ledata --label backup=hourly redis
```
<!--
```meta
^K
```
-->
]
---
## Using labels as filters
- `docker ps` can take a `--filter` argument
.exercise[
- List only containers that have a `backup` label:
```bash
docker ps --filter label=backup
```
- List only containers where the `backup` label
has a specific value:
```bash
docker ps --filter label=backup=hourly
```
]
---
## Filtering events
- On a large cluster, there will be *lots* of events
<br/>(especially when using short-lived containers)
- `docker events` can also take a `--filter` argument
.exercise[
- Show events only for containers with a "backup" label:
```bash
docker events --filter label=backup
```
<!--
```meta
^Z
```
-->
- In a different terminal, terminate our containers:
```bash
docker kill leweb ledata
```
<!--
```meta
^K
```
-->
]
---
## Using `docker ps` in scripts
- The default output of `docker ps` has two flaws:
- it is not machine-readable
- some information is not shown
- This can be changed with the `--format` flag
.exercise[
- List containers that have a `backup` label;
<br/>show their container ID, image, and the label:
```bash
docker ps --filter label=backup \
--format '{{ .ID }} {{ .Image }} {{ .Label "backup" }}'
```
]
---
# Logs
- Two strategies:
- log to plain files on volumes
- log to stdout
<br/>(and use a logging driver)
---
## Logging to plain files on volumes
(Sorry, that part won't be hands-on!)
- Start a container with `-v /logs`
- Make sure that all log files are in `/logs`
- To check logs, run e.g.
```bash
docker run --volumes-from ... ubuntu sh -c "grep WARN /logs/*.log"
```
- Or just go interactive:
```bash
docker run --volumes-from ... -ti ubuntu
```
- You can (should) start a log shipper that way
---
## Logging to stdout
- All containers should write to stdout/stderr
- Docker will collect logs and pass them to a logging driver
- Logging driver can specified globally, and per container
<br/>(changing it for a container overrides the global setting)
- To change the global logging driver, pass extra flags to the daemon
<br/>(requires a daemon restart)
- To override the logging driver for a container, pass extra flags to `docker run`
---
## Specifying logging flags
- `--log-driver`
*selects the driver*
- `--log-opt key=val`
*adds driver-specific options*
<br/>*(can be repeated multiple times)*
- The flags are identical for `docker daemon` and `docker run`
---
## Logging flags in practice
- If you provision your nodes with Docker Machine,
you can set global logging flags (which will apply to all
containers started by a given Engine) like this:
```bash
docker-machine create ... --engine-opt log-driver=...
```
- Otherwise, use your favorite method to edit or manage configuration files
- You can set per-container logging options in Compose files
---
## Available drivers
- json-file (default)
- syslog (can send to UDP, TCP, TCP+TLS, UNIX sockets)
- awslogs (AWS CloudWatch)
- journald
- gelf
- fluentd
- splunk
---
## About json-file ...
- It doesn't rotate logs by default, so your disks will fill up
(Unless you set `maxsize` *and* `maxfile` log options.)
- It's the only one supporting logs retrieval
(If you want to use `docker logs`, `docker-compose logs`,
or fetch logs from the Docker API, you need json-file!)
- This might change in the future
(But it's complex since there is no standard protocol
to *retrieve* log entries.)
All about logging in the documentation:
https://docs.docker.com/reference/logging/overview/
---
# Storing container logs in an ELK stack
*Important foreword: this is not an "official" or "recommended"
setup; it is just an example. We do not endorse ELK, GELF,
or the other elements of the stack more than others!*
What we will do:
- Spin up an ELK stack, with Compose
- Gaze at the spiffy Kibana web UI
- Manually send a few log entries over GELF
- Reconfigure our DockerCoins app to send logs to ELK
---
## What's in an ELK stack?
- ELK is three components:
- ElasticSearch (to store and index log entries)
- Logstash (to receive log entries from various
sources, process them, and forward them to various
destinations)
- Kibana (to view/search log entries with a nice UI)
- The only component that we will configure is Logstash
- We will accept log entries using the GELF protocol
- Log entries will be stored in ElasticSearch,
<br/>and displayed on Logstash's stdout for debugging
---
## Starting our ELK stack
- We will use a *separate* Compose file
- The Compose file is in the `elk` directory
.exercise[
- Go to the `elk` directory:
```bash
cd ~/orchestration-workshop/elk
```
- Start the ELK stack:
```bash
docker-compose up -d
```
]
---
## Checking that our ELK stack works
- Our default Logstash configuration sends a test
message every minute
- All messages are stored into ElasticSearch,
but also shown on Logstash stdout
.exercise[
- Look at Logstash stdout:
```bash
docker-compose logs logstash
```
]
After less than one minute, you should see a `"message" => "ok"`
in the output.
---
## Connect to Kibana
- Our ELK stack exposes two public services:
<br/>the Kibana web server, and the GELF UDP socket
- They are both exposed on their default port numbers
<br/>(5601 for Kibana, 12201 for GELF)
.exercise[
- Open the UI in your browser: http://instance-address:5601/
]
---
## "Configuring" Kibana
- If you see a status page with a yellow item, wait a minute and reload
(Kibana is probably still initializing)
- Kibana should offer you to "Configure an index pattern",
just click the "Create" button
- Then:
- click "Discover" (in the top-left corner)
- click "Last 15 minutes" (in the top-right corner)
- click "Last 1 hour" (in the list in the middle)
- click "Auto-refresh" (top-right corner)
- click "5 seconds" (top-left of the list)
- You should see a series of green bars (with one new green bar every minute)
---
![Screenshot of Kibana](kibana.png)
---
## Sending container output to Kibana
- We will create a simple container displaying "hello world"
- We will override the container logging driver
- The GELF address is `127.0.0.1:12201`, because the Compose file
explicitly exposes the GELF socket on port 12201
.exercise[
- Start our one-off container:
```bash
docker run --rm --log-driver gelf \
--log-opt gelf-address=udp://127.0.0.1:12201 \
alpine echo hello world
```
]
---
## Visualizing container logs in Kibana
- Less than 5 seconds later (the refresh rate of the UI),
the log line should be visible in the web UI
- We can customize the web UI to be more readable
.exercise[
- In the left column, move the mouse over the following
columns, and click the "Add" button that appears:
- host
- container_name
- message
]
---
## Removing the old deployment of DockerCoins
- Before redeploying DockerCoins, remove everything
.exercise[
- Go back to the dockercoins directory:
<br/>`cd ~/orchestration-workshop/dockercoins`
- Stop and remove all DockerCoins containers:
<br/>`docker-compose kill && docker-compose down --remove-orphans`
- Reset the Compose file:
<br/>`git checkout docker-compose.yml`
- Point the Docker API to a single node:
<br/>`unset DOCKER_HOST`
]
---
## Add the logging driver to the Compose file
- We need to add the logging section to each container
- We need the GELF endpoint that we
got earlier with `docker-compose ps logstash`
.exercise[
- Edit the `docker-compose.yml` file, adding the following lines **to each container**:
```yaml
logging:
driver: gelf
options:
gelf-address: "udp://127.0.0.1:12201"
```
<!--
```edit
cp docker-compose.yml-logging docker-compose.yml
```
-->
]
Shortcut: `docker-compose.yml-logging`
---
## Start the DockerCoins app
.exercise[
- Use Compose normally:
```bash
docker-compose up -d
```
]
If you look in the Kibana web UI, you will see log lines
refreshed every 5 seconds.
Note: to do interesting things (graphs, searches...) we
would need to create indexes. This is beyond the scope
of this workshop.
---
## Logging in production
- If we were using an ELK stack:
- scale ElasticSearch
- scale Logstash
- move away from UDP *or* put one Logstash per node
- interpose a Redis or Kafka queue
- Configure your Engines to send all logs to ELK by default
- Start the logging containers with a different logging system
<br/>(to avoid a logging loop)
- Make sure you don't end up writing *all logs* on the nodes running Logstash!
---
# Security upgrades
- This section is not hands-on
- Public Service Announcement
- We'll discuss:
- how to upgrade the Docker daemon
- how to upgrade container images
---
## Upgrading the Docker daemon
- Stop all containers cleanly
- Stop the Docker daemon
- Upgrade the Docker daemon
- Start the Docker daemon
- Start all containers
- This is like upgrading your Linux kernel, but it will get better
(Docker Engine 1.11 is using containerd, which will ultimately allow seamless upgrades.)
---
## In practice
- Keep track of running containers before stopping the Engine:
```bash
docker ps --no-trunc -q |
tee /tmp/running |
xargs -n1 -P10 docker stop
```
- Restart those containers after the Engine is running again:
```bash
xargs docker start < /tmp/running
```
<br/>(Run this multiple times if you have linked containers!)
---
## Upgrading container images
- When a vulnerability is announced:
- if it affects your base images: make sure they are fixed first
- if it affects downloaded packages: make sure they are fixed first
- re-pull base images
- rebuild
- restart containers
---
## How do we know when to upgrade?
- Subscribe to CVE notifications
- https://cve.mitre.org/
- your distros' security announcements
- Check CVE status in official images
<br/>(tag [cve-tracker](
https://github.com/docker-library/official-images/labels/cve-tracker)
in [docker-library/official-images](
https://github.com/docker-library/official-images/labels/cve-tracker)
repo)
- Coming soon: Project Nautilus
<br/>(see [this DC15EU presentation](
http://www.slideshare.net/Docker/official-repos-and-project-nautilus/26))
---
## Upgrading with Compose
Compose makes this particularly easy:
```bash
docker-compose build --pull --no-cache
docker-compose up -d
```
This will automatically:
- pull base images;
- rebuild all container images;
- bring up the new containers.
Remember: Compose will automatically move our
volumes to the new containers, so data is preserved.
---
# Network traffic analysis
- We still have `myredis` running
- We will use *shared network namespaces* to perform network analysis
- Two containers sharing the same network namespace...
- have the same IP addresses
- have the same network interfaces
- `eth0` is therefore the same in both containers
---
## Install and start `ngrep`
Ngrep uses libpcap (like tcpdump) to sniff network traffic.
.exercise[
<!--
```meta
^{
```
-->
- Start a container with the same network namespace:
<br/>`docker run --net container:dockercoins_redis_1 -ti alpine sh`
- Install ngrep:
<br/>`apk update && apk add ngrep`
- Run ngrep:
<br/>`ngrep -tpd eth0 -Wbyline . tcp`
<!--
```meta
^}
```
-->
]
You should see a stream of Redis requests and responses.
---
class: title
# Dynamic orchestration
---
## Static vs Dynamic
- Static
- you decide what goes where
- simple to describe and implement
- seems easy at first but doesn't scale efficiently
- Dynamic
- the system decides what goes where
- requires extra components (HA KV...)
- scaling can be finer-grained, more efficient
---
## Mesos (overview)
- First presented in 2009
- Initial goal: resource scheduler (two-level/pessimistic)
- top-level "master" knows the global cluster state
- "slave" nodes report status and resources to master
- master allocates resources to "frameworks"
- Container support added recently (had to fit existing model)
- Network and service discovery is complex
---
## Mesos (in practice)
- Super easy to setup a test cluster
<br/>(e.g. [minimesos](https://minimesos.org/) put all Mesos components in container)
- Great to accommodate mixed workloads
<br/>(see Marathon, Chronos, Aurora, and many more)
- "Meh" if you only want to run Docker containers
- In production on clusters of thousands of nodes
- Open source project; commercial support available
---
## Kubernetes (overview)
- Started in June 2014
- Designed specifically as a platform for containers ("greenfield" design)
- "pods" = groups of containers sharing network/storage
- Scaling and HA managed by "replication controllers"
- extensive use of "tags" instead of e.g. tree hierarchy
- Initially designed around Docker, but doesn't hesitate to diverge in a few places
---
## Kubernetes (in practice)
- Network and service discovery is powerful, but complex
<br/>.small[(different mechanisms within pod, between pods, for inbound traffic...)]
- Initially designed around GCE
<br/>.small[(networking and persistence work better on GCE, but this is getting better)]
- Tends to be loved by ops more than devs
<br/>.small[(but keep in mind that it's evolving quite as fast as Docker)]
- Setting up a local development cluster is non-trivial
<br/>.small[(the easiest way is to stage it up on e.g. GKE)]
- Bottom line: Kubernetes is not Docker!
<br/>.small[(different concepts, APIs, tooling, concepts, configuration files...)]
---
## ECS (overview)
- Amazon EC2 Container Service
- "Bring your own instance"
- "Native" container scheduler on AWS
- Some integration with other AWS products
- No extra component to operate
- Defines new concepts:
- task
- task definition
- service
---
## ECS (in practice)
- Task definitions look like Compose files, but are significantly different
- Integration with e.g. ELB is suboptimal
<br/>(ELB requires all backends to run on the same port)
- Cluster deployment is made easier thanks to ECS CLI
- Docker API gets partially exposed through ECS API, with some features lagging behind
- Service discovery is painful
---
## Nomad (overview)
- Generic job scheduler
<br/>(not only for containers)
- Desired state is stored in Consul
- Nodes pull jobs from Consul
- Scheduling happens in parallel
---
## Nomad (in practice)
*Disclaimer: I have little first-hand experience with Nomad!*
- Does only one thing, but does it really well
- Works with jobs, not applications, services, etc.
- As I understand it: Nomad is an excellent building block,
<br/>but you need to add other components to deploy your apps
---
## Swarm (in theory)
- Consolidates multiple Docker hosts into a single one
- You talk to Swarm using the Docker API
→ you can use all existing tools: Docker CLI, Docker Compose, etc.
- Swarm talks to your Docker Engines using the Docker API too
→ you can use existing Engines without modification
- Dispatches (schedules) your containers across the cluster, transparently
- Open source and written in Go (like Docker)
- Started by two of the original Docker authors
([@aluzzardi](https://twitter.com/aluzzardi) and [@vieux](https://twitter.com/vieux))
---
## Swarm (in practice)
- Stable since November 2015
- Initially for small setups, easy to setup
- Tested with 1000 nodes + 50000 containers
<br/>.small[(without particular tuning; see DockerCon EU opening keynotes!)]
- Still easy to setup, but scales perfectly to larger deployments too
- Requires a key/value store for advanced features
- We'll see it in action!
???
## PAAS on Docker
- The PAAS workflow: *just push code*
<br/>(inspired by Heroku, dotCloud...)
- TL,DR: easier for devs, harder for ops,
<br/>some very opinionated choices
- A few examples:
<br/>(Non-exhaustive list!!!)
- Cloud Foundry
- Deis
- Dokku
- Flynn
- Tsuru
*Docker made it very easy to cobble your own PAAS.*
???
## A few other tools
- Volume plugins (Convoy, Flocker...)
- manage/migrate stateful containers (and more)
- Network plugins (Contiv, Weave...)
- overlay network so that containers can ping each other
- Docker Cloud (Tutum), Docker UCP (Universal Control Plane)
- dashboards to manage fleets of Docker hosts
... And many more!
---
class: title
# Hands-on Swarm
---
class: pic
![Swarm Logo](swarm.png)
---
## Setting up our Swarm cluster
- This can be done manually or with **Docker Machine**
- Manual deployment:
- with TLS: certificate generation is painful
<br/>(needs dual-use certs)
- without TLS: easier, but insecure
<br/>(unless you run on your internal/private network)
- Docker Machine deployment:
- generates keys, certificates, and deploys them for you
- can also create VMs
---
## The Way Of The Machine
- Install `docker-machine` (single binary download)
- Set a few environment variables (cloud credentials)
- Create one or more machines:
<br/>`docker-machine create -d digitalocean node42`
- List machines and their status:
<br/>`docker-machine ls`
- Select a machine for use:
<br/>`eval $(docker-machine env node42)`
<br/>(this will set a few environment variables)
- Execute regular commands with Docker, Compose, etc.
<br/>(they will pick up remote host address from environment)
---
## Docker Machine `generic` driver
- Most drivers work the same way:
- use cloud API to create instance
- connect to instance over SSH
- install Docker
- The `generic` driver skips the first step
- It can install Docker on any machine, as long as you have SSH access
- We will use that!
---
# Deploying Swarm
- Components involved:
- cluster discovery mechanism
<br/>(so that the manager can learn about the nodes)
- Swarm manager
<br/>(your frontend to the cluster)
- Swarm agent
<br/>(runs on each node, registers it with service discovery)
---
# Cluster discovery
- Possible backends:
- dynamic, self-hosted
<br/>(requires to run a Consul/Etcd/Zookeeper cluster)
- static, through command-line or file
<br/>(great for testing, or for private subnets, see [this article](
https://medium.com/on-docker/docker-swarm-flat-file-engine-discovery-2b23516c71d4#.6vp94h5wn)
- external, token-based
<br/>(dynamic; nothing to operate; relies on external service operated by Docker Inc.)
- We will use the token mechanism
---
## Generating our Swarm discovery token
The token is a unique identifier, corresponding to a bucket
in the discovery service hosted by Docker Inc.
(You can consider it as a rendez-vous point for your cluster.)
.exercise[
- Create your token, saving it preciously to disk as well:
```bash
TOKEN=$(docker run swarm create | tee ~/token)
echo $TOKEN
```
]
---
## Swarm agent
- Used only for dynamic discovery (zk, etcd, consul, token)
- Must run on each node
- Every 20s (by default), tells to the discovery system:
</br>"Hello, there is a Swarm node at A.B.C.D:EFGH"
- Must know the node's IP address
<br/>(sorry, it can't figure it out by itself, because
<br/>it doesn't know whether to use public or private addresses)
- The node continues to work even if the agent dies
- Automatically started by Docker Machine (when the `--swarm` option is passed)
---
## Swarm manager
- Accepts Docker API requests
- Communicates with the cluster nodes
- Performs healthchecks, scheduling...
---
## Pre-flight checks
- According to data gathered during previous workshops, the most
common ways to get the next part wrong are:
- provisioning from the wrong node
- provisioning from a terminal where `$TOKEN` is not set
.exercise[
- Double-check that you are on `node1`
- Double-check that the `$TOKEN` environment variable is set:
```bash
echo $TOKEN
```
]
---
## Provision our first node with Docker Machine
- If `$TOKEN` is set properly, we can move forward!
.exercise[
<!--
```placeholder
AA.BB.CC.DD $(getent hosts node1 | awk '{print $1}')
```
-->
- Invoke Docker Machine with *all the flags*:
```bash
echo $TOKEN
docker-machine create --driver generic \
--swarm --swarm-master --swarm-discovery token://$TOKEN \
--generic-ssh-user docker --generic-ip-address `AA.BB.CC.DD` node1
```
]
(Don't forget to replace AA.BB.CC.DD with the node IP address!)
---
## Redundancy
- The manager is a SPOF
- If you lose the manager:
- you can't control the cluster anymore
- you can still control individual nodes
- you can start a new manager (at this point, it is stateless)
- We'll setup active/passive redundancy later
---
## Check our node
Let's connect to the node *individually*.
.exercise[
- Select the node with Machine
```bash
eval $(docker-machine env node1)
```
- Execute some Docker commands
```bash
docker version
docker info
docker ps
```
]
Two containers should show up: the agent and the manager.
---
## Check our (single-node) Swarm cluster
Let's connect to the manager instead.
.exercise[
- Select the Swarm manager with Machine
```bash
eval $(docker-machine env node1 --swarm)
```
<!--
```bash
while ! docker info | grep -q "^Nodes: 1"; do
sleep 1
done
```
-->
- Execute some Docker commands
```bash
docker version
docker info
docker ps
```
]
The output is different! Let's review this.
---
## `docker version`
Swarm identifies itself clearly:
```
Client:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:40:35 2016
OS/Arch: linux/amd64
Server:
Version: swarm/1.1.3
API version: 1.22
Go version: go1.5.3
Git commit: 7e9c6bd
Built: Wed Mar 2 00:15:12 UTC 2016
OS/Arch: linux/amd64
```
---
## `docker info`
Swarm gives cluster information, showing all nodes:
.small[
```
Containers: 3
Images: 6
Role: primary
Strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 1
node1: 52.58.50.15:2376
└ Status: Healthy
└ Containers: 3
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 3.859 GiB
└ Labels: executiondriver=native-0.2,
kernelversion=4.2.0-30-generic,
operatingsystem=Ubuntu 15.10,
provider=generic,
storagedriver=aufs
└ Error: (none)
└ UpdatedAt: 2016-03-09T14:01:43Z
Kernel Version: 4.2.0-30-generic
Operating System: linux
Architecture: amd64
CPUs: 2
Total Memory: 3.86 GiB
Name: node1
```
]
---
## `docker ps`
- This one should show nothing at this point
- The Swarm containers are hidden
- This avoids unneeded pollution
- This also avoids killing them by mistake
- We can still see them with `docker ps -a`, though
---
## Add other nodes to the cluster
- Let's use *almost* the same command line (but without `--swarm-master`)
.exercise[
- Stay on `node1` (it has keys and certificates now!)
- Add another node with Docker Machine
<!--
```placeholder
AA.BB.CC.DD $(getent hosts node2 | awk '{print $1}')
```
-->
```bash
TOKEN=$(cat ~/token)
docker-machine create --driver generic \
--swarm --swarm-discovery token://$TOKEN \
--generic-ssh-user docker --generic-ip-address `AA.BB.CC.DD` node2
```
]
Remember to replace AA.BB.CC.DD with the correct IP address.
---
## Scripting
- We can automate the remaining nodes with a little shell script
.exercise[
- Deploy nodes 3, 4, and 5:
```bash
TOKEN=$(cat ~/token)
grep node[345] /etc/hosts | grep -v ^127 |
while read IPADDR NODENAME
do docker-machine create --driver generic \
--swarm --swarm-discovery token://$TOKEN \
--generic-ssh-user docker \
--generic-ip-address $IPADDR $NODENAME
done
```
]
---
## Checking the state of our cluster
.exercise[
<!--
```bash
while ! docker info | grep -q "^Nodes: 5"; do
sleep 1
done
```
-->
- Check that all nodes are here:
```bash
docker info
```
]
If less than 5 nodes show up, just retry every few seconds: the
nodes need some time to register themselves with the service discovery mechanism.
---
## Running containers on Swarm
Try to run a few `busybox` containers.
Then, let's get serious:
.exercise[
- Start a Redis service:
<br/>`docker run -dP redis`
- See the service address:
<br/>`docker port $(docker ps -lq) 6379`
]
This can be any of your five nodes.
---
## Scheduling strategies
- Random: pick a node at random
<br/>(but honor resource constraints)
- Spread: pick the node with the least containers
<br/>(including stopped containers)
- Binpack: try to maximize resource usage
<br/>(in other words: use as few hosts as possible)
---
# Resource allocation
- Swarm can honor resource reservations
- This requires containers to be started with resource limits
- Swarm refuses to schedule a container if it cannot honor a reservation
.exercise[
- Start Redis containers with 1 GB of RAM until Swarm refuses to start more:
```bash
docker run -d -m 1G redis
```
]
On a cluster of 5 nodes with ~3.8 GB of RAM per node, Swarm will refuse to start the 16th container.
---
## Things to know about resource allocation
- `docker info` shows resource allocation for each node
- Swarm allows a 5% resource overcommit (tunable)
- Containers without resource reservation can always be started
- Resources of stopped containers are still counted as being reserved
- this guarantees that it will be possible to restart a stopped container
- containers have to be deleted to free up their resources
- `docker update` can be used to change resource allocation on the fly
---
## Connecting containers with Swarm (1/2)
- Implement service discovery in the application
- requires extensive code changes
- doesn't require extra services or containers
- provides load balancing and failover
- Inject service addresses in environment variables
- requires minimal code changes
- doesn't require extra services or containers
- doesn't provide load balancing and failover
---
## Connecting containers with Swarm (2/2)
- Ambassadors
- don't require code changes
- require additional containers
- provide load balancing and failover
- Overlay networks
- don't require code changes
- don't require extra services or containers
- doesn't provide load balancing and failover (yet)
---
# Connecting containers with ambassadors
- Hands off section (for information only!)
- We will use one-tier, dynamic ambassadors
- Each link to a service will be replaced by an ambassador
- Each ambassador will be placed in the network namespace
of the service using the ambassador
- Ambassadors will be dynamically reconfigured when
linked services are updated
---
## Revisiting `jpetazzo/hamba`
- Configuration is stored in a *volume*
- A watcher process looks for configuration updates,
<br/>and restarts HAProxy when needed
- It can be started without configuration:
```bash
docker run --name amba jpetazzo/hamba run
```
- There is a helper to inject a new configuration:
```bash
docker run --rm --volumes-from amba jpetazzo/hamba \
reconfigure 80 backend1 port1 backend2 port2 ...
```
---
## Network namespaces and `extra_hosts`
This is our plan:
- Replace each `link` with an `extra_host`, pointing to the `127.127.X.X` address space
- Start app containers normally (`docker-compose up`, `docker-compose scale`)
- Start ambassadors after app containers are up:
- ambassadors bind to `127.127.X.X`
- they share their client's network namespace
- Reconfigure ambassadors each time something changes
---
## Our plan for service discovery
- Replace all `links` with static `/etc/hosts` entries
- Those entries will map to `127.127.0.X` (with different `X` for each service)
- Example: `redis` will point to `127.127.0.2` (instead of a container address)
- Start all services; scale them if we want (at this point, they will all fail to connect)
- Start ambassadors in the services' namespace;
<br/>each ambassador will listen on the right `127.127.0.X`
- Gather all backend addresses and configure ambassadors
.warning[Services should try to reconnect!]
---
## "Design for failure," they said
- When the containers are started, the network is not ready
- First connection attempts **will fail**
- App should try to reconnect
- It is OK to crash and restart
- Exponential back-off is nice
---
## Pictures worth 1000 words
- In the following diagrams, we are connecting a
`www` service to a `redis` service through
an ambassador.
---
class: pic
![](ambassadors/simple-1.png)
---
class: pic
![](ambassadors/simple-2.png)
---
class: pic
![](ambassadors/simple-3.png)
---
class: pic
![](ambassadors/simple-4.png)
---
class: pic
![](ambassadors/simple-5.png)
---
class: pic
![](ambassadors/simple-6.png)
---
## Our tools
- `link-to-ambassadors.py`
- replaces all `links` with `extra_hosts` entries
- `create-ambassadors.py`
- scans running containers
- allocates `127.127.X.X` addresses
- starts (unconfigured) ambassadors
- `configure-ambassadors.py`
- scans running containers
- gathers backend addresses
- sends configuration to ambassadors
---
## Convert links to ambassadors
- When we ran `build-tag-push.py` earlier,
<br/>it generated a new `docker-compose.yml-NNN` file.
.notexercise[
- Run the first script to create a new YAML file:
<br/>`../bin/link-to-ambassadors.py`
]
In the Compose file, all links have been replaced
by `extra_hosts` sections.
---
class: pic
## Current state
![Two empty hosts](ambassadors/dockercoins-1.png)
---
## Bring up the application
The application can now be started and scaled.
.notexercise[
- Start the application:
<br/>`docker-compose up -d`
]
Note: you can scale everything as you like, *except Redis*,
because it is stateful.
---
class: pic
## Current state
![Containers are running, but disconnected](ambassadors/dockercoins-2.png)
---
## Create the ambassadors
This has to be executed each time you create new services
or scale up existing ones.
After reading `$COMPOSE_FILE`, it will scan running containers, and compare:
- the list of app containers,
- the list of ambassadors.
It will create missing ambassadors.
.notexercise[
- Run the script!
<br/>`../bin/create-ambassadors.py`
]
---
class: pic
## Current state
![Ambassadors are started, but unconfigured](ambassadors/dockercoins-3.png)
---
## Configure the ambassadors
All ambassadors are created but they still need configuration.
That's the purpose of the last script.
It will read `$COMPOSE_FILE` and gather:
- the list of app backends,
- the list of ambassadors.
Then it configures all ambassadors with all found backends.
.notexercise[
- Run it!
<br/>`../bin/configure-ambassadors.py`
]
---
class: pic
## Current state
![Ambassadors are configured](ambassadors/dockercoins-4.png)
---
## Check what we did
.notexercise[
- Find out the address of the web UI:
<br/>`docker-compose ps webui`
- Point your browser to it
]
---
## Scale
- We will now add more containers.
.notexercise[
- Scale worker and rng:
```bash
docker-compose scale worker=5 rng=10
```
]
The performance graph stays at the same level.
If we look at the logs of the added workers, we will
see screenfuls of "connection refused" exceptions.
---
class: pic
## Current state
![New containers are there, but not ambassadors](ambassadors/dockercoins-5.png)
---
## Add ambassadors
- The new containers don't have ambassadors at this point.
.notexercise[
- Create the missing ambassadors with the script:
```bash
../bin/create-ambassadors.py
```
]
The performance graph stays at the same level.
If we look at the logs of the added workers, we will
now see timeout errors instead of "connection refused."
---
class: pic
## Current state
![Now ambassadors are here, but unconfigured](ambassadors/dockercoins-6.png)
---
## Configure ambassadors
- The last step is to inject the updated configuration.
.notexercise[
- Run the last script one more time:
```bash
../bin/configure-ambassadors.py
```
]
Now the performance graph climbs up, and the worker
logs show normal operation.
---
class: pic
## Current state
![All ambassadors are here and configured](ambassadors/dockercoins-7.png)
---
## A few words about those ambassadors
- There is "a lot" of added complexity here
<br/>(5 scripts of almost 50 lines each!)
- Snark aside, those scripts tap into those concepts:
- network namespaces
- dynamic load balancer reconfiguration
- sidekick containers that are *mandatory*
- ... and have to be managed manually
- We are going to see an easier way to manage this!
---
# Setting up Consul and overlay networks
- We will reconfigure our Swarm cluster to enable overlays
- We will deploy a Consul cluster
- We will connect containers running on different machines
---
## First, let's Clean All The Things!
- We need to remove the old containers
<br/>(in particular the `swarm` agents and managers)
.exercise[
- The following snippet will nuke all containers on all hosts:
```bash
for N in 1 2 3 4 5
do
ssh node$N "docker ps -qa | xargs -r docker rm -f"
done
```
(If it asks you to confirm SSH keys, just do it!)
]
Note: our Swarm cluster is now broken.
---
## Remove old Machine information
- We will use `docker-machine rm`
- With the `generic` driver, this doesn't do anything (it just deletes local configuration)
- With cloud/VM drivers, this would actually delete VMs
.exercise[
- Remove our nodes from Docker Machine config database:
```bash
for N in 1 2 3 4 5
do
docker-machine rm -f node$N
done
```
]
---
## Add extra options to our Engines
- We need two new options for our engines:
- `cluster-store` (to indicate which key/value store to use)
- `cluster-advertise` (to indicate which IP address to register)
- `cluster-store` will be `consul://localhost:8500`
<br/>(we will run one Consul node on each machine)
- `cluster-advertise` will be `eth0:2376`
<br/>(Engine will automatically pick up eth0's IP address)
---
## Reconfiguring Swarm clusters, the Docker way
- The traditional way to reconfigure a service is to edit
its configuration (or init script), then restart
- Instead, we can use Machine's `generic` driver to
redeploy with new parameters
.exercise[
- Re-provision the manager node:
<!--
```placeholder
AA.BB.CC.DD $(getent hosts node1 | awk '{print $1}')
```
-->
```bash
docker-machine create --driver generic \
--engine-opt cluster-store=consul://localhost:8500 \
--engine-opt cluster-advertise=eth0:2376 \
--swarm --swarm-master --swarm-discovery consul://localhost:8500 \
--generic-ssh-user docker --generic-ip-address `AA.BB.CC.DD` node1
```
]
---
## Reconfigure the other nodes
- Once again, scripting to the rescue!
.exercise[
```bash
grep node[2345] /etc/hosts | grep -v ^127 |
while read IPADDR NODENAME
do docker-machine create --driver generic \
--engine-opt cluster-store=consul://localhost:8500 \
--engine-opt cluster-advertise=eth0:2376 \
--swarm --swarm-discovery consul://localhost:8500 \
--generic-ssh-user docker \
--generic-ip-address $IPADDR $NODENAME
done
```
]
---
## Checking what we did
.exercise[
- Directly point the CLI to a node and check configuration:
```bash
eval $(docker-machine env node1)
docker info
```
(should show `Cluster store` and `Cluster advertise`)
- Try to talk to the Swarm cluster:
```bash
eval $(docker-machine env node1 --swarm)
docker info
```
(should show zero node)
]
---
## Why zero node?
- We haven't started Consul yet
- Swarm discovery is not operationl
- Swarm can't discover the nodes
Note: Docker will start (and be functional) without a K/V store.
This lets us run Consul itself in a container.
---
## Adding Consul
- We will run Consul in containers
- We will use the [Consul official image](
https://hub.docker.com/_/consul/) that was released *very recently*
- We will tell Docker to automatically restart it on reboots
- To simplify network setup, we will use `host` networking
---
## A few words about `host` networking
- Consul needs to be aware of its actual IP address (seen by other nodes)
- It also binds a bunch of different ports
- It makes sense (from a security point of view) to have Consul listening on localhost only
(and have "users", i.e. Engine, Swarm, etc. connect over localhost)
- Therefore, we will use `host` networking!
- Also: Docker Machine 0.6 starts the Swarm containers in `host` networking ...
- ... but Docker Machine 0.7 doesn't (which is why we stick to 0.6 for now)
---
## Consul fundamentals (if I must give you just one slide...)
- Consul nodes can be "just an agent" or "server"
- From the client's perspective, they behave the same
- Only servers are members in the Raft consensus / leader election / etc
(non-server agents forward requests to a server)
- All nodes (except maybe one) must be told the address of at least another node to join
- At least the first nodes must know how many nodes to expect to have quorum
- Consul can have only one "truth" at a time (hence the importance of quorum)
---
## Starting our Consul cluster
.exercise[
- Make sure you're logged into `node1`, and:
```bash
IPADDR=$(ip a ls dev eth0 | sed -n 's,.*inet \(.*\)/.*,\1,p')
for N in 1 2 3 4 5; do
ssh node$N -- docker run -d --restart=always --name consul_node$N \
-e CONSUL_BIND_INTERFACE=eth0 --net host consul \
agent -server -retry-join $IPADDR -bootstrap-expect 3 \
-ui -client 0.0.0.0
done
```
]
Note: in production, you probably want to remove `-client 0.0.0.0` since it
gives public access to your cluster! Also adapt `-bootstrap-expect` to your quorum.
---
## Check that our Consul cluster is up
- With your browser, navigate to any instance on port 8500
<br/>(in "NODES" you should see the five nodes)
- Let's run a couple of useful Consul commands
.exercise[
- Ask Consul the list of members it knows:
```bash
docker run --net host --rm consul members
```
- Ask Consul which node is the current leader:
```bash
curl localhost:8500/v1/status/leader
```
]
---
## Check that our Swarm cluster is up
.exercise[
- Try again the `docker info` from earlier:
```bash
eval $(docker-machine env --swarm node1)
docker info
```
- Now all nodes should be visible
<br/>(Give them a minute or two to register)
]
---
# Multi-host networking
- Docker 1.9 has the concept of *networks*
- By default, containers are on the default "bridge" network
- You can create additional networks
- Containers can be on multiple networks
- Containers can dynamically join/leave networks
- The "overlay" driver lets networks span multiple hosts
- Let's see that in action!
---
## Create a few networks and containers
.exercise[
- Create two networks, *blue* and *green*:
```bash
docker network create --driver overlay blue
docker network create --driver overlay green
docker network ls
```
- Create containers with names of blue and green
things, on their respective networks:
```bash
docker run -d --net-alias things --name sky --net blue -m 3G redis
docker run -d --net-alias things --name navy --net blue -m 3G redis
docker run -d --net-alias things --name grass --net green -m 3G redis
docker run -d --net-alias things --name forest --net green -m 3G redis
```
]
---
## Check connectivity within networks
.exercise[
- Check that our containers are on different networks:
```bash
docker ps
```
- This will work:
```bash
docker run --rm --net blue alpine ping -c 3 navy
```
- This will not:
```bash
docker run --rm --net blue alpine ping -c 3 grass
```
]
---
## Containers connected to multiple networks
- Some colors aren't *quite* blue *nor* green
.exercise[
- Create a container that we want to be on both networks:
```bash
docker run -d --net-alias things --net blue --name turquoise redis
```
- Check connectivity:
```bash
docker exec -ti turquoise ping -c 3 navy
docker exec -ti turquoise ping -c 3 grass
```
(First works; second doesn't)
]
---
## Dynamically connecting containers
- This is achieved with the command:
<br/>`docker network connect NETNAME CONTAINER`
.exercise[
- Dynamically connect to the green network:
```bash
docker network connect green turquoise
```
- Check connectivity:
```bash
docker exec -ti turquoise ping -c 3 navy
docker exec -ti turquoise ping -c 3 grass
```
(Both commands work now)
]
---
## Network aliases
- Each container was created with the network alias `things`
- Network aliases are scoped by network
.exercise[
- Resolve the `things` alias from both networks:
```bash
docker run --rm --net blue alpine sh -c \
"apk add --update drill && drill things"
docker run --rm --net green alpine sh -c \
"apk add --update drill && drill things"
```
]
---
## Under the hood
- Each network has an interface in the container
- There is also an interface for the default gateway
.exercise[
- View interfaces in our `turquoise` container:
```bash
docker exec -ti turquoise ip addr ls
```
]
---
## Dynamically disconnecting containers
- There is a mirror command to `docker network connect`
.exercise[
- Disconnect the *turquoise* container from *blue*
(its original network):
```bash
docker network disconnect blue turquoise
docker network connect green turquoise
```
- Check connectivity:
```bash
docker exec -ti turquoise ping -c 3 navy
docker exec -ti turquoise ping -c 3 grass
```
(First command fails, second one works)
]
---
## Cleaning up
.exercise[
- Destroy containers:
```bash
docker rm -f sky navy grass forest turquoise
```
- Destroy networks:
```bash
docker network rm blue
docker network rm green
```
]
---
## Cleaning up after an outage or a crash
- You cannot remove a network if it still has containers
- There is no `"rm -f"` for network
- If a network still has stale endpoints, you can use `"disconnect -f"`
---
# Building images with Swarm
- This is a relatively new feature (Compose ~1.5)
- Builds are scheduled on random nodes
- This disrupts caching
- Independent builds will be fine, though
---
## Caveats when building with Swarm
- Containers are only scheduled where they were built
- cause: Swarm will not automatically copy images across nodes
- solution: distribute images through a registry (e.g. Docker Hub)
- You can end up with inconsistent versions
<br/>(i.e. `dockercoins_rng:latest` being different on two nodes)
- cause: builds can happen on different nodes
- solution: always pin builds to the same node; use explicit version tags
- Also, build caching doesn't work all the time
---
## Why can't Swarm do this automatically for us?
- Let's step back and think for a minute ...
- What should `docker build` do on Swarm?
- build on one machine
- build everywhere ($$$)
- After the build, what should `docker run` do?
- run where we built (how do we know where it is?)
- run on any machine that has the image
- Could Compose+Swarm solve this automatically?
---
## A few words about "sane defaults"
- *It would be nice if Swarm could pick a node, and build there!*
- but which node should it pick?
- what if the build is very expensive?
- what if we want to distribute the build across nodes?
- what if we want to tag some builder nodes?
- ok but what if no node has been tagged?
- *It would be nice if Swarm could automatically push images!*
- using the Docker Hub is an easy choice
<br/>(you just need an account)
- but some of us can't/won't use Docker Hub
<br/>(for compliance reasons or because no network access)
.small[("Sane" defaults are nice only if we agree on the definition of "sane")]
---
## The plan
- Build on a single node (`node1`)
- Tag images
- Upload them to a registry
- Update the Compose file to use those images
*That's the purpose of the `build-tag-push.py` script!*
---
## Which registry do we want to use?
.small[
- **Docker Hub**
- hosted by Docker Inc.
- requires an account (free, no credit card needed)
- images will be public (unless you pay)
- located in AWS EC2 us-east-1
- **Docker Trusted Registry**
- self-hosted commercial product
- requires a subscription (free 30-day trial available)
- images can be public or private
- located wherever you want
- **Docker open source registry**
- self-hosted barebones repository hosting
- doesn't require anything
- doesn't come with anything either
- located wherever you want
]
---
## Using Docker Hub
- Set the `DOCKER_REGISTRY` environment variable to your Docker Hub user name
<br/>(the `build-tag-push.py` script prefixes each image name with that variable)
- We will also see how to run the open source registry
<br/>(so use whatever option you want!)
.exercise[
<!--
```meta
^{
```
-->
- Set the following environment variable:
<br/>`export DOCKER_REGISTRY=jpetazzo`
- (Use *your* Docker Hub login, of course!)
- Log into the Docker Hub:
<br/>`docker login`
<!--
```meta
^}
```
-->
]
---
## Using Docker Trusted Registry
If we wanted to use DTR, we would:
- make sure we have a Docker Hub account
- [activate a Docker Datacenter subscription](
https://hub.docker.com/enterprise/trial/)
- install DTR on our machines
- set `DOCKER_REGISTRY` to `dtraddress:port/user`
*This is out of the scope of this workshop!*
---
## Using open source registry
- We need to run a `registry:2` container
<br/>(make sure you specify tag `:2` to run the new version!)
- It will store images and layers to the local filesystem
<br/>(but you can add a config file to use S3, Swift, etc.)
- Docker *requires* TLS when communicating with the registry,
unless for registries on `localhost` or with the Engine
flag `--insecure-registry`
- Our strategy: run a reverse proxy on `localhost:5000` on each node
---
# Deploying a local registry
- There is a Compose file for that
.exercise[
- Go to the `registry` directory in the repository:
```bash
cd ~/orchestration-workshop/registry
```
]
Let's examine the `docker-compose.yml` file.
---
## Running a local registry with Compose
```yaml
version: "2"
services:
backend:
image: registry:2
frontend:
image: jpetazzo/hamba
command: 5000 backend:5000
ports:
- "127.0.0.1:5000:5000"
depends_on:
- backend
```
- *Backend* is the actual registry.
- *Frontend* is the ambassador that we deployed earlier.
<br/>
It communicates with *backend* using an internal network
and network aliases.
---
## Starting a local registry with Compose
- We will bring up the registry
- Then we will ensure that one *frontend* is running
on each node by scaling it to our number of nodes
.exercise[
- Start the registry:
```bash
docker-compose up -d
```
]
---
## "Scaling" the local registry
- This is a particular kind of scaling
- We just want to ensure that one *frontend*
is running on every single node of the cluster
.exercise[
- Scale the registry:
```bash
for N in $(seq 1 5); do
docker-compose scale frontend=$N
done
```
]
Note: Swarm might do that automatically for us in the future.
---
## Testing our local registry
- We can retag a small image, and push it to the registry
.exercise[
- Make sure we have the busybox image:
```bash
docker pull busybox
```
- Retag the busybox image:
```bash
docker tag busybox localhost:5000/busybox
```
- Push it:
```bash
docker push localhost:5000/busybox
```
]
---
## Adapting our Compose file to run on Swarm
- We can get rid of all the `ports` section, except for the web UI
.exercise[
- Go back to the dockercoins directory:
```bash
cd ~/orchestration-workshop/dockercoins
```
]
---
## Our new Compose file
.small[
```yaml
version: '2'
services:
rng:
build: rng
hasher:
build: hasher
webui:
build: webui
ports:
- "8000:80"
redis:
image: redis
worker:
build: worker
```
]
Copy-paste this into `docker-compose.yml`
<br/>(or you can `cp docker-compose.yml-v2 docker-compose.yml`)
---
## Use images, not builds
- We need to replace each `build` with an `image`
- We will use the `build-tag-push.py` script for that
.exercise[
- Set `DOCKER_REGISTRY` to use our local registry,
<br/>then build, tag, and push the application:
```bash
export DOCKER_REGISTRY=localhost:5000
eval $(docker-machine env node1)
../bin/build-tag-push.py
```
]
---
## Run the application
- At this point, our app is ready to run
- We don't need ambassadors or extra containers
.exercise[
- Start the application:
```bash
export COMPOSE_FILE=docker-compose.yml-`NNN`
eval $(docker-machine env node1 --swarm)
docker-compose up -d
```
- Observe that it's running on multiple nodes:
<br/>(each container name is prefixed with the node it's running on)
```bash
docker ps
```
]
---
## View the performance graph
- Load up the graph in the browser
.exercise[
- Check the `webui` service address and port:
```bash
docker-compose port webui 80
```
- Open it in your browser
]
---
# Scaling workers
- Scaling the `worker` service works out of the box
(like before)
.exercise[
- Scale `worker`:
```bash
docker-compose scale worker=10
```
]
We will hit the bottleneck caused by the `rng` service.
How can we scale that service?
---
## Load balancing with overlay networks, take 1
- Replace `rng` with:
- multiple copies `rng1`, `rng2`, `rng3`, ...
- a load balancer taking over the name `rng`,
<br/>and spreading traffic accross all instances
- Can we do better?
- In a perfect world, we would like to do:
```bash
docker-compose scale rng=10
```
---
## Load balancing with overlay networks, take 2
- Scale `rng` and rely on DNS round robin records
- Problem: the load might end up being totally unbalanced
- We need to send `rng` traffic to an actual load balancer
- This poses a naming problem
---
## Naming problem
- Service is called `rng`
- It therefore takes the network name `rng`
- Worker code connects to `rng`
- So `rng` should point to the load balancer
- What do‽
---
## Naming is *per-network*
- Solution: put `rng` on its own network
- That way, it doesn't take the network name `rng`
<br/>(at least not on the default network)
- Have the load balancer sit on both networks
- Add the name `rng` to the load balancer
---
class: pic
Original DockerCoins
![](dockercoins-single-node.png)
---
class: pic
Load-balanced DockerCoins
![](dockercoins-multi-node.png)
---
## Declaring networks
- Networks (other than the default one)
*must* be declared
in a top-level `networks` section,
placed anywhere in the file
.exercise[
- Add the `rng` network to the Compose file, `docker-compose.yml-NNN`:
```yaml
version: '2'
networks:
rng:
services:
rng:
image: ...
...
```
]
---
## Putting the `rng` service in its network
- Services can have a `networks` section
- If they don't: they are placed in the default network
- If they do: they are placed only in the mentioned networks
.exercise[
- Change the `rng` service to put it in its network:
```yaml
rng:
image: localhost:5000/dockercoins_rng:…
networks:
rng:
```
]
---
## Adding the load balancer
- The load balancer has to be in both networks: `rng` and `default`
- In the `default` network, it must have the `rng` alias
- We will use the `jpetazzo/hamba` image
.exercise[
- Add the `rng-lb` service to the Compose file:
```yaml
rng-lb:
image: jpetazzo/hamba
command: run
networks:
rng:
default:
aliases: [ rng ]
```
]
---
## Load balancer initial configuration
- We specified `run` as the initial command
- This tells `hamba` to wait for an initial configuration
- The load balancer will not be operational
<br/>(until we feed it its configuration)
---
## Start the application
.exercise[
- Bring up DockerCoins:
```bash
docker-compose up -d
```
- See that `worker` is complaining:
```bash
docker-compose logs worker
```
]
---
## Configure the load balancer
- Multiple solutions:
- lookup the IP address of the `rng` backend
- use the backend's network name
- use the backend's container name (easiest!)
.exercise[
- Configure the load balancer:
```bash
docker run --rm --volumes-from dockercoins_rng-lb_1 \
--net container:dockercoins_rng-lb_1 \
jpetazzo/hamba reconfigure 80 dockercoins_rng_1 80
```
]
The application should now be working correctly.
---
## Scale the application
- Use `docker-compose scale` as planned
.exercise[
- Scale `rng`:
```bash
docker-compose scale rng=10
```
]
Of course, the graph doesn't change *yet*.
We need to add the new backends to the load balancer
configuration first.
---
## Reconfigure the load balancer
- The command is similar to the one before
- We need to pass the list of all backends
.exercise[
- Reconfigure the load balancer:
```bash
docker run --rm \
--volumes-from dockercoins_rng-lb_1 \
--net container:dockercoins_rng-lb_1 \
jpetazzo/hamba reconfigure 80 \
$(for N in $(seq 1 10); do
echo dockercoins_rng_$N:80
done)
```
]
---
## Automating the process
- Nobody loves artisan YAML handy craft
- This can be automated very easily
- To make things easier, we can use a label:
*each container behind a load balancer will
have a `loadbalancer` label giving the name
of that loadbalancer*
- See script `reconfigure-load-balancers.py` for an example
---
## Use DNS to discover the addresses of all the backends
- When multiple containers have the same network alias:
- Engine 1.10 returns only one of them (the same one across the whole network)
- Engine 1.11 returns all of them (in a random order)
- A "smart" client can use all records to implement load balancing
- We can compose `jpetazzo/hamba` with a special-purpose container,
which will dynamically generate HAProxy's configuration when
the DNS records are updated
---
## Introducing `jpetazzo/watchdns`
- [100 lines of pure POSIX scriptery](
https://github.com/jpetazzo/watchdns/blob/master/watchdns)
- Resolves a given DNS name every second
- Each time the result changes, a new HAProxy configuration is generated
- When used together with `--volumes-from` and `jpetazzo/hamba`, it
updates the configuration of an existing load balancer
- Comes with a companion script, `add-load-balancer-v2.py`, to update
your Compose files
---
## Using `jpetazzo/watchdns`
.exercise[
- First, revert the Compose file to remove the load balancer
- Then, run `add-load-balancer-v2.py`:
```bash
../bin/add-load-balancer-v2.py rng
```
- Inspect the resulting Compose file
]
---
## Scaling with `watchdns`
.exercise[
- Start the application with the new sidekick containers:
```bash
docker-compose up -d
```
- Scale `rng`:
```bash
docker-compose scale rng=10
```
- Check logs:
```bash
docker-compose logs rng-wd
```
]
---
## Cleaning up
.exercise[
- Terminate containers and remove them:
```bash
docker-compose down
```
]
Note: `docker-compose down` also deletes the
networks that had been created for the application.
---
## Comments
- This is a very crude implementation of the pattern
- A Go version would only be a bit longer, but use much less resources
- When there are many backends, reacting quickly to change is less important
(i.e. it's not necessary to re-resolve records every second!)
???
# Going further
Deploying a new version (difficulty: easy)
- Just re-run all the steps!
- However, Compose will re-create the containers
- You will have to re-create ambassadors
<br/>(and configure them)
- You will have to cleanup old ambassadors
<br/>(left as an exercise for the reader)
- You will experience a little bit of downtime
???
## Going further
Zero-downtime deployment (difficulty: medium)
- Isolate stateful services (like we did earlier for Redis)
- Do blue/green deployment:
- deploy and scale version N
- point a "top-level" load balancer to the app
- deploy and scale version N+1
- put both apps in the "top-level" balancer
- slowly switch traffic over to app version N+1
???
## Going further
Harder projects:
- Two-tier or three-tier ambassador deployments
- Deploy to Mesos or Kubernetes
---
class: title
Resiliency
<br/>
and
<br/>
high availability
---
# Distributing Machine credentials
- All the credentials (TLS keys and certs) are on node1
<br/>(the node on which we ran `docker-machine create`)
- If we lose node1, we're toast
- We need to move (or copy) the credentials somewhere safe
- Credentials are regular files, and relatively small
- Ah, if only we had a highly available, hierarchic store ...
--
- Wait a minute, we have one!
--
(That's Consul, if you were wondering)
---
## Storing files in Consul
- We will use [Benjamin Wester's consulfs](
https://github.com/bwester/consulfs)
- It mounts a Consul key/value store as a local filesystem
- Performance will be horrible
<br/>(don't run a database on top of that!)
- But to store files of a few KB, nobody will notice
- We will copy/link/sync... `~/.docker/machine` to Consul
---
## Installing consulfs
- Option 1: install Go, git clone, go build ...
- Option 2: be lazy and use [jpetazzo/consulfs](
https://hub.docker.com/r/jpetazzo/consulfs/)
.exercise[
- Be lazy and use the Docker image:
```bash
eval $(docker-machine env node1)
docker run --rm -v /usr/local/bin:/target jpetazzo/consulfs
```
]
Note: the `jpetazzo/consulfs` image contains the
`consulfs` binary.
It copies it to `/target` (if `/target` is a volume).
---
## Can't we run consulfs in a container?
- Yes we can!
- The filesystem will be mounted in the container
- It won't be visible outside of the container (from the host)
- We can use *shared mounts* to propagate mounts from containers to Docker
- But propagating from Docker to the host requires particular systemd flags
- ... So we'll run it on the host for now
---
## Running consulfs
- The `consulfs` binary takes two arguments:
- the Consul server address
- a mount point (that has to be created first)
.exercise[
- Create a mount point and mount Consul as a local filesystem:
```bash
mkdir ~/consul
consulfs localhost:8500 ~/consul
```
]
Leave this running in the foreground.
---
## Checking our consulfs mount point
- All key/values will be visible:
- Swarm discovery
- overlay networks
- ... anything you put in Consul!
.exercise[
- Check that Consul key/values are visible:
```bash
ls -l ~/consul/
```
]
---
## Copying our credentials to Consul
- Use standard UNIX commands
- Don't try to preserve permissions, though (`consulfs` doesn't store permissions)
.exercise[
- Copy Machine credentials into Consul:
```bash
cp -r ~/.docker/machine/. ~/consul/machine/
```
]
(This command can be re-executed to update the copy.)
---
## Install consulfs on another node
- We will repeat the previous steps to install consulfs
.exercise[
- Connect to node2:
```bash
ssh node2
```
- Install `consulfs`:
```bash
docker run --rm -v /usr/local/bin:/target jpetazzo/consulfs
```
]
---
## Mount Consul
- The procedure is still the same as on the first node
.exercise[
- Create the mount point:
```bash
mkdir ~/consul
```
- Mount the filesystem:
```bash
consulfs localhost:8500 ~/consul &
```
]
At this point, `ls -l ~/consul` should show `docker` and
`machine` directories.
---
## Access the credentials from the other node
- We will create a symlink
- We could also copy the credentials
.exercise[
- Create the symlink:
```bash
mkdir -p ~/.docker/
ln -s ~/consul/machine ~/.docker/
```
- Check that all nodes are visible:
```bash
docker-machine ls
```
]
---
## A few words on this strategy
- Anyone accessing Consul can control your Docker cluster
<br/>(to be fair: anyone accessing Consul can wreck
serious havoc to your cluster anyway)
- ConsulFS doesn't support *all* POSIX operations,
so a few things (like `mv`) will not work)
- As a consequence, with Machine 0.6, you cannot
run `docker-machine create` directly on top of ConsulFS
---
## What if Consul becomes unavailable?
- If Consul becomes unavailable (e.g. loses quorum),
<br/>you won't be able to access your credentials
- If Consul becomes unavailable ...
<br/>your cluster will be in a bad state anyway
- You can still access each Docker Engine over the
local UNIX socket
<br/>(and repair Consul that way)
---
# Highly available Swarm managers
- Until now, the Swarm manager was a SPOF
<br/>(Single Point Of Failure)
- Swarm has support for replication
- When replication is enabled, you deploy multiple (identical) managers
- one will be "primary"
- the other(s) will be "secondary"
- this is determined automatically
<br/>(through *leader election*)
---
## Swarm leader election
- The leader election mechanism relies on a key/value store
<br/>(consul, etcd, zookeeper)
- There is no requirement on the number of replicas
<br/>(the quorum is achieved through the key/value store)
- When the leader (or "primary") is unavailable,
<br/>a new election happens automatically
- You can issue API requests to any manager:
<br/>if you talk to a secondary, it forwards to the primary
.warning[There is currently a bug when
the Consul cluster itself has a leader election;
<br/>see [docker/swarm#1782](https://github.com/docker/swarm/issues/1782).]
---
## Swarm replication in practice
- We need to give two extra flags to the Swarm manager:
- `--replication`
*enables replication (duh!)*
- `--advertise ip.ad.dr.ess:port`
*address and port where this Swarm manager is reachable*
- Do you deploy with Docker Machine?
<br/>Then you can use `--swarm-opt`
to automatically pass flags to the Swarm manager
---
## Cleaning up our current Swarm containers
- We will use Docker Machine to re-provision Swarm
- We need to:
- remove the nodes from the Machine registry
- remove the Swarm containers
.exercise[
- Remove the current configuration (remember to go back to node1!):
```bash
for N in 1 2 3 4 5; do
ssh node$N docker rm -f swarm-agent swarm-agent-master
docker-machine rm -f node$N
done
```
]
---
## Re-deploy with the new configuration
- This time, all nodes can be deployed identically
<br/>(instead of 1 manager + 4 non-managers)
.exercise[
```bash
grep node[12345] /etc/hosts | grep -v ^127 |
while read IPADDR NODENAME; do
docker-machine create --driver generic \
--engine-opt cluster-store=consul://localhost:8500 \
--engine-opt cluster-advertise=eth0:2376 \
--swarm --swarm-master \
--swarm-discovery consul://localhost:8500 \
--swarm-opt replication --swarm-opt advertise=$IPADDR:3376 \
--generic-ssh-user docker --generic-ip-address $IPADDR $NODENAME
done
```
]
.small[
Note: Consul is still running thanks to the `--restart=always` policy.
Other containers are now stopped, because the engines have been
reconfigured and restarted.
]
---
## Assess our new cluster health
- The output of `docker info` will tell us the status
of the node that we are talking to (primary or replica)
- If we talk to a replica, it will tell us who is the primary
.exercise[
- Talk to a random node, and ask its view of the cluster:
```bash
eval $(docker-machine env node3 --swarm)
docker info | grep -e ^Name -e ^Role -e ^Primary
```
]
Note: `docker info` is one of the only commands that will
work even when there is no elected primary. This helps
debugging.
---
## Test Swarm manager failover
- The previous command told us which node was the primary manager
- if `Role` is `primary`,
<br/>then the primary is indicated by `Name`
- if `Role` is `replica`,
<br/>then the primary is indicated by `Primary`
.exercise[
- Kill the primary manager:
```bash
ssh node`N` docker kill swarm-agent-master
```
]
Look at the output of `docker info` every few seconds.
---
# Highly available containers
- Swarm has support for *rescheduling* on node failure
- It has to be explicitly enabled on a per-container basis
- When the primary manager detects that a node goes down,
<br/>those containers are rescheduled elsewhere
- If the containers can't be rescheduled (constraints issue),
<br/>they are lost (there is no reconciliation loop yet)
- In Swarm 1.1, this is an *experimental* feature
<br/>(To enable it, you must pass the `--experimental` flag when you start Swarm itself!)
- In Swarm 1.2, you don't need the `--experimental` flag anymore
---
## About Swarm generic flags
- Some flags like `--experimental` and `--debug` must be *before* the Swarm command
<br/>(i.e. `docker run swarm --debug manage ...`)
- We cannot use Docker Machine to pass that flag ☹
<br/>(Machine adds flags *after* the Swarm command)
- Instead, we can use a custom Swarm image:
```dockerfile
FROM swarm
ENTRYPOINT ["/swarm", "--debug"]
```
- We can tell Machine to use this with `--swarm-image`
---
## Start a resilient container
- By default, containers will not be restarted when their node goes down
- You must pass an explicit *rescheduling policy* to make that happen
- For now, the only policy is "on-node-failure"
.exercise[
- Start a container with a rescheduling policy:
```bash
docker run --name highlander -d -e reschedule:on-node-failure nginx
```
]
Check that the container is up and running.
---
## Simulate a node failure
- We will reboot the node running this container
- Swarm will reschedule it
.exercise[
- Check on which node the container is running:
</br>`NODE=$(docker inspect --format '{{.Node.Name}}' highlander)`
- Reboot that node:
<br/>`ssh $NODE sudo reboot`
- Check that the container has been recheduled:
<br/>`docker ps -a`
]
---
## Reboots
- When rebooting a node, Docker is stopped cleanly, and containers are stopped
- Our container is rescheduled, but not started
- To simulate a "proper" failure, we can use the Chaos Monkey script instead
```bash
~/orchestration-workshop/bin/chaosmonkey $NODE <connect|disconnect|reboot>
```
---
## Cluster reconciliation
- After the cluster rejoins, we can end up with duplicate containers
.exercise[
- Once the node is back, remove one of the extraneous containers:
```bash
docker rm -f node`N`/highlander
```
]
---
## .warning[Caveats]
- There are some corner cases when the node is also
the Swarm leader or the Consul leader; this is being improved
right now!
- The safest way to address for now this is to run the Consul
servers, the Swarm managers, and your containers, on
different nodes.
- Swarm doesn't handle gracefully the fact that after the
reboot, you have *two* containers named `highlander`,
and attempts to manipulate the container with its name
will not work. This will be improved too.
---
# Conclusions
- Bad news: we still have work to do to deploy our apps
- it's not all unicorns, ponies, and rainbows
- *no, Docker will not make your job obsolete*
- Good news: a lot of hard things are becoming easier
- building, packaging, distributing apps
- running distributed systems on clusters
---
## What's next?
- November 2015: Compose 1.5 + Engine 1.9 =
<br/>first release with multi-host networking
- January 2016: Compose 1.6 + Engine 1.10 =
<br/>embedded DNS server, experimental high availability
- April 2016: Compose 1.7 + Engine 1.11 =
<br/>round robin DNS records, huge improvements in HA
- Next release: another truckload of features
- I will deliver this workshop about twice a month
- Check out the GitHub repo for updated content!
<br/>(there is a tag for each big round of updates)
---
## High availability
- Docker in general, and Swarm in particular, move *fast*
- Current high availability features are not Chaos-Monkey proof (yet)
- We (well, the Swarm team) is working to change that
---
## Overall complexity
- The scripts used here are pretty simple (each is less than 100 LOCs)
- You can easily rewrite them in your favorite language,
<br/>adapt and customize them, in a few hours of time
- FYI: those scripts are smaller and simpler than the
scripts (cloud init etc) used to deploy the VMs for this
workshop!
- Docker Inc. has commercial products to wrap all this:
- Docker Cloud
<br/>(manage your Docker nodes from a SAAS portal)
- Docker Datacenter
<br/>(buzzword-compliant management solution:
<br/>turnkey, enterprise-class, on-premise, etc.)
---
class: title
# Thanks! <br/> Questions?
## [@jpetazzo](https://twitter.com/jpetazzo) <br/> [@docker](https://twitter.com/docker)
</textarea>
<script src="https://gnab.github.io/remark/downloads/remark-0.13.min.js" type="text/javascript">
</script>
<script type="text/javascript">
var slideshow = remark.create({
ratio: '16:9',
highlightSpans: true
});
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink