diff --git a/docs/index.html b/docs/index.html
deleted file mode 100644
index 9db48fe1..00000000
--- a/docs/index.html
+++ /dev/null
@@ -1,8552 +0,0 @@
-<!DOCTYPE html>
-<html>
-  <head>
-    <base target="_blank">
-    <title>Docker Orchestration Workshop</title>
-    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
-    <style type="text/css">
-      @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
-      @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
-      @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
-
-      /* For print! Borrowed from https://github.com/gnab/remark/issues/50 */
-      @page {
-        size: 1210px 681px;
-        margin: 0;
-       }
-
-      @media print {
-        .remark-slide-scaler {
-            width: 100% !important;
-            height: 100% !important;
-            transform: scale(1) !important;
-            top: 0 !important;
-            left: 0 !important;
-        }
-      }
-
-      body { font-family: 'Droid Serif'; }
-
-      h1, h2, h3 {
-        font-family: 'Yanone Kaffeesatz';
-        font-weight: normal;
-        margin-top: 0.5em;
-      }
-      a {
-        text-decoration: none;
-        color: blue;
-      }
-      .remark-slide-content { padding: 1em 2.5em 1em 2.5em; }
-
-      .remark-slide-content { font-size: 25px; }
-      .remark-slide-content h1 { font-size: 50px; }
-      .remark-slide-content h2 { font-size: 50px; }
-      .remark-slide-content h3 { font-size: 25px; }
-      .footnote {
-        position: absolute;
-        bottom: 3em;
-      }
-      .remark-code { font-size: 25px; }
-      .small .remark-code { font-size: 16px; }
-
-      .remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
-      .remark-inline-code {
-         background-color: #ccc;
-      }
-      .red { color: #fa0000; }
-      .gray { color: #ccc; }
-      .small { font-size: 70%; }
-      .big { font-size: 140%; }
-      .underline { text-decoration: underline; }
-      .strike { text-decoration: line-through; }
-      .pic {
-        vertical-align: middle;
-        text-align: center;
-        padding: 0 0 0 0 !important;
-      }
-      img {
-        max-width: 100%;
-        max-height: 550px;
-      }
-      .title {
-        vertical-align: middle;
-        text-align: center;
-      }
-      .title h1 { font-size: 5em; }
-      .title p { font-size: 3em; }
-      .quote {
-        background: #eee;
-        border-left: 10px solid #ccc;
-        margin: 1.5em 10px;
-        padding: 0.5em 10px;
-        quotes: "\201C""\201D""\2018""\2019";
-        font-style: italic;
-      }
-      .quote:before {
-        color: #ccc;
-        content: open-quote;
-        font-size: 4em;
-        line-height: 0.1em;
-        margin-right: 0.25em;
-        vertical-align: -0.4em;
-      }
-      .quote p {
-        display: inline;
-      }
-      .warning {
-        background-image: url("warning.png");
-        background-size: 1.5em;
-        background-repeat: no-repeat;
-        padding-left: 2em;
-      }
-      .exercise {
-        background-color: #eee;
-        background-image: url("keyboard.png");
-        background-size: 1.4em;
-        background-repeat: no-repeat;
-        background-position: 0.2em 0.2em;
-        border: 2px dotted black;
-      }
-      .exercise::before {
-        content: "Exercise";
-        margin-left: 1.8em;
-      }
-      li p { line-height: 1.25em; }
-      div.extra-details {
-        background-image: url(extra-details.png);
-        background-position: 99.5% 1%;
-        background-size: 4%;
-      }
-      /* This is used only for the history slide (the only table in this doc) */
-      td {
-        padding: 0.1em 0.5em;
-        background: #eee;
-      }
-    </style>
-  </head>
-  <body>
-    <textarea id="source">
-
-class: title, self-paced
-
-Docker <br/> Orchestration <br/> Workshop
-
----
-
-class: title, in-person
-
-.small[
-
-Deploy and scale containers with Docker native, open source orchestration
-
-.small[.small[
-
-**Be kind to the WiFi!**
-
-*Use the 5G network*
-<br/>
-*Don't use your hotspot*
-<br/>
-*Don't stream videos from YouTube, Netflix, etc.
-<br/>(if you're bored, watch local content instead)*
-
-Thank you!
-
-]
-]
-]
-
----
-
-class: in-person
-
-## Intros
-
-- Hello! We are
-  AJ ([@s0ulshake](https://twitter.com/s0ulshake))
-  &
-  Jérôme ([@jpetazzo](https://twitter.com/jpetazzo))
-
---
-
-class: in-person
-
-- This is our collective Docker knowledge:
-
-  ![Bell Curve](bell-curve.jpg)
-
-<!--
-Reminder, when updating the agenda: when people are told to show
-up at 9am, they usually trickle in until 9:30am (except for paid
-training sessions). If you're not sure that people will be there
-on time, it's a good idea to have a breakfast with the attendees
-at e.g. 9am, and start at 9:30.
--->
-
----
-
-class: in-person
-
-## Agenda
-
-<!--
-- Agenda:
--->
-
-<!--
-
-.small[
-- 09:00-10:30 part 1
-- 10:30-11:00 coffee break
-- 11:00-12:30 part 2
-- 12:00-13:00 lunch break
-- 13:00-14:30 part 3
-- 14:30-14:45 coffee break
-- 14:45-16:00 part 4
-- 16:00-16:01 Q&A
-]
-
--->
-
-- The tutorial will run from 9:00am to 12:20pm
-
-- This will be fast-paced, but DON'T PANIC!
-
-- All the content is publicly available (slides, code samples, scripts)
-
-  Upstream URL: https://github.com/jpetazzo/orchestration-workshop
-
-- There will be a coffee break at 10:30am
-  <br/>
-  (please remind me if I forget about it!)
-
-- Feel free to interrupt for questions at any time
-
-- Live feedback, questions, help on [Gitter](chat)
-
-  http://container.training/chat
-
----
-
-## A brief introduction
-
-- This was initially written to support in-person,
-  instructor-led workshops and tutorials
-
-- You can also follow along on your own, at your own pace
-
-- We included as much information as possible in these slides
-
-- We recommend having a mentor to help you ...
-
-- ... Or be comfortable spending some time reading the Docker
- [documentation](https://docs.docker.com/) ...
-
-- ... And looking for answers in the [Docker forums](forums.docker.com),
-  [StackOverflow](http://stackoverflow.com/questions/tagged/docker),
-  and other outlets
-
----
-
-class: self-paced
-
-## Hands on, you shall practice
-
-- Nobody ever became a Jedi by spending their lives reading Wookiepedia
-
-- Likewise, it will take more than merely *reading* these slides
-  to make you an expert
-
-- These slides include *tons* of exercises
-
-- They assume that you have access to a cluster of Docker nodes
-
-- If you are attending a workshop or tutorial:
-  <br/>you will be given specific instructions to access your cluster
-
-- If you are doing this on your own:
-  <br/>you can use
-  [Play-With-Docker](http://www.play-with-docker.com/) and
-  read [these instructions](https://github.com/jpetazzo/orchestration-workshop#using-play-with-docker) for extra
-  details
-
-???
-
-<!--
-grep '^# ' index.html | grep -v '<br' | tr '#' '-'
--->
-
----
-
-class: in-person
-
-## Chapter 1: getting started
-
-- Pre-requirements
-
-- VM environment
-
-- Our sample application
-
-- Running the application
-
-- Identifying bottlenecks
-
-- Introducing SwarmKit
-
----
-
-class: in-person
-
-## Chapter 2: scaling out our app on Swarm
-
-- Creating our first Swarm
-
-- Docker Machine
-
-- Running our first Swarm service
-
-- Deploying a local registry
-
-- Overlay networks
-
-- Global scheduling
-
-- Scripting image building and pushing
-
-- Integration with Compose
-
----
-
-class: in-person
-
-## Chapter 3: operating the Swarm
-
-- Breaking into an overlay network
-
-- Securing overlay networks
-
-- Rolling updates
-
-- (Secrets management and encryption at rest)
-
-- [Centralized logging](#logging)
-
-- Metrics collection
-
----
-
-class: in-person
-
-## Chapter 4: bonus material
-
-- Dealing with stateful services
-
-- Controlling Docker from a container
-
-- Node management
-
-- What's next?
-
----
-
-# Pre-requirements
-
-- Computer with internet connection and a web browser
-
-- For instructor-led workshops: an SSH client to connect to remote machines
-
-  - on Linux, OS X, FreeBSD... you are probably all set
-
-  - on Windows, get [putty](http://www.putty.org/),
-  Microsoft [Win32 OpenSSH](https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH),
-  [Git BASH](https://git-for-windows.github.io/), or
-  [MobaXterm](http://mobaxterm.mobatek.net/)
-
-- For self-paced learning: SSH is not necessary if you use
-  [Play-With-Docker](http://www.play-with-docker.com/)
-
-- Some Docker knowledge
-
-  (but that's OK if you're not a Docker expert!)
-
----
-
-class: in-person, extra-details
-
-## Nice-to-haves
-
-- [Mosh](https://mosh.org/) instead of SSH, if your internet connection tends to lose packets
-  <br/>(available with `(apt|yum|brew) install mosh`; then connect with `mosh user@host`)
-
-- [GitHub](https://github.com/join) account
-  <br/>(if you want to fork the repo; also used to join Gitter)
-
-- [Gitter](https://gitter.im/) account
-  <br/>(to join the conversation during the workshop)
-
-- [Slack](https://community.docker.com/registrations/groups/4316) account
-  <br/>(to join the conversation after the workshop)
-
-- [Docker Hub](https://hub.docker.com) account
-  <br/>(it's one way to distribute images on your Swarm cluster)
-
----
-
-class: extra-details
-
-## Extra details
-
-- This slide should have a little magnifying glass in the top right corner
-
-  (If it doesn't, it's because CSS is hard — Jérôme is only a backend person, alas)
-
-- Slides with that magnifying glass indicate slides providing extra details
-
-- Feel free to skip them if you're in a hurry!
-
----
-
-## Hands-on sections
-
-- The whole workshop is hands-on
-
-- We will see Docker in action
-
-- You are invited to reproduce all the demos
-
-- All hands-on sections are clearly identified, like the gray rectangle below
-
-.exercise[
-
-- This is the stuff you're supposed to do!
-- Go to [container.training](http://container.training/) to view these slides
-- Join the [chat room](chat)
-
-]
-
----
-
-class: in-person
-
-# VM environment
-
-- To follow along, you need a cluster of five Docker Engines
-
-- If you are doing this with an instructor, see next slide
-
-- If you are doing (or re-doing) this on your own, you can:
-
-  - create your own cluster (local or cloud VMs) with Docker Machine
-    ([instructions](https://github.com/jpetazzo/orchestration-workshop/tree/master/prepare-machine))
-
-  - use [Play-With-Docker](http://play-with-docker.com) ([instructions](https://github.com/jpetazzo/orchestration-workshop#using-play-with-docker))
-
-  - create a bunch of clusters for you and your friends
-    ([instructions](https://github.com/jpetazzo/orchestration-workshop/tree/master/prepare-vms))
-
----
-
-class: pic, in-person
-
-![You get five VMs](you-get-five-vms.jpg)
-
----
-
-class: in-person
-
-## You get five VMs
-
-- Each person gets 5 private VMs (not shared with anybody else)
-- They'll remain up until the day after the tutorial
-- You should have a little card with login+password+IP addresses
-- You can automatically SSH from one VM to another
-
-.exercise[
-
-<!--
-```bash
-for N in $(seq 1 5); do
-  ssh -o StrictHostKeyChecking=no node$N true
-done
-for N in $(seq 1 5); do
-  (.
-  docker-machine rm -f node$N
-  ssh node$N "docker ps -aq | xargs -r docker rm -f"
-  ssh node$N sudo rm -f /etc/systemd/system/docker.service
-  ssh node$N sudo systemctl daemon-reload
-  echo Restarting node$N.
-  ssh node$N sudo systemctl restart docker
-  echo Restarted node$N.
-  ) &
-done
-wait
-```
--->
-
-- Log into the first VM (`node1`) with SSH or MOSH
-- Check that you can SSH (without password) to `node2`:
-  ```bash
-  ssh node2
-  ```
-- Type `exit` or `^D` to come back to node1
-
-<!--
-```meta
-^D
-```
--->
-
-]
-
----
-
-class: in-person
-
-## If doing or re-doing the workshop on your own ...
-
----
-
-class: self-paced
-
-## How to get your own Docker nodes?
-
-- Use [Play-With-Docker](http://www.play-with-docker.com/)!
-
---
-
-- Main differences:
-
-  - you don't need to SSH to the machines
-    <br/>(just click on the node that you want to control in the left tab bar)
-
-  - Play-With-Docker automagically detects exposed ports
-    <br/>(and displays them as little badges with port numbers, above the terminal)
-
-  - You can access HTTP services by clicking on the port numbers
-
-  - exposing TCP services requires something like
-    [ngrok](https://ngrok.com/)
-    or [supergrok](https://github.com/jpetazzo/orchestration-workshop#using-play-with-docker)
-
-<!--
-
-- If you use VMs deployed with Docker Machine:
-
-  - you won't have pre-authorized SSH keys to bounce across machines
-
-  - you won't have host aliases
-
--->
-
----
-
-class: self-paced
-
-## Using Play-With-Docker
-
-- Open a new browser tab to [www.play-with-docker.com](http://www.play-with-docker.com/)
-
-- Confirm that you're not a robot
-
-- Click on "ADD NEW INSTANCE": congratulations, you have your first Docker node!
-
-- When you will need more nodes, just click on "ADD NEW INSTANCE" again
-
-- Note the countdown in the corner; when it expires, your instances are destroyed
-
-- If you give your URL to somebody else, they can access your nodes too
-  <br/>
-  (You can use that for pair programming, or to get help from a mentor)
-
-- Loving it? Not loving it? Tell it to the wonderful authors,
-  [@marcosnils](https://twitter.com/marcosnils) &
-  [@xetorthio](https://twitter.com/xetorthio)!
-
----
-
-## We will (mostly) interact with node1 only
-
-- Unless instructed, **all commands must be run from the first VM, `node1`**
-
-- We will only checkout/copy the code on `node1`
-
-- When we will use the other nodes, we will do it mostly through the Docker API
-
-- We will log into other nodes only for initial setup and a few "out of band" operations
-  <br/>(checking internal logs, debugging...)
-
----
-
-## Terminals
-
-Once in a while, the instructions will say:
-<br/>"Open a new terminal."
-
-There are multiple ways to do this:
-
-- create a new window or tab on your machine, and SSH into the VM;
-
-- use screen or tmux on the VM and open a new window from there.
-
-You are welcome to use the method that you feel the most comfortable with.
-
----
-
-## Tmux cheatsheet
-
-- Ctrl-b c → creates a new window
-- Ctrl-b n → go to next window
-- Ctrl-b p → go to previous window
-- Ctrl-b " → split window top/bottom
-- Ctrl-b % → split window left/right
-- Ctrl-b Alt-1 → rearrange windows in columns
-- Ctrl-b Alt-2 → rearrange windows in rows
-- Ctrl-b arrows → navigate to other windows
-- Ctrl-b d → detach session
-- tmux attach → reattach to session
-
----
-
-## Brand new versions!
-
-- Engine 17.09
-- Compose 1.16
-- Machine 0.12
-
-.exercise[
-
-- Check all installed versions:
-  ```bash
-  docker version
-  docker-compose -v
-  docker-machine -v
-  ```
-
-]
-
----
-
-## Wait, what, 17.09 ?!?
-
---
-
-- Docker 1.13 = Docker 17.03 (year.month, like Ubuntu)
-
-- Every month, there is a new "edge" release (with new features)
-
-- Every quarter, there is a new "stable" release
-
-- Docker CE releases are maintained 4+ months
-
-- Docker EE releases are maintained 12+ months
-
-- For more details, check the [Docker EE announcement blog post](https://blog.docker.com/2017/03/docker-enterprise-edition/)
-
----
-
-class: extra-details
-
-## Docker CE vs Docker EE
-
-- Docker EE:
-
-  - $$$
-  - certification for select distros, clouds, and plugins
-  - advanced management features (fine-grained access control, security scanning...)
-
-- Docker CE:
-
-  - free
-  - available through Docker Mac, Docker Windows, and major Linux distros
-  - perfect for individuals and small organizations
-
----
-
-class: extra-details
-
-## Why?
-
-- More readable for enterprise users
-
-  (i.e. the very nice folks who are kind enough to pay us big $$$ for our stuff)
-
-- No impact for the community
-
-  (beyond CE/EE suffix and version numbering change)
-
-- Both trains leverage the same open source components
-
-  (containerd, libcontainer, SwarmKit...)
-
-- More predictible release schedule (see next slide)
-
----
-
-class: pic
-
-![Docker CE/EE release cycle](lifecycle.png)
-
----
-
-## What was added when?
-
-||||
-| ---- | ----- | --- |
-| 2015 |  1.9  | Overlay (multi-host) networking, network/IPAM plugins
-| 2016 |  1.10 | Embedded dynamic DNS
-| 2016 |  1.11 | DNS round robin load balancing
-| 2016 |  1.12 | Swarm mode, routing mesh, encrypted networking, healthchecks
-| 2017 |  1.13 | Stacks, attachable overlays, image squash and compress
-| 2017 |  1.13 | Windows Server 2016 Swarm mode
-| 2017 | 17.03 | Secrets
-| 2017 | 17.04 | Update rollback, placement preferences (soft constraints)
-| 2017 | 17.05 | Multi-stage image builds, service logs
-| 2017 | 17.06 | Swarm configs, node/service events
-| 2017 | 17.06 | Windows Server 2016 Swarm overlay networks, secrets
-
----
-
-class: title
-
-All right!
-<br/>
-We're all set.
-<br/>
-Let's do this.
-
----
-
-name: part-1
-
-class: title, self-paced
-
-Part 1
-
----
-
-# Our sample application
-
-- Visit the GitHub repository with all the materials of this workshop:
-  <br/>https://github.com/jpetazzo/orchestration-workshop
-
-- The application is in the [dockercoins](
-  https://github.com/jpetazzo/orchestration-workshop/tree/master/dockercoins)
-  subdirectory
-
-- Let's look at the general layout of the source code:
-
-  there is a Compose file [docker-compose.yml](
-  https://github.com/jpetazzo/orchestration-workshop/blob/master/dockercoins/docker-compose.yml) ...
-
-  ... and 4 other services, each in its own directory:
-
-  - `rng` = web service generating random bytes
-  - `hasher` = web service computing hash of POSTed data
-  - `worker` = background process using `rng` and `hasher`
-  - `webui` = web interface to watch progress
-
----
-
-class: extra-details
-
-## Compose file format version
-
-*Particularly relevant if you have used Compose before...*
-
-- Compose 1.6 introduced support for a new Compose file format (aka "v2")
-
-- Services are no longer at the top level, but under a `services` section
-
-- There has to be a `version` key at the top level, with value `"2"` (as a string, not an integer)
-
-- Containers are placed on a dedicated network, making links unnecessary
-
-- There are other minor differences, but upgrade is easy and straightforward
-
----
-
-## Links, naming, and service discovery
-
-- Containers can have network aliases (resolvable through DNS)
-
-- Compose file version 2+ makes each container reachable through its service name
-
-- Compose file version 1 did require "links" sections
-
-- Our code can connect to services using their short name
-
-  (instead of e.g. IP address or FQDN)
-
-- Network aliases are automatically namespaced
-
-  (i.e. you can have multiple apps declaring and using a service named `database`)
-
----
-
-## Example in `worker/worker.py`
-
-![Service discovery](service-discovery.png)
-
----
-
-## What's this application?
-
----
-
-class: pic
-
-![DockerCoins logo](dockercoins.png)
-
-(DockerCoins 2016 logo courtesy of [@XtlCnslt](https://twitter.com/xtlcnslt) and [@ndeloof](https://twitter.com/ndeloof). Thanks!)
-
----
-
-## What's this application?
-
-- It is a DockerCoin miner! 💰🐳📦🚢
-
---
-
-- No, you can't buy coffee with DockerCoins
-
---
-
-- How DockerCoins works:
-
-  - `worker` asks to `rng` to generate a few random bytes
-
-  - `worker` feeds these bytes into `hasher`
-
-  - and repeat forever!
-
-  - every second, `worker` updates `redis` to indicate how many loops were done
-
-  - `webui` queries `redis`, and computes and exposes "hashing speed" in your browser
-
----
-
-## Getting the application source code
-
-- We will clone the GitHub repository
-
-- The repository also contains scripts and tools that we will use through the workshop
-
-.exercise[
-
-<!--
-```bash
-[ -d orchestration-workshop ] && mv orchestration-workshop orchestration-workshop.$$
-```
--->
-
-- Clone the repository on `node1`:
-  ```bash
-  git clone git://github.com/jpetazzo/orchestration-workshop
-  ```
-
-]
-
-(You can also fork the repository on GitHub and clone your fork if you prefer that.)
-
----
-
-# Running the application
-
-Without further ado, let's start our application.
-
-.exercise[
-
-- Go to the `dockercoins` directory, in the cloned repo:
-  ```bash
-  cd ~/orchestration-workshop/dockercoins
-  ```
-
-- Use Compose to build and run all containers:
-  ```bash
-  docker-compose up
-  ```
-
-]
-
-Compose tells Docker to build all container images (pulling
-the corresponding base images), then starts all containers,
-and displays aggregated logs.
-
----
-
-## Lots of logs
-
-- The application continuously generates logs
-
-- We can see the `worker` service making requests to `rng` and `hasher`
-
-- Let's put that in the background
-
-.exercise[
-
-- Stop the application by hitting `^C`
-
-<!--
-```meta
-^C
-```
--->
-
-]
-
-- `^C` stops all containers by sending them the `TERM` signal
-
-- Some containers exit immediately, others take longer
-  <br/>(because they don't handle `SIGTERM` and end up being killed after a 10s timeout)
-
----
-
-## Restarting in the background
-
-- Many flags and commands of Compose are modeled after those of `docker`
-
-.exercise[
-
-- Start the app in the background with the `-d` option:
-  ```bash
-  docker-compose up -d
-  ```
-
-- Check that our app is running with the `ps` command:
-  ```bash
-  docker-compose ps
-  ```
-
-]
-
-`docker-compose ps` also shows the ports exposed by the application.
-
----
-
-class: extra-details
-
-## Viewing logs
-
-- The `docker-compose logs` command works like `docker logs`
-
-.exercise[
-
-- View all logs since container creation and exit when done:
-  ```bash
-  docker-compose logs
-  ```
-
-- Stream container logs, starting at the last 10 lines for each container:
-  ```bash
-  docker-compose logs --tail 10 --follow
-  ```
-
-<!--
-```meta
-^C
-```
--->
-
-]
-
-Tip: use `^S` and `^Q` to pause/resume log output.
-
----
-
-class: extra-details
-
-## Upgrading from Compose 1.6
-
-.warning[The `logs` command has changed between Compose 1.6 and 1.7!]
-
-- Up to 1.6
-
-  - `docker-compose logs` is the equivalent of `logs --follow`
-
-  - `docker-compose logs` must be restarted if containers are added
-
-- Since 1.7
-
-  - `--follow` must be specified explicitly
-
-  - new containers are automatically picked up by `docker-compose logs`
-
----
-
-## Connecting to the web UI
-
-- The `webui` container exposes a web dashboard; let's view it
-
-.exercise[
-
-- With a web browser, connect to `node1` on port 8000
-
-- Remember: the `nodeX` aliases are valid only on the nodes themselves
-
-- In your browser, you need to enter the IP address of your node
-
-]
-
-You should see a speed of approximately 4 hashes/second.
-
-More precisely: 4 hashes/second, with regular dips down to zero.
-<br/>This is because Jérôme is incapable of writing good frontend code.
-<br/>Don't ask. Seriously, don't ask. This is embarrassing.
-
----
-
-class: extra-details
-
-## Why does the speed seem irregular?
-
-- The app actually has a constant, steady speed: 3.33 hashes/second
-  <br/>
-  (which corresponds to 1 hash every 0.3 seconds, for *reasons*)
-
-- The worker doesn't update the counter after every loop, but up to once per second
-
-- The speed is computed by the browser, checking the counter about once per second
-
-- Between two consecutive updates, the counter will increase either by 4, or by 0
-
-- The perceived speed will therefore be 4 - 4 - 4 - 0 - 4 - 4 - etc.
-
-*We told you to not ask!!!*
-
----
-
-## Scaling up the application
-
-- Our goal is to make that performance graph go up (without changing a line of code!)
-
---
-
-- Before trying to scale the application, we'll figure out if we need more resources
-
-  (CPU, RAM...)
-
-- For that, we will use good old UNIX tools on our Docker node
-
----
-
-## Looking at resource usage
-
-- Let's look at CPU, memory, and I/O usage
-
-.exercise[
-
-- run `top` to see CPU and memory usage (you should see idle cycles)
-
-- run `vmstat 3` to see I/O usage (si/so/bi/bo)
-  <br/>(the 4 numbers should be almost zero, except `bo` for logging)
-
-]
-
-We have available resources.
-
-- Why?
-- How can we use them?
-
----
-
-## Scaling workers on a single node
-
-- Docker Compose supports scaling
-- Let's scale `worker` and see what happens!
-
-.exercise[
-
-- Start one more `worker` container:
-  ```bash
-  docker-compose scale worker=2
-  ```
-
-- Look at the performance graph (it should show a x2 improvement)
-
-- Look at the aggregated logs of our containers (`worker_2` should show up)
-
-- Look at the impact on CPU load with e.g. top (it should be negligible)
-
-]
-
----
-
-## Adding more workers
-
-- Great, let's add more workers and call it a day, then!
-
-.exercise[
-
-- Start eight more `worker` containers:
-  ```bash
-  docker-compose scale worker=10
-  ```
-
-- Look at the performance graph: does it show a x10 improvement?
-
-- Look at the aggregated logs of our containers
-
-- Look at the impact on CPU load and memory usage
-
-<!--
-```bash
-sleep 5
-killall docker-compose
-```
--->
-
-]
-
----
-
-# Identifying bottlenecks
-
-- You should have seen a 3x speed bump (not 10x)
-
-- Adding workers didn't result in linear improvement
-
-- *Something else* is slowing us down
-
---
-
-- ... But what?
-
---
-
-- The code doesn't have instrumentation
-
-- Let's use state-of-the-art HTTP performance analysis!
-  <br/>(i.e. good old tools like `ab`, `httping`...)
-
----
-
-## Accessing internal services
-
-- `rng` and `hasher` are exposed on ports 8001 and 8002
-
-- This is declared in the Compose file:
-
-  ```yaml
-    ...
-    rng:
-      build: rng
-      ports:
-      - "8001:80"
-
-    hasher:
-      build: hasher
-      ports:
-      - "8002:80"
-    ...
-  ```
-
----
-
-## Measuring latency under load
-
-We will use `httping`.
-
-.exercise[
-
-- Check the latency of `rng`:
-  ```bash
-  httping -c 10 localhost:8001
-  ```
-
-- Check the latency of `hasher`:
-  ```bash
-  httping -c 10 localhost:8002
-  ```
-
-]
-
-`rng` has a much higher latency than `hasher`.
-
----
-
-## Let's draw hasty conclusions
-
-- The bottleneck seems to be `rng`
-
-- *What if* we don't have enough entropy and can't generate enough random numbers?
-
-- We need to scale out the `rng` service on multiple machines!
-
-Note: this is a fiction! We have enough entropy. But we need a pretext to scale out.
-
-(In fact, the code of `rng` uses `/dev/urandom`, which never runs out of entropy...
-<br/>
-...and is [just as good as `/dev/random`](http://www.slideshare.net/PacSecJP/filippo-plain-simple-reality-of-entropy).)
-
----
-
-## Clean up
-
-- Before moving on, let's remove those containers
-
-.exercise[
-
-- Tell Compose to remove everything:
-  ```bash
-  docker-compose down
-  ```
-
-]
-
----
-
-class: title
-
-Scaling out
-
----
-
-# SwarmKit
-
-- [SwarmKit](https://github.com/docker/swarmkit) is an open source
-  toolkit to build multi-node systems
-
-- It is a reusable library, like libcontainer, libnetwork, vpnkit ...
-
-- It is a plumbing part of the Docker ecosystem
-
----
-
-## SwarmKit features
-
-- Highly-available, distributed store based on [Raft](
-  https://en.wikipedia.org/wiki/Raft_%28computer_science%29)
-  <br/>(avoids depending on an external store: easier to deploy; higher performance)
-
-- Dynamic reconfiguration of Raft without interrupting cluster operations
-
-- *Services* managed with a *declarative API*
-  <br/>(implementing *desired state* and *reconciliation loop*)
-
-- Integration with overlay networks and load balancing
-
-- Strong emphasis on security:
-
-  - automatic TLS keying and signing; automatic cert rotation
-  - full encryption of the data plane; automatic key rotation
-  - least privilege architecture (single-node compromise ≠ cluster compromise)
-  - on-disk encryption with optional passphrase
-
----
-
-class: extra-details
-
-## Where is the key/value store?
-
-- Many orchestration systems use a key/value store backed by a consensus algorithm
-  <br/>
-  (k8s→etcd→Raft, mesos→zookeeper→ZAB, etc.)
-
-- SwarmKit implements the Raft algorithm directly
-  <br/>
-  (Nomad is similar; thanks [@cbednarski](https://twitter.com/@cbednarski),
-  [@diptanu](https://twitter.com/diptanu) and others for point it out!)
-
-- Analogy courtesy of [@aluzzardi](https://twitter.com/aluzzardi):
-
-  *It's like B-Trees and RDBMS. They are different layers, often
-  associated. But you don't need to bring up a full SQL server when
-  all you need is to index some data.*
-
-- As a result, the orchestrator has direct access to the data
-  <br/>
-  (the main copy of the data is stored in the orchestrator's memory)
-
-- Simpler, easier to deploy and operate; also faster
-
----
-
-## SwarmKit concepts (1/2)
-
-- A *cluster* will be at least one *node* (preferably more)
-
-- A *node* can be a *manager* or a *worker*
-
-- A *manager* actively takes part in the Raft consensus, and keeps the Raft log
-
-- You can talk to a *manager* using the SwarmKit API
-
-- One *manager* is elected as the *leader*; other managers merely forward requests to it
-
-- The *workers* get their instructions from the *managers*
-
-- Both *workers* and *managers* can run containers
-
----
-
-## Illustration
-
-![Illustration](swarm-mode.svg)
-
----
-
-## SwarmKit concepts (2/2)
-
-- The *managers* expose the SwarmKit API
-
-- Using the API, you can indicate that you want to run a *service*
-
-- A *service* is specified by its *desired state*: which image, how many instances...
-
-- The *leader* uses different subsystems to break down services into *tasks*:
-  <br/>orchestrator, scheduler, allocator, dispatcher
-
-- A *task* corresponds to a specific container, assigned to a specific *node*
-
-- *Nodes* know which *tasks* should be running, and will start or stop containers accordingly (through the Docker Engine API)
-
-You can refer to the [NOMENCLATURE](https://github.com/docker/swarmkit/blob/master/design/nomenclature.md) in the SwarmKit repo for more details.
-
----
-
-## Swarm Mode
-
-- Since version 1.12, Docker Engine embeds SwarmKit
-
-- All the SwarmKit features are "asleep" until you enable "Swarm Mode"
-
-- Examples of Swarm Mode commands:
-
-  - `docker swarm` (enable Swarm mode; join a Swarm; adjust cluster parameters)
-
-  - `docker node` (view nodes; promote/demote managers; manage nodes)
-
-  - `docker service` (create and manage services)
-
-???
-
-- The Docker API exposes the same concepts
-
-- The SwarmKit API is also exposed (on a separate socket)
-
----
-
-## You need to enable Swarm mode to use the new stuff
-
-- By default, all this new code is inactive
-
-- Swarm Mode can be enabled, "unlocking" SwarmKit functions
-  <br/>(services, out-of-the-box overlay networks, etc.)
-
-.exercise[
-
-- Try a Swarm-specific command:
-  ```bash
-  docker node ls
-  ```
-
-]
-
---
-
-You will get an error message:
-```
-Error response from daemon: This node is not a swarm manager. [...]
-```
-
----
-
-# Creating our first Swarm
-
-- The cluster is initialized with `docker swarm init`
-
-- This should be executed on a first, seed node
-
-- .warning[DO NOT execute `docker swarm init` on multiple nodes!]
-
-  You would have multiple disjoint clusters.
-
-.exercise[
-
-- Create our cluster from node1:
-  ```bash
-  docker swarm init
-  ```
-
-]
-
---
-
-class: advertise-addr
-
-If Docker tells you that it `could not choose an IP address to advertise`, see next slide!
-
----
-
-class: advertise-addr
-
-## IP address to advertise
-
-- When running in Swarm mode, each node *advertises* its address to the others
-  <br/>
-  (i.e. it tells them *"you can contact me on 10.1.2.3:2377"*)
-
-- If the node has only one IP address (other than 127.0.0.1), it is used automatically
-
-- If the node has multiple IP addresses, you **must** specify which one to use
-  <br/>
-  (Docker refuses to pick one randomly)
-
-- You can specify an IP address or an interface name
-  <br/>(in the latter case, Docker will read the IP address of the interface and use it)
-
-- You can also specify a port number
-  <br/>(otherwise, the default port 2377 will be used)
-
----
-
-class: advertise-addr
-
-## Which IP address should be advertised?
-
-- If your nodes have only one IP address, it's safe to let autodetection do the job
-
-  .small[(Except if your instances have different private and public addresses, e.g.
-  on EC2, and you are building a Swarm involving nodes inside and outside the
-  private network: then you should advertise the public address.)]
-
-- If your nodes have multiple IP addresses, pick an address which is reachable
-  *by every other node* of the Swarm
-
-- If you are using [play-with-docker](http://play-with-docker.com/), use the IP
-  address shown next to the node name
-
-  .small[(This is the address of your node on your private internal overlay network.
-  The other address that you might see is the address of your node on the
-  `docker_gwbridge` network, which is used for outbound traffic.)]
-
-Examples:
-
-```bash
-docker swarm init --advertise-addr 10.0.9.2
-docker swarm init --advertise-addr eth0:7777
-```
-
----
-
-class: extra-details
-
-## Using a separate interface for the data path
-
-- You can use different interfaces (or IP addresses) for control and data
-
-- You set the _control plane path_ with `--advertise-addr`
-
-  (This will be used for SwarmKit manager/worker communication, leader election, etc.)
-
-- You set the _data plane path_ with `--data-path-addr`
-
-  (This will be used for traffic between containers)
-
-- Both flags can accept either an IP address, or an interface name
-
-  (When specifying an interface name, Docker will use its first IP address)
-
----
-
-## Token generation
-
-- In the output of `docker swarm init`, we have a message
-  confirming that our node is now the (single) manager:
-
-  ```
-  Swarm initialized: current node (8jud...) is now a manager.
-  ```
-
-- Docker generated two security tokens (like passphrases or passwords) for our cluster
-
-- The CLI shows us the command to use on other nodes to add them to the cluster using the "worker"
-  security token:
-
-  ```
-    To add a worker to this swarm, run the following command:
-      docker swarm join \
-      --token SWMTKN-1-59fl4ak4nqjmao1ofttrc4eprhrola2l87... \
-      172.31.4.182:2377
-  ```
-
----
-
-class: extra-details
-
-## Checking that Swarm mode is enabled
-
-.exercise[
-
-- Run the traditional `docker info` command:
-  ```bash
-  docker info
-  ```
-
-]
-
-The output should include:
-
-```
-Swarm: active
- NodeID: 8jud7o8dax3zxbags3f8yox4b
- Is Manager: true
- ClusterID: 2vcw2oa9rjps3a24m91xhvv0c
- ...
-```
-
----
-
-## Running our first Swarm mode command
-
-- Let's retry the exact same command as earlier
-
-.exercise[
-
-- List the nodes (well, the only node) of our cluster:
-  ```bash
-  docker node ls
-  ```
-
-]
-
-The output should look like the following:
-```
-ID             HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
-8jud...ox4b *  node1     Ready   Active        Leader
-```
-
----
-
-## Adding nodes to the Swarm
-
-- A cluster with one node is not a lot of fun
-
-- Let's add `node2`!
-
-- We need the token that was shown earlier
-
---
-
-- You wrote it down, right?
-
---
-
-- Don't panic, we can easily see it again 😏
-
----
-
-## Adding nodes to the Swarm
-
-.exercise[
-
-- Show the token again:
-  ```bash
-  docker swarm join-token worker
-  ```
-
-- Switch to `node2`
-
-- Copy-paste the `docker swarm join ...` command
-  <br/>(that was displayed just before)
-
-]
-
----
-
-class: extra-details
-
-## Check that the node was added correctly
-
-- Stay on `node2` for now!
-
-.exercise[
-
-- We can still use `docker info` to verify that the node is part of the Swarm:
-  ```bash
-  docker info | grep ^Swarm
-  ```
-
-]
-
-- However, Swarm commands will not work; try, for instance:
-  ```
-  docker node ls
-  ```
-
-- This is because the node that we added is currently a *worker*
-
-- Only *managers* can accept Swarm-specific commands
-
----
-
-## View our two-node cluster
-
-- Let's go back to `node1` and see what our cluster looks like
-
-.exercise[
-
-- Switch back to `node1`
-
-- View the cluster from `node1`, which is a manager:
-  ```bash
-  docker node ls
-  ```
-
-]
-
-The output should be similar to the following:
-```
-ID             HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
-8jud...ox4b *  node1     Ready   Active        Leader
-ehb0...4fvx    node2     Ready   Active
-```
-
----
-
-class: docker-machine
-
-## Adding nodes using the Docker API
-
-- We don't have to SSH into the other nodes, we can use the Docker API
-
-- If you are using Play-With-Docker:
-
-  - the nodes expose the Docker API over port 2375/tcp, without authentication
-
-  - we will connect by setting the `DOCKER_HOST` environment variable
-
-- Otherwise:
-
-  - the nodes expose the Docker API over port 2376/tcp, with TLS mutual authentication
-
-  - we will use Docker Machine to set the correct environment variables
-    <br/>(the nodes have been suitably pre-configured to be controlled through `node1`)
-
----
-
-class: docker-machine
-
-# Docker Machine
-
-- Docker Machine has two primary uses:
-
-  - provisioning cloud instances running the Docker Engine
-
-  - managing local Docker VMs within e.g. VirtualBox
-
-- Docker Machine is purely optional
-
-- It makes it easy to create, upgrade, manage... Docker hosts:
-
-  - on your favorite cloud provider
-
-  - locally (e.g. to test clustering, or different versions)
-
-  - across different cloud providers
-
----
-
-class: self-paced, docker-machine
-
-## If you're using Play-With-Docker ...
-
-- You won't need to use Docker Machine
-
-- Instead, to "talk" to another node, we'll just set `DOCKER_HOST`
-
-- You can skip the exercises telling you to do things with Docker Machine!
-
----
-
-class: docker-machine
-
-## Docker Machine basic usage
-
-- We will learn two commands:
-
-  - `docker-machine ls` (list existing hosts)
-
-  - `docker-machine env` (switch to a specific host)
-
-.exercise[
-
-- List configured hosts:
-  ```bash
-  docker-machine ls
-  ```
-
-]
-
-You should see your 5 nodes.
-
----
-
-class: in-person, docker-machine
-
-## How did we make our 5 nodes show up there?
-
-*For the curious...*
-
-- This was done by our VM provisioning scripts
-
-- After setting up everything else, `node1` adds the 5 nodes
-  to the local Docker Machine configuration
-  (located in `$HOME/.docker/machine`)
-
-- Nodes are added using [Docker Machine generic driver](https://docs.docker.com/machine/drivers/generic/)
-
-  (It skips machine provisioning and jumps straight to the configuration phase)
-
-- Docker Machine creates TLS certificates and deploys them to the nodes through SSH
-
----
-
-class: docker-machine
-
-## Using Docker Machine to communicate with a node
-
-- To select a node, use `eval $(docker-machine env nodeX)`
-
-- This sets a number of environment variables
-
-- To unset these variables, use `eval $(docker-machine env -u)`
-
-.exercise[
-
-- View the variables used by Docker Machine:
-  ```bash
-  docker-machine env node3
-  ```
-
-]
-
-(This shows which variables *would* be set by Docker Machine; but it doesn't change them.)
-
----
-
-class: docker-machine
-
-## Getting the token
-
-- First, let's store the join token in a variable
-
-- This must be done from a manager
-
-.exercise[
-
-- Make sure we talk to the local node, or `node1`:
-  ```bash
-  eval $(docker-machine env -u)
-  ```
-
-- Get the join token:
-  ```bash
-  TOKEN=$(docker swarm join-token -q worker)
-  ```
-
-]
-
----
-
-class: docker-machine
-
-## Change the node targeted by the Docker CLI
-
-- We need to set the right environment variables to communicate with `node3`
-
-.exercise[
-
-- If you're using Play-With-Docker:
-  ```bash
-  export DOCKER_HOST=tcp://node3:2375
-  ```
-
-- Otherwise, use Docker Machine:
-  ```bash
-  eval $(docker-machine env node3)
-  ```
-
-]
-
----
-
-class: docker-machine
-
-## Checking which node we're talking to
-
-- Let's use the Docker API to ask "who are you?" to the remote node
-
-.exercise[
-
-- Extract the node name from the output of `docker info`:
-  ```bash
-  docker info | grep ^Name
-  ```
-
-]
-
-This should tell us that we are talking to `node3`.
-
-Note: it can be useful to use a [custom shell prompt](
-https://github.com/jpetazzo/orchestration-workshop/blob/master/prepare-vms/scripts/postprep.rc#L68)
-reflecting the `DOCKER_HOST` variable.
-
----
-
-class: docker-machine
-
-## Adding a node through the Docker API
-
-- We are going to use the same `docker swarm join` command as before
-
-.exercise[
-
-- Add `node3` to the Swarm:
-  ```bash
-  docker swarm join --token $TOKEN node1:2377
-  ```
-
-]
-
----
-
-class: docker-machine
-
-## Going back to the local node
-
-- We need to revert the environment variable(s) that we had set previously
-
-.exercise[
-
-- If you're using Play-With-Docker, just clear `DOCKER_HOST`:
-  ```bash
-  unset DOCKER_HOST
-  ```
-
-- Otherwise, use Docker Machine to reset all the relevant variables:
-  ```bash
-  eval $(docker-machine env -u)
-  ```
-
-]
-
-From that point, we are communicating with `node1` again.
-
----
-
-class: docker-machine
-
-## Checking the composition of our cluster
-
-- Now that we're talking to `node1` again, we can use management commands
-
-.exercise[
-
-- Check that the node is here:
-  ```bash
-  docker node ls
-  ```
-
-]
-
----
-
-class: under-the-hood
-
-## Under the hood: docker swarm init
-
-When we do `docker swarm init`:
-
-- a keypair is created for the root CA of our Swarm
-
-- a keypair is created for the first node
-
-- a certificate is issued for this node
-
-- the join tokens are created
-
----
-
-class: under-the-hood
-
-## Under the hood: join tokens
-
-There is one token to *join as a worker*, and another to *join as a manager*.
-
-The join tokens have two parts:
-
-- a secret key (preventing unauthorized nodes from joining)
-
-- a fingerprint of the root CA certificate (preventing MITM attacks)
-
-If a token is compromised, it can be rotated instantly with:
-```
-docker swarm join-token --rotate <worker|manager>
-```
-
----
-
-class: under-the-hood
-
-## Under the hood: docker swarm join
-
-When a node joins the Swarm:
-
-- it is issued its own keypair, signed by the root CA
-
-- if the node is a manager:
-
-  - it joins the Raft consensus
-  - it connects to the current leader
-  - it accepts connections from worker nodes
-
-- if the node is a worker:
-
-  - it connects to one of the managers (leader or follower)
-
----
-
-class: under-the-hood
-
-## Under the hood: cluster communication
-
-- The *control plane* is encrypted with AES-GCM; keys are rotated every 12 hours
-
-- Authentication is done with mutual TLS; certificates are rotated every 90 days
-
-  (`docker swarm update` allows to change this delay or to use an external CA)
-
-- The *data plane* (communication between containers) is not encrypted by default
-
-  (but this can be activated on a by-network basis, using IPSEC,
-  leveraging hardware crypto if available)
-
----
-
-class: under-the-hood
-
-## Under the hood: I want to know more!
-
-Revisit SwarmKit concepts:
-
-- Docker 1.12 Swarm Mode Deep Dive Part 1: Topology
-  ([video](https://www.youtube.com/watch?v=dooPhkXT9yI))
-
-- Docker 1.12 Swarm Mode Deep Dive Part 2: Orchestration
-  ([video](https://www.youtube.com/watch?v=_F6PSP-qhdA))
-
-Some presentations from the Docker Distributed Systems Summit in Berlin:
-
-- Heart of the SwarmKit: Topology Management
-  ([slides](https://speakerdeck.com/aluzzardi/heart-of-the-swarmkit-topology-management))
-
-- Heart of the SwarmKit: Store, Topology & Object Model
-  ([slides](http://www.slideshare.net/Docker/heart-of-the-swarmkit-store-topology-object-model))
-  ([video](https://www.youtube.com/watch?v=EmePhjGnCXY))
-
----
-
-## Adding more manager nodes
-
-- Right now, we have only one manager (node1)
-
-- If we lose it, we lose quorum - and that's *very bad!*
-
-- Containers running on other nodes will be fine ...
-
-- But we won't be able to get or set anything related to the cluster
-
-- If the manager is permanently gone, we will have to do a manual repair!
-
-- Nobody wants to do that ... so let's make our cluster highly available
-
----
-
-class: self-paced
-
-## Adding more managers
-
-With Play-With-Docker:
-
-```bash
-TOKEN=$(docker swarm join-token -q manager)
-for N in $(seq 4 5); do
-  export DOCKER_HOST=tcp://node$N:2375
-  docker swarm join --token $TOKEN node1:2377
-done
-unset DOCKER_HOST
-```
-
----
-
-class: docker-machine
-
-## Adding more managers
-
-With Docker Machine:
-
-```bash
-TOKEN=$(docker swarm join-token -q manager)
-for N in $(seq 4 5); do
-  eval $(docker-machine env node$N)
-  docker swarm join --token $TOKEN node1:2377
-done
-eval $(docker-machine env -u)
-```
-
----
-
-class: in-person
-
-## Building our full cluster
-
-- We could SSH to nodes 3, 4, 5; and copy-paste the command
-
---
-
-class: in-person
-
-- Or we could use the AWESOME POWER OF THE SHELL!
-
---
-
-class: in-person
-
-![Mario Red Shell](mario-red-shell.png)
-
---
-
-class: in-person
-
-- No, not *that* shell
-
----
-
-class: in-person
-
-## Let's form like Swarm-tron
-
-- Let's get the token, and loop over the remaining nodes with SSH
-
-.exercise[
-
-- Obtain the manager token:
-  ```bash
-  TOKEN=$(docker swarm join-token -q manager)
-  ```
-
-- Loop over the 3 remaining nodes:
-  ```bash
-    for NODE in node3 node4 node5; do
-      ssh $NODE docker swarm join --token $TOKEN node1:2377
-    done
-  ```
-
-]
-
-[That was easy.](https://www.youtube.com/watch?v=3YmMNpbFjp0)
-
----
-
-## You can control the Swarm from any manager node
-
-.exercise[
-
-- Try the following command on a few different nodes:
-  ```bash
-  docker node ls
-  ```
-
-]
-
-On manager nodes:
-<br/>you will see the list of nodes, with a `*` denoting
-the node you're talking to.
-
-On non-manager nodes:
-<br/>you will get an error message telling you that
-the node is not a manager.
-
-As we saw earlier, you can only control the Swarm through a manager node.
-
----
-
-class: self-paced
-
-## Play-With-Docker node status icon
-
-- If you're using Play-With-Docker, you get node status icons
-
-- Node status icons are displayed left of the node name
-
-  - No icon = no Swarm mode detected
-  - Solid blue icon = Swarm manager detected
-  - Blue outline icon = Swarm worker detected
-
-![Play-With-Docker icons](pwd-icons.png)
-
----
-
-## Dynamically changing the role of a node
-
-- We can change the role of a node on the fly:
-
-  `docker node promote XXX` → make XXX a manager
-  <br/>
-  `docker node demote XXX` → make XXX a worker
-
-.exercise[
-
-- See the current list of nodes:
-  ```
-  docker node ls
-  ```
-
-- Promote any worker node to be a manager:
-  ```
-  docker node promote <node_name_or_id>
-  ```
-
-]
-
----
-
-## How many managers do we need?
-
-- 2N+1 nodes can (and will) tolerate N failures
-  <br/>(you can have an even number of managers, but there is no point)
-
---
-
-- 1 manager = no failure
-
-- 3 managers = 1 failure
-
-- 5 managers = 2 failures (or 1 failure during 1 maintenance)
-
-- 7 managers and more = now you might be overdoing it a little bit
-
----
-
-## Why not have *all* nodes be managers?
-
-- Intuitively, it's harder to reach consensus in larger groups
-
-- With Raft, writes have to go to (and be acknowledged by) all nodes
-
-- More nodes = more network traffic
-
-- Bigger network = more latency
-
----
-
-## What would McGyver do?
-
-- If some of your machines are more than 10ms away from each other,
-  <br/>
-  try to break them down in multiple clusters
-  (keeping internal latency low)
-
-- Groups of up to 9 nodes: all of them are managers
-
-- Groups of 10 nodes and up: pick 5 "stable" nodes to be managers
-  <br/>
-  (Cloud pro-tip: use separate auto-scaling groups for managers and workers)
-
-- Groups of more than 100 nodes: watch your managers' CPU and RAM
-
-- Groups of more than 1000 nodes:
-
-  - if you can afford to have fast, stable managers, add more of them
-  - otherwise, break down your nodes in multiple clusters
-
----
-
-## What's the upper limit?
-
-- We don't know!
-
-- Internal testing at Docker Inc.: 1000-10000 nodes is fine
-
-  - deployed to a single cloud region
-
-  - one of the main take-aways was *"you're gonna need a bigger manager"*
-
-- Testing by the community: [4700 heterogenous nodes all over the 'net](https://sematext.com/blog/2016/11/14/docker-swarm-lessons-from-swarm3k/)
-
-  - it just works
-
-  - more nodes require more CPU; more containers require more RAM
-
-  - scheduling of large jobs (70000 containers) is slow, though (working on it!)
-
----
-
-## Real-life deployment methods
-
--- Running commands manually over SSH
-
---
-
-  (lol jk)
-
---
-
-- Using your favorite configuration management tool
-
-- [Docker for AWS](https://docs.docker.com/docker-for-aws/#quickstart)
-
-- [Docker for Azure](https://docs.docker.com/docker-for-azure/)
-
----
-
-# Running our first Swarm service
-
-- How do we run services? Simplified version:
-
-  `docker run` → `docker service create`
-
-.exercise[
-
-- Create a service featuring an Alpine container pinging Google resolvers:
-  ```bash
-  docker service create alpine ping 8.8.8.8
-  ```
-
-- Check the result:
-  ```bash
-  docker service ps <serviceID>
-  ```
-
-]
-
----
-
-## `--detach` for service creation
-
-(New in Docker Engine 17.05)
-
-If you are running Docker 17.05 or later, you will see the following message:
-
-```
-Since --detach=false was not specified, tasks will be created in the background.
-In a future release, --detach=false will become the default.
-```
-
-Let's ignore it for now; but we'll come back to it in just a few minutes!
-
----
-
-## Checking service logs
-
-(New in Docker Engine 17.05)
-
-- Just like `docker logs` shows the output of a specific local container ...
-
-- ... `docker service logs` shows the output of all the containers of a specific service
-
-.exercise[
-
-- Check the output of our ping command:
-  ```bash
-  docker service logs <serviceID>
-  ```
-
-]
-
-Flags `--follow` and `--tail` are available, as well as a few others.
-
-Note: by default, when a container is destroyed (e.g. when scaling down), its logs are lost.
-
----
-
-class: extra-details
-
-## Before Docker Engine 17.05
-
-- Docker 1.13/17.03/17.04 have `docker service logs` as an experimental feature
-  <br/>(available only when enabling the experimental feature flag)
-
-- We have to use `docker logs`, which only works on local containers
-
-- We will have to connect to the node running our container
-  <br/>(unless it was scheduled locally, of course)
-
----
-
-class: extra-details
-
-## Looking up where our container is running
-
-- The `docker service ps` command told us where our container was scheduled
-
-.exercise[
-
-- Look up the `NODE` on which the container is running:
-  ```bash
-  docker service ps <serviceID>
-  ```
-
-- If you use Play-With-Docker, switch to that node's tab, or set `DOCKER_HOST`
-
-- Otherwise, `ssh` into tht node or use `$(eval docker-machine env node...)`
-
-]
-
----
-
-class: extra-details
-
-## Viewing the logs of the container
-
-.exercise[
-
-- See that the container is running and check its ID:
-  ```bash
-  docker ps
-  ```
-
-- View its logs:
-  ```bash
-  docker logs <containerID>
-  ```
-
-- Go back to `node1` afterwards
-
-]
-
----
-
-## Scale our service
-
-- Services can be scaled in a pinch with the `docker service update` command
-
-.exercise[
-
-- Scale the service to ensure 2 copies per node:
-  ```bash
-  docker service update <serviceID> --replicas 10
-  ```
-
-- Check that we have two containers on the current node:
-  ```bash
-  docker ps
-  ```
-
-]
-
----
-
-## View deployment progress
-
-(New in Docker Engine 17.05)
-
-- Commands that create/update/delete services can run with `--detach=false`
-
-- The CLI will show the status of the command, and exit once it's done working
-
-.exercise[
-
-- Scale the service to ensure 3 copies per node:
-  ```bash
-  docker service update <serviceID> --replicas 15 --detach=false
-  ```
-
-]
-
-Note: `--detach=false` will eventually become the default.
-
-With older versions, you can use e.g.: `watch docker service ps <serviceID>`
-
----
-
-## Expose a service
-
-- Services can be exposed, with two special properties:
-
-  - the public port is available on *every node of the Swarm*,
-
-  - requests coming on the public port are load balanced across all instances.
-
-- This is achieved with option `-p/--publish`; as an approximation:
-
-  `docker run -p → docker service create -p`
-
-- If you indicate a single port number, it will be mapped on a port
-  starting at 30000
-  <br/>(vs. 32768 for single container mapping)
-
-- You can indicate two port numbers to set the public port number
-  <br/>(just like with `docker run -p`)
-
----
-
-## Expose ElasticSearch on its default port
-
-.exercise[
-
-- Create an ElasticSearch service (and give it a name while we're at it):
-  ```bash
-  docker service create --name search --publish 9200:9200 --replicas 7 \
-         --detach=false elasticsearch`:2`
-  ```
-
-]
-
-Note: don't forget the **:2**!
-
-The latest version of the ElasticSearch image won't start without mandatory configuration.
-
----
-
-## Tasks lifecycle
-
-- During the deployment, you will be able to see multiple states:
-
-  - assigned (the task has been assigned to a specific node)
-
-  - preparing (this mostly means "pulling the image")
-
-  - starting
-
-  - running
-
-- When a task is terminated (stopped, killed...) it cannot be restarted
-
-  (A replacement task will be created)
-
----
-
-class: extra-details
-
-![diagram showing what happens during docker service create, courtesy of @aluzzardi](docker-service-create.svg)
-
----
-
-## Test our service
-
-- We mapped port 9200 on the nodes, to port 9200 in the containers
-
-- Let's try to reach that port!
-
-.exercise[
-
-- Try the following command:
-  ```bash
-  curl localhost:9200
-  ```
-
-]
-
-(If you get `Connection refused`: congratulations, you are very fast indeed! Just try again.)
-
-ElasticSearch serves a little JSON document with some basic information
-about this instance; including a randomly-generated super-hero name.
-
----
-
-## Test the load balancing
-
-- If we repeat our `curl` command multiple times, we will see different names
-
-.exercise[
-
-- Send 10 requests, and see which instances serve them:
-  ```bash
-    for N in $(seq 1 10); do
-      curl -s localhost:9200 | jq .name
-    done
-  ```
-
-]
-
-Note: if you don't have `jq` on your Play-With-Docker instance, just install it:
-```bash
-apk add --no-cache jq
-```
-
----
-
-## Load balancing results
-
-Traffic is handled by our clusters [TCP routing mesh](
-https://docs.docker.com/engine/swarm/ingress/).
-
-Each request is served by one of the 7 instances, in rotation.
-
-Note: if you try to access the service from your browser,
-you will probably see the same
-instance name over and over, because your browser (unlike curl) will try
-to re-use the same connection.
-
----
-
-## Under the hood of the TCP routing mesh
-
-- Load balancing is done by IPVS
-
-- IPVS is a high-performance, in-kernel load balancer
-
-- It's been around for a long time (merged in the kernel since 2.4)
-
-- Each node runs a local load balancer
-
-  (Allowing connections to be routed directly to the destination,
-  without extra hops)
-
----
-
-## Managing inbound traffic
-
-There are many ways to deal with inbound traffic on a Swarm cluster.
-
-- Put all (or a subset) of your nodes in a DNS `A` record
-
-- Assign your nodes (or a subset) to an ELB
-
-- Use a virtual IP and make sure that it is assigned to an "alive" node
-
-- etc.
-
----
-
-class: btw-labels
-
-## Managing HTTP traffic
-
-- The TCP routing mesh doesn't parse HTTP headers
-
-- If you want to place multiple HTTP services on port 80, you need something more
-
-- You can setup NGINX or HAProxy on port 80 to do the virtual host switching
-
-- Docker Universal Control Plane provides its own [HTTP routing mesh](
-  https://docs.docker.com/datacenter/ucp/2.1/guides/admin/configure/use-domain-names-to-access-services/)
-
-  - add a specific label starting with `com.docker.ucp.mesh.http` to your services
-
-  - labels are detected automatically and dynamically update the configuration
-
----
-
-class: btw-labels
-
-## You should use labels
-
-- Labels are a great way to attach arbitrary information to services
-
-- Examples:
-
-  - HTTP vhost of a web app or web service
-
-  - backup schedule for a stateful service
-
-  - owner of a service (for billing, paging...)
-
-  - etc.
-
----
-
-## Pro-tip for ingress traffic management
-
-- It is possible to use *local* networks with Swarm services
-
-- This means that you can do something like this:
-  ```bash
-  docker service create --network host --mode global traefik ...
-  ```
-
-  (This runs the `traefik` load balancer on each node of your cluster, in the `host` network)
-
-- This gives you native performance (no iptables, no proxy, no nothing!)
-
-- The load balancer will "see" the clients' IP addresses
-
-- But: a container cannot simultaneously be in the `host` network and another network
-
-  (You will have to route traffic to containers using exposed ports or UNIX sockets)
-
----
-
-class: extra-details
-
-## Using local networks (`host`, `macvlan` ...) with Swarm services
-
-- Using the `host` network is fairly straightforward
-
-  (With the caveats described on the previous slide)
-
-- It is also possible to use drivers like `macvlan`
-
-  - see [this guide](
-https://docs.docker.com/engine/userguide/networking/get-started-macvlan/
-) to get started on `macvlan`
-
-  - see [this PR](https://github.com/moby/moby/pull/32981) for more information about local network drivers in Swarm mode
-
----
-
-## Visualize container placement
-
-- Let's leverage the Docker API!
-
-.exercise[
-
-- Get the source code of this simple-yet-beautiful visualization app:
-  ```bash
-  cd ~
-  git clone git://github.com/dockersamples/docker-swarm-visualizer
-  ```
-
-- Build and run the Swarm visualizer:
-  ```bash
-  cd docker-swarm-visualizer
-  docker-compose up -d
-  ```
-
-]
-
----
-
-## Connect to the visualization webapp
-
-- It runs a web server on port 8080
-
-.exercise[
-
-- Point your browser to port 8080 of your node1's public ip
-
-  (If you use Play-With-Docker, click on the (8080) badge)
-
-]
-
-- The webapp updates the display automatically (you don't need to reload the page)
-
-- It only shows Swarm services (not standalone containers)
-
-- It shows when nodes go down
-
-- It has some glitches (it's not Carrier-Grade Enterprise-Compliant ISO-9001 software)
-
----
-
-## Why This Is More Important Than You Think
-
-- The visualizer accesses the Docker API *from within a container*
-
-- This is a common pattern: run container management tools *in containers*
-
-- Instead of viewing your cluster, this could take care of logging, metrics, autoscaling ...
-
-- We can run it within a service, too! We won't do it, but the command would look like:
-
-  ```bash
-    docker service create \
-      --mount source=/var/run/docker.sock,type=bind,target=/var/run/docker.sock \
-      --name viz --constraint node.role==manager ...
-  ```
-
-Credits: the visualization code was written by
-[Francisco Miranda](https://github.com/maroshii).
-<br/>
-[Mano Marks](https://twitter.com/manomarks) adapted
-it to Swarm and maintains it.
-
----
-
-## Terminate our services
-
-- Before moving on, we will remove those services
-
-- `docker service rm` can accept multiple services names or IDs
-
-- `docker service ls` can accept the `-q` flag
-
-- A Shell snippet a day keeps the cruft away
-
-.exercise[
-
-- Remove all services with this one liner:
-  ```bash
-  docker service ls -q | xargs docker service rm
-  ```
-
-]
-
----
-
-class: title
-
-Our app on Swarm
-
----
-
-## What's on the menu?
-
-In this part, we will:
-
-- **build** images for our app,
-
-- **ship** these images with a registry,
-
-- **run** services using these images.
-
----
-
-## Why do we need to ship our images?
-
-- When we do `docker-compose up`, images are built for our services
-
-- These images are present only on the local node
-
-- We need these images to be distributed on the whole Swarm
-
-- The easiest way to achieve that is to use a Docker registry
-
-- Once our images are on a registry, we can reference them when
-  creating our services
-
----
-
-class: extra-details
-
-## Build, ship, and run, for a single service
-
-If we had only one service (built from a `Dockerfile` in the
-current directory), our workflow could look like this:
-
-```
-docker build -t jpetazzo/doublerainbow:v0.1 .
-docker push jpetazzo/doublerainbow:v0.1
-docker service create jpetazzo/doublerainbow:v0.1
-```
-
-We just have to adapt this to our application, which has 4 services!
-
----
-
-## The plan
-
-- Build on our local node (`node1`)
-
-- Tag images so that they are named `localhost:5000/servicename`
-
-- Upload them to a registry
-
-- Create services using the images
-
----
-
-## Which registry do we want to use?
-
-.small[
-
-- **Docker Hub**
-
-  - hosted by Docker Inc.
-  - requires an account (free, no credit card needed)
-  - images will be public (unless you pay)
-  - located in AWS EC2 us-east-1
-
-- **Docker Trusted Registry**
-
-  - self-hosted commercial product
-  - requires a subscription (free 30-day trial available)
-  - images can be public or private
-  - located wherever you want
-
-- **Docker open source registry**
-
-  - self-hosted barebones repository hosting
-  - doesn't require anything
-  - doesn't come with anything either
-  - located wherever you want
-
-]
-
----
-
-class: extra-details
-
-## Using Docker Hub
-
-*If we wanted to use the Docker Hub...*
-
-<!--
-```meta
-^{
-```
--->
-
-- We would log into the Docker Hub:
-  ```bash
-  docker login
-  ```
-
-- And in the following slides, we would use our Docker Hub login
-  (e.g. `jpetazzo`) instead of the registry address (i.e. `127.0.0.1:5000`)
-
-<!--
-```meta
-^}
-```
--->
-
----
-
-class: extra-details
-
-## Using Docker Trusted Registry
-
-*If we wanted to use DTR, we would...*
-
-- Make sure we have a Docker Hub account
-
-- [Activate a Docker Datacenter subscription](
-  https://hub.docker.com/enterprise/trial/)
-
-- Install DTR on our machines
-
-- Use `dtraddress:port/user` instead of the registry address
-
-*This is out of the scope of this workshop!*
-
----
-
-## Using the open source registry
-
-- We need to run a `registry:2` container
-  <br/>(make sure you specify tag `:2` to run the new version!)
-
-- It will store images and layers to the local filesystem
-  <br/>(but you can add a config file to use S3, Swift, etc.)
-
-- Docker *requires* TLS when communicating with the registry
-
-  - unless for registries on `127.0.0.0/8` (i.e. `localhost`)
-
-  - or with the Engine flag `--insecure-registry`
-
-<!-- -->
-
-- Our strategy: publish the registry container on port 5000,
-  <br/>so that it's available through `127.0.0.1:5000` on each node
-
----
-
-class: manual-btp
-
-# Deploying a local registry
-
-- We will create a single-instance service, publishing its port
-  on the whole cluster
-
-.exercise[
-
-- Create the registry service:
-  ```bash
-  docker service create --name registry --publish 5000:5000 registry:2
-  ```
-
-- Try the following command, until it returns `{"repositories":[]}`:
-  ```bash
-  curl 127.0.0.1:5000/v2/_catalog
-  ```
-
-]
-
-(Retry a few times, it might take 10-20 seconds for the container to be started. Patience.)
-
----
-
-class: manual-btp
-
-## Testing our local registry
-
-- We can retag a small image, and push it to the registry
-
-.exercise[
-
-- Make sure we have the busybox image, and retag it:
-  ```bash
-  docker pull busybox
-  docker tag busybox 127.0.0.1:5000/busybox
-  ```
-
-- Push it:
-  ```bash
-  docker push 127.0.0.1:5000/busybox
-  ```
-
-]
-
----
-
-class: manual-btp
-
-## Checking what's on our local registry
-
-- The registry API has endpoints to query what's there
-
-.exercise[
-
-- Ensure that our busybox image is now in the local registry:
-  ```bash
-  curl http://127.0.0.1:5000/v2/_catalog
-  ```
-
-]
-
-The curl command should now output:
-```json
-{"repositories":["busybox"]}
-```
-
----
-
-class: manual-btp
-
-## Build, tag, and push our application container images
-
-- Compose has named our images `dockercoins_XXX` for each service
-
-- We need to retag them (to `127.0.0.1:5000/XXX:v1`) and push them
-
-.exercise[
-
-- Set `REGISTRY` and `TAG` environment variables to use our local registry
-- And run this little for loop:
-  ```bash
-    cd ~/orchestration-workshop/dockercoins
-    REGISTRY=127.0.0.1:5000 TAG=v1
-    for SERVICE in hasher rng webui worker; do
-      docker tag dockercoins_$SERVICE $REGISTRY/$SERVICE:$TAG
-      docker push $REGISTRY/$SERVICE
-    done
-  ```
-
-]
-
----
-
-class: manual-btp
-
-# Overlay networks
-
-- SwarmKit integrates with overlay networks
-
-- Networks are created with `docker network create`
-
-- Make sure to specify that you want an *overlay* network
-  <br/>(otherwise you will get a local *bridge* network by default)
-
-.exercise[
-
-- Create an overlay network for our application:
-  ```bash
-  docker network create --driver overlay dockercoins
-  ```
-
-]
-
----
-
-class: manual-btp
-
-## Viewing existing networks
-
-- Let's confirm that our network was created
-
-.exercise[
-
-- List existing networks:
-  ```bash
-  docker network ls
-  ```
-
-]
-
----
-
-class: manual-btp
-
-## Can you spot the differences?
-
-The networks `dockercoins` and `ingress` are different from the other ones.
-
-Can you see how?
-
---
-
-class: manual-btp
-
-- They are using a different kind of ID, reflecting the fact that they
-  are SwarmKit objects instead of "classic" Docker Engine objects.
-
-- Their *scope* is `swarm` instead of `local`.
-
-- They are using the overlay driver.
-
----
-
-class: manual-btp, extra-details
-
-## Caveats
-
-.warning[In Docker 1.12, you cannot join an overlay network with `docker run --net ...`.]
-
-Starting with version 1.13, you can, if the network was created with the `--attachable` flag.
-
-*Why is that?*
-
-Placing a container on a network requires allocating an IP address for this container.
-
-The allocation must be done by a manager node (worker nodes cannot update Raft data).
-
-As a result, `docker run --net ...` requires collaboration with manager nodes.
-
-It alters the code path for `docker run`, so it is allowed only under strict circumstances.
-
----
-
-class: manual-btp
-
-## Run the application
-
-- First, create the `redis` service; that one is using a Docker Hub image
-
-.exercise[
-
-- Create the `redis` service:
-  ```bash
-  docker service create --network dockercoins --name redis redis
-  ```
-
-]
-
----
-
-class: manual-btp
-
-## Run the other services
-
-- Then, start the other services one by one
-
-- We will use the images pushed previously
-
-.exercise[
-
-- Start the other services:
-  ```bash
-  REGISTRY=127.0.0.1:5000
-  TAG=v1
-  for SERVICE in hasher rng webui worker; do
-    docker service create --network dockercoins --detach=true \
-           --name $SERVICE $REGISTRY/$SERVICE:$TAG
-  done
-  ```
-
-]
-
-???
-
-## Wait for our application to be up
-
-- We will see later a way to watch progress for all the tasks of the cluster
-
-- But for now, a scrappy Shell loop will do the trick
-
-.exercise[
-
-- Repeatedly display the status of all our services:
-  ```bash
-  watch "docker service ls -q | xargs -n1 docker service ps"
-  ```
-
-- Stop it once everything is running
-
-]
-
----
-
-class: manual-btp
-
-## Expose our application web UI
-
-- We need to connect to the `webui` service, but it is not publishing any port
-
-- Let's reconfigure it to publish a port
-
-.exercise[
-
-- Update `webui` so that we can connect to it from outside:
-  ```bash
-  docker service update webui --publish-add 8000:80 --detach=false
-  ```
-
-]
-
-Note: to "de-publish" a port, you would have to specify the container port.
-</br>(i.e. in that case, `--publish-rm 80`)
-
----
-
-class: manual-btp
-
-## What happens when we modify a service?
-
-- Let's find out what happened to our `webui` service
-
-.exercise[
-
-- Look at the tasks and containers associated to `webui`:
-  ```bash
-  docker service ps webui
-  ```
-]
-
---
-
-class: manual-btp
-
-The first version of the service (the one that was not exposed) has been shutdown.
-
-It has been replaced by the new version, with port 80 accessible from outside.
-
-(This will be discussed with more details in the section about stateful services.)
-
----
-
-class: manual-btp
-
-## Connect to the web UI
-
-- The web UI is now available on port 8000, *on all the nodes of the cluster*
-
-.exercise[
-
-- If you're using Play-With-Docker, just click on the `(8000)` badge
-
-- Otherwise, point your browser to any node, on port 8000
-
-]
-
----
-
-## Scaling the application
-
-- We can change scaling parameters with `docker update` as well
-
-- We will do the equivalent of `docker-compose scale`
-
-.exercise[
-
-- Bring up more workers:
-  ```bash
-  docker service update worker --replicas 10 --detach=false
-  ```
-
-- Check the result in the web UI
-
-]
-
-You should see the performance peaking at 10 hashes/s (like before).
-
----
-
-class: manual-btp
-
-# Global scheduling
-
-- We want to utilize as best as we can the entropy generators
-  on our nodes
-
-- We want to run exactly one `rng` instance per node
-
-- SwarmKit has a special scheduling mode for that, let's use it
-
-- We cannot enable/disable global scheduling on an existing service
-
-- We have to destroy and re-create the `rng` service
-
----
-
-class: manual-btp
-
-## Scaling the `rng` service
-
-.exercise[
-
-- Remove the existing `rng` service:
-  ```bash
-  docker service rm rng
-  ```
-
-- Re-create the `rng` service with *global scheduling*:
-  ```bash
-    docker service create --name rng --network dockercoins --mode global \
-      --detach=false $REGISTRY/rng:$TAG
-  ```
-
-- Look at the result in the web UI
-
-]
-
----
-
-class: extra-details, manual-btp
-
-## Why do we have to re-create the service to enable global scheduling?
-
-- Enabling it dynamically would make rolling updates semantics very complex
-
-- This might change in the future (after all, it was possible in 1.12 RC!)
-
-- As of Docker Engine 17.05, other parameters requiring to `rm`/`create` the service are:
-
-  - service name
-
-  - hostname
-
-  - network
-
----
-
-class: swarm-ready
-
-## How did we make our app "Swarm-ready"?
-
-This app was written in June 2015. (One year before Swarm mode was released.)
-
-What did we change to make it compatible with Swarm mode?
-
---
-
-.exercise[
-
-- Go to the app directory:
-  ```bash
-  cd ~/orchestration-workshop/dockercoins
-  ```
-
-- See modifications in the code:
-  ```bash
-  git log -p --since "4-JUL-2015" -- . ':!*.yml*' ':!*.html'
-  ```
-
-]
-
----
-
-class: swarm-ready
-
-## What did we change in our app since its inception?
-
-- Compose files
-
-- HTML file (it contains an embedded contextual tweet)
-
-- Dockerfiles (to switch to smaller images)
-
-- That's it!
-
---
-
-class: swarm-ready
-
-*We didn't change a single line of code in this app since it was written.*
-
---
-
-class: swarm-ready
-
-*The images that were [built in June 2015](
-https://hub.docker.com/r/jpetazzo/dockercoins_worker/tags/)
-(when the app was written) can still run today ...
-<br/>... in Swarm mode (distributed across a cluster, with load balancing) ...
-<br/>... without any modification.*
-
----
-
-class: swarm-ready
-
-## How did we design our app in the first place?
-
-- [Twelve-Factor App](https://12factor.net/) principles
-
-- Service discovery using DNS names
-
-  - Initially implemented as "links"
-
-  - Then "ambassadors"
-
-  - And now "services"
-
-- Existing apps might require more changes!
-
----
-
-class: manual-btp
-
-# Integration with Compose
-
-- The previous section showed us how to streamline image build and push
-
-- We will now see how to streamline service creation
-
-  (i.e. get rid of the `for SERVICE in ...; do docker service create ...` part)
-
----
-
-## Compose file version 3
-
-(New in Docker Engine 1.13)
-
-- Almost identical to version 2
-
-- Can be directly used by a Swarm cluster through `docker stack ...` commands
-
-- Introduces a `deploy` section to pass Swarm-specific parameters
-
-- Resource limits are moved to this `deploy` section
-
-- See [here](https://github.com/aanand/docker.github.io/blob/8524552f99e5b58452fcb1403e1c273385988b71/compose/compose-file.md#upgrading) for the complete list of changes
-
-- Supersedes *Distributed Application Bundles*
-
-  (JSON payload describing an application; could be generated from a Compose file)
-
----
-
-class: manual-btp
-
-## Removing everything
-
-- Before deploying using "stacks," let's get a clean slate
-
-.exercise[
-
-- Remove *all* the services:
-  ```bash
-  docker service ls -q | xargs docker service rm
-  ```
-
-]
-
----
-
-## Our first stack
-
-We need a registry to move images around.
-
-Without a stack file, it would be deployed with the following command:
-
-```bash
-docker service create --publish 5000:5000 registry:2
-```
-
-Now, we are going to deploy it with the following stack file:
-
-```yaml
-version: "3"
-
-services:
-  registry:
-    image: registry:2
-    ports:
-      - "5000:5000"
-```
-
----
-
-## Checking our stack files
-
-- All the stack files that we will use are in the `stacks` directory
-
-.exercise[
-
-- Go to the `stacks` directory:
-  ```bash
-  cd ~/orchestration-workshop/stacks
-  ```
-
-- Check `registry.yml`:
-  ```bash
-  cat registry.yml
-  ```
-
-]
-
----
-
-## Deploying our first stack
-
-- All stack manipulation commands start with `docker stack`
-
-- Under the hood, they map to `docker service` commands
-
-- Stacks have a *name* (which also serves as a namespace)
-
-- Stacks are specified with the aforementioned Compose file format version 3
-
-.exercise[
-
-- Deploy our local registry:
-  ```bash
-  docker stack deploy registry --compose-file registry.yml
-  ```
-
-]
-
----
-
-## Inspecting stacks
-
-- `docker stack ps` shows the detailed state of all services of a stack
-
-.exercise[
-
-- Check that our registry is running correctly:
-  ```bash
-  docker stack ps registry
-  ```
-
-- Confirm that we get the same output with the following command:
-  ```bash
-  docker service ps registry_registry
-  ```
-
-]
-
----
-
-class: manual-btp
-
-## Specifics of stack deployment
-
-Our registry is not *exactly* identical to the one deployed with `docker service create`!
-
-- Each stack gets its own overlay network
-
-- Services of the task are connected to this network
-  <br/>(unless specified differently in the Compose file)
-
-- Services get network aliases matching their name in the Compose file
-  <br/>(just like when Compose brings up an app specified in a v2 file)
-
-- Services are explicitly named `<stack_name>_<service_name>`
-
-- Services and tasks also get an internal label indicating which stack they belong to
-
----
-
-## Testing our local registry
-
-- Connecting to port 5000 *on any node of the cluster* routes us to the registry
-
-- Therefore, we can use `localhost:5000` or `127.0.0.1:5000` as our registry
-
-.exercise[
-
-- Issue the following API request to the registry:
-  ```bash
-  curl 127.0.0.1:5000/v2/_catalog
-  ```
-
-]
-
-It should return:
-
-```json
-{"repositories":[]}
-```
-
-If that doesn't work, retry a few times; perhaps the container is still starting.
-
----
-
-## Pushing an image to our local registry
-
-- We can retag a small image, and push it to the registry
-
-.exercise[
-
-- Make sure we have the busybox image, and retag it:
-  ```bash
-  docker pull busybox
-  docker tag busybox 127.0.0.1:5000/busybox
-  ```
-
-- Push it:
-  ```bash
-  docker push 127.0.0.1:5000/busybox
-  ```
-
-]
-
----
-
-## Checking what's on our local registry
-
-- The registry API has endpoints to query what's there
-
-.exercise[
-
-- Ensure that our busybox image is now in the local registry:
-  ```bash
-  curl http://127.0.0.1:5000/v2/_catalog
-  ```
-
-]
-
-The curl command should now output:
-```json
-"repositories":["busybox"]}
-```
-
----
-
-## Building and pushing stack services
-
-- When using Compose file version 2 and above, you can specify *both* `build` and `image`
-
-- When both keys are present:
-
-  - Compose does "business as usual" (uses `build`)
-
-  - but the resulting image is named as indicated by the `image` key
-    <br/>
-    (instead of `<projectname>_<servicename>:latest`)
-
-  - it can be pushed to a registry with `docker-compose push`
-
-- Example:
-
-  ```yaml
-    webfront:
-      build: www
-      image: myregistry.company.net:5000/webfront
-  ```
-
----
-
-## Using Compose to build and push images
-
-.exercise[
-
-- Try it:
-  ```bash
-  docker-compose -f dockercoins.yml build
-  docker-compose -f dockercoins.yml push
-  ```
-
-]
-
-Let's have a look at the `dockercoins.yml` file while this is building and pushing.
-
----
-
-```yaml
-version: "3"
-
-services:
-  rng:
-    build: dockercoins/rng
-    image: ${REGISTRY-127.0.0.1:5000}/rng:${TAG-latest}
-    deploy:
-      mode: global
-  ...
-  redis:
-    image: redis
-  ...
-  worker:
-    build: dockercoins/worker
-    image: ${REGISTRY-127.0.0.1:5000}/worker:${TAG-latest}
-    ...
-    deploy:
-      replicas: 10
-```
-
----
-
-## Deploying the application
-
-- Now that the images are on the registry, we can deploy our application stack
-
-.exercise[
-
-- Create the application stack:
-  ```bash
-  docker stack deploy dockercoins --compose-file dockercoins.yml
-  ```
-
-]
-
-We can now connect to any of our nodes on port 8000, and we will see the familiar hashing speed graph.
-
----
-
-## Maintaining multiple environments
-
-There are many ways to handle variations between environments.
-
-- Compose loads `docker-compose.yml` and (if it exists) `docker-compose.override.yml`
-
-- Compose can load alternate file(s) by setting the `-f` flag or the `COMPOSE_FILE` environment variable
-
-- Compose files can *extend* other Compose files, selectively including services:
-
-  ```yaml
-    web:
-      extends:
-        file: common-services.yml
-        service: webapp
-  ```
-
-See [this documentation page](https://docs.docker.com/compose/extends/) for more details about these techniques.
-
-
----
-
-class: extra-details
-
-## Good to know ...
-
-- Compose file version 3 adds the `deploy` section
-
-- Compose file version 3.1 adds support for secrets
-
-- You can re-run `docker stack deploy` to update a stack
-
-- ... But unsupported features will be wiped each time you redeploy (!)
-
-  (This will likely be fixed/improved soon)
-
-- `extends` doesn't work with `docker stack deploy`
-
-  (But you can use `docker-compose config` to "flatten" your configuration)
-
----
-
-## Summary
-
-- We've seen how to set up a Swarm
-
-- We've used it to host our own registry
-
-- We've built our app container images
-
-- We've used the registry to host those images
-
-- We've deployed and scaled our application
-
-- We've seen how to use Compose to streamline deployments
-
-- Awesome job, team!
-
----
-
-class: title, in-person
-
-Operating the Swarm
-
----
-
-name: part-2
-
-class: title, self-paced
-
-Part 2
-
----
-
-class: self-paced
-
-## Before we start ...
-
-The following exercises assume that you have a 5-nodes Swarm cluster.
-
-If you come here from a previous tutorial and still have your cluster: great!
-
-Otherwise: check [part 1](#part-1) to learn how to setup your own cluster.
-
-We pick up exactly where we left you, so we assume that you have:
-
-- a five nodes Swarm cluster,
-
-- a self-hosted registry,
-
-- DockerCoins up and running.
-
-The next slide has a cheat sheet if you need to set that up in a pinch.
-
----
-
-class: self-paced
-
-## Catching up
-
-Assuming you have 5 nodes provided by
-[Play-With-Docker](http://www.play-with-docker/), do this from `node1`:
-
-```bash
-docker swarm init --advertise-addr eth0
-TOKEN=$(docker swarm join-token -q manager)
-for N in $(seq 2 5); do
-  DOCKER_HOST=tcp://node$N:2375 docker swarm join --token $TOKEN node1:2377
-done
-git clone git://github.com/jpetazzo/orchestration-workshop
-cd orchestration-workshop/stacks
-docker stack deploy --compose-file registry.yml registry
-docker-compose -f dockercoins.yml build
-docker-compose -f dockercoins.yml push
-docker stack deploy --compose-file dockercoins.yml dockercoins
-```
-
-You should now be able to connect to port 8000 and see the DockerCoins web UI.
-
----
-
-class: netshoot, extra-details
-
-## Troubleshooting overlay networks
-
-<!--
-
-## Finding the real cause of the bottleneck
-
-- We want to debug our app as we scale `worker` up and down
-
--->
-
-- We want to run tools like `ab` or `httping` on the internal network
-
---
-
-class: netshoot, extra-details
-
-- Ah, if only we had created our overlay network with the `--attachable` flag ...
-
---
-
-class: netshoot, extra-details
-
-- Oh well, let's use this as an excuse to introduce New Ways To Do Things
-
----
-
-class: netshoot
-
-# Breaking into an overlay network
-
-- We will create a dummy placeholder service on our network
-
-- Then we will use `docker exec` to run more processes in this container
-
-.exercise[
-
-- Start a "do nothing" container using our favorite Swiss-Army distro:
-  ```bash
-    docker service create --network dockercoins_default --name debug \
-           --constraint node.hostname==$HOSTNAME alpine sleep 1000000000
-  ```
-
-]
-
-The `constraint` makes sure that the container will be created on the local node.
-
----
-
-class: netshoot
-
-## Entering the debug container
-
-- Once our container is started (which should be really fast because the alpine image is small), we can enter it (from any node)
-
-.exercise[
-
-- Locate the container:
-  ```bash
-  docker ps
-  ```
-
-- Enter it:
-  ```bash
-  docker exec -ti <containerID> sh
-  ```
-
-]
-
----
-
-class: netshoot
-
-## Labels
-
-- We can also be fancy and find the ID of the container automatically
-
-- SwarmKit places labels on containers
-
-.exercise[
-
-- Get the ID of the container:
-  ```bash
-  CID=$(docker ps -q --filter label=com.docker.swarm.service.name=debug)
-  ```
-
-- And enter the container:
-  ```bash
-  docker exec -ti $CID sh
-  ```
-
-]
-
----
-
-class: netshoot
-
-## Installing our debugging tools
-
-- Ideally, you would author your own image, with all your favorite tools, and use it instead of the base `alpine` image
-
-- But we can also dynamically install whatever we need
-
-.exercise[
-
-- Install a few tools:
-  ```bash
-  apk add --update curl apache2-utils drill
-  ```
-
-]
-
----
-
-class: netshoot
-
-## Investigating the `rng` service
-
-- First, let's check what `rng` resolves to
-
-.exercise[
-
-- Use drill or nslookup to resolve `rng`:
-  ```bash
-  drill rng
-  ```
-
-]
-
-This give us one IP address. It is not the IP address of a container.
-It is a virtual IP address (VIP) for the `rng` service.
-
----
-
-class: netshoot
-
-## Investigating the VIP
-
-.exercise[
-
-- Try to ping the VIP:
-  ```bash
-  ping rng
-  ```
-
-]
-
-It *should* ping. (But this might change in the future.)
-
-With Engine 1.12: VIPs respond to ping if a
-backend is available on the same machine.
-
-With Engine 1.13: VIPs respond to ping if a
-backend is available anywhere.
-
-(Again: this might change in the future.)
-
----
-
-class: netshoot
-
-## What if I don't like VIPs?
-
-- Services can be published using two modes: VIP and DNSRR.
-
-- With VIP, you get a virtual IP for the service, and a load balancer
-  based on IPVS
-
-  (By the way, IPVS is totally awesome and if you want to learn more about it in the context of containers,
-  I highly recommend [this talk](https://www.youtube.com/watch?v=oFsJVV1btDU&index=5&list=PLkA60AVN3hh87OoVra6MHf2L4UR9xwJkv) by [@kobolog](https://twitter.com/kobolog) at DC15EU!)
-
-- With DNSRR, you get the former behavior (from Engine 1.11), where
-  resolving the service yields the IP addresses of all the containers for
-  this service
-
-- You change this with `docker service create --endpoint-mode [VIP|DNSRR]`
-
----
-
-class: netshoot
-
-## Looking up VIP backends
-
-- You can also resolve a special name: `tasks.<name>`
-
-- It will give you the IP addresses of the containers for a given service
-
-.exercise[
-
-- Obtain the IP addresses of the containers for the `rng` service:
-  ```bash
-  drill tasks.rng
-  ```
-
-]
-
-This should list 5 IP addresses.
-
----
-
-class: netshoot, extra-details
-
-## Testing and benchmarking our service
-
-- We will check that the service is up with `rng`, then
-  benchmark it with `ab`
-
-.exercise[
-
-- Make a test request to the service:
-  ```bash
-  curl rng
-  ```
-
-- Open another window, and stop the workers, to test in isolation:
-  ```bash
-  docker service update dockercoins_worker --replicas 0
-  ```
-
-]
-
-Wait until the workers are stopped (check with `docker service ls`)
-before continuing.
-
----
-
-class: netshoot, extra-details
-
-## Benchmarking `rng`
-
-We will send 50 requests, but with various levels of concurrency.
-
-.exercise[
-
-- Send 50 requests, with a single sequential client:
-  ```bash
-  ab -c 1 -n 50 http://rng/10
-  ```
-
-- Send 50 requests, with fifty parallel clients:
-  ```bash
-  ab -c 50 -n 50 http://rng/10
-  ```
-
-]
-
----
-
-class: netshoot, extra-details
-
-## Benchmark results for `rng`
-
-- When serving requests sequentially, they each take 100ms
-
-- In the parallel scenario, the latency increased dramatically:
-
-- What about `hasher`?
-
----
-
-class: netshoot, extra-details
-
-## Benchmarking `hasher`
-
-We will do the same tests for `hasher`.
-
-The command is slightly more complex, since we need to post random data.
-
-First, we need to put the POST payload in a temporary file.
-
-.exercise[
-
-- Install curl in the container, and generate 10 bytes of random data:
-  ```bash
-  curl http://rng/10 >/tmp/random
-  ```
-
-]
-
----
-
-class: netshoot, extra-details
-
-## Benchmarking `hasher`
-
-Once again, we will send 50 requests, with different levels of concurrency.
-
-.exercise[
-
-- Send 50 requests with a sequential client:
-  ```bash
-    ab -c 1 -n 50 -T application/octet-stream -p /tmp/random http://hasher/
-  ```
-
-- Send 50 requests with 50 parallel clients:
-  ```bash
-    ab -c 50 -n 50 -T application/octet-stream -p /tmp/random http://hasher/
-  ```
-
-]
-
----
-
-class: netshoot, extra-details
-
-## Benchmark results for `hasher`
-
-- The sequential benchmarks takes ~5 seconds to complete
-
-- The parallel benchmark takes less than 1 second to complete
-
-- In both cases, each request takes a bit more than 100ms to complete
-
-- Requests are a bit slower in the parallel benchmark
-
-- It looks like `hasher` is better equiped to deal with concurrency than `rng`
-
----
-
-class: netshoot, extra-details, title
-
-Why?
-
----
-
-class: netshoot, extra-details
-
-## Why does everything take (at least) 100ms?
-
-`rng` code:
-
-![RNG code screenshot](delay-rng.png)
-
-`hasher` code:
-
-![HASHER code screenshot](delay-hasher.png)
-
----
-
-class: netshoot, extra-details, title
-
-But ...
-
-WHY?!?
-
----
-
-class: netshoot, extra-details
-
-## Why did we sprinkle this sample app with sleeps?
-
-- Deterministic performance
-  <br/>(regardless of instance speed, CPUs, I/O...)
-
-- Actual code sleeps all the time anyway
-
-- When your code makes a remote API call:
-
-  - it sends a request;
-
-  - it sleeps until it gets the response;
-
-  - it processes the response.
-
----
-
-class: netshoot, extra-details, in-person
-
-## Why do `rng` and `hasher` behave differently?
-
-![Equations on a blackboard](equations.png)
-
-(Synchronous vs. asynchronous event processing)
-
----
-
-class: netshoot, extra-details
-
-## Global scheduling → global debugging
-
-- Traditional approach:
-
-  - log into a node
-  - install our Swiss Army Knife (if necessary)
-  - troubleshoot things
-
-- Proposed alternative:
-
-  - put our Swiss Army Knife in a container (e.g. [nicolaka/netshoot](https://hub.docker.com/r/nicolaka/netshoot/))
-  - run tests from multiple locations at the same time
-
-(This becomes very practical with the `docker service log` command, available since 17.05.)
-
----
-
-class: nbt, extra-details
-
-## Measuring network conditions on the whole cluster
-
-- Since we have built-in, cluster-wide discovery, it's relatively straightforward
-  to monitor the whole cluster automatically
-
-- [Alexandros Mavrogiannis](https://github.com/alexmavr) wrote
-  [Swarm NBT](https://github.com/alexmavr/swarm-nbt), a tool doing exactly that!
-
-.exercise[
-
-- Start Swarm NBT:
-  ```bash
-    docker run --rm -v inventory:/inventory \
-           -v /var/run/docker.sock:/var/run/docker.sock \
-           alexmavr/swarm-nbt start
-  ```
-
-]
-
-Note: in this mode, Swarm NBT connects to the Docker API socket,
-and issues additional API requests to start all the components it needs.
-
----
-
-class: nbt, extra-details
-
-## Viewing network conditions with Prometheus
-
-- Swarm NBT relies on Prometheus to scrape and store data
-
-- We can directly consume the Prometheus endpoint to view telemetry data
-
-.exercise[
-
-- Point your browser to any Swarm node, on port 9090
-
-  (If you're using Play-With-Docker, click on the (9090) badge)
-
-- In the drop-down, select `icmp_rtt_gauge_seconds`
-
-- Click on "Graph"
-
-]
-
-You are now seeing ICMP latency across your cluster.
-
----
-
-class: nbt, in-person, extra-details
-
-## Viewing network conditions with Grafana
-
-- If you are using a "real" cluster (not Play-With-Docker) you can use Grafana
-
-.exercise[
-
-- Start Grafana with `docker service create -p 3000:3000 grafana`
-- Point your browser to Grafana, on port 3000 on any Swarm node
-- Login with username `admin` and password `admin`
-- Click on the top-left menu and browse to Data Sources
-- Create a prometheus datasource with any name
-- Point it to http://any-node-IP:9090
-- Set access to "direct" and leave credentials blank
-- Click on the top-left menu, highlight "Dashboards" and select the "Import" option
-- Copy-paste [this JSON payload](
-  https://raw.githubusercontent.com/alexmavr/swarm-nbt/master/grafana.json),
-  then use the Prometheus Data Source defined before
-- Poke around the dashboard that magically appeared!
-
-]
-
----
-
-class: ipsec
-
-# Securing overlay networks
-
-- By default, overlay networks are using plain VXLAN encapsulation
-
-  (~Ethernet over UDP, using SwarmKit's control plane for ARP resolution)
-
-- Encryption can be enabled on a per-network basis
-
-  (It will use IPSEC encryption provided by the kernel, leveraging hardware acceleration)
-
-- This is only for the `overlay` driver
-
-  (Other drivers/plugins will use different mechanisms)
-
----
-
-class: ipsec
-
-## Creating two networks: encrypted and not
-
-- Let's create two networks for testing purposes
-
-.exercise[
-
-- Create an "insecure" network:
-  ```bash
-  docker network create insecure --driver overlay --attachable
-  ```
-
-- Create a "secure" network:
-  ```bash
-  docker network create secure --opt encrypted --driver overlay --attachable
-  ```
-
-]
-
-.warning[Make sure that you don't typo that option; errors are silently ignored!]
-
----
-
-class: ipsec
-
-## Deploying a web server sitting on both networks
-
-- Let's use good old NGINX
-
-- We will attach it to both networks
-
-- We will use a placement constraint to make sure that it is on a different node
-
-.exercise[
-
-- Create a web server running somewhere else:
-  ```bash
-    docker service create --name web \
-           --network secure --network insecure \
-           --constraint node.hostname!=node1 \
-           nginx
-  ```
-
-]
-
----
-
-class: ipsec
-
-## Sniff HTTP traffic
-
-- We will use `ngrep`, which allows to grep for network traffic
-
-- We will run it in a container, using host networking to access the host's interfaces
-
-.exercise[
-
-- Sniff network traffic and display all packets containing "HTTP":
-  ```bash
-  docker run --net host nicolaka/netshoot ngrep -tpd eth0 HTTP
-  ```
-
-]
-
---
-
-class: ipsec
-
-Seeing tons of HTTP request? Shutdown your DockerCoins workers:
-```bash
-docker service update dockercoins_worker --replicas=0
-```
-
----
-
-class: ipsec
-
-## Check that we are, indeed, sniffing traffic
-
-- Let's see if we can intercept our traffic with Google!
-
-.exercise[
-
-- Open a new terminal
-
-- Issue an HTTP request to Google (or anything you like):
-  ```bash
-  curl google.com
-  ```
-
-]
-
-The ngrep container will display one `#` per packet traversing the network interface.
-
-When you do the `curl`, you should see the HTTP request in clear text in the output.
-
----
-
-class: ipsec, extra-details
-
-## If you are using Play-With-Docker, Vagrant, etc.
-
-- You will probably have *two* network interfaces
-
-- One interface will be used for outbound traffic (to Google)
-
-- The other one will be used for internode traffic
-
-- You might have to adapt/relaunch the `ngrep` command to specify the right one!
-
----
-
-class: ipsec
-
-## Try to sniff traffic across overlay networks
-
-- We will run `curl web` through both secure and insecure networks
-
-.exercise[
-
-- Access the web server through the insecure network:
-  ```bash
-  docker run --rm --net insecure nicolaka/netshoot curl web
-  ```
-
-- Now do the same through the secure network:
-  ```bash
-  docker run --rm --net secure nicolaka/netshoot curl web
-  ```
-
-]
-
-When you run the first command, you will see HTTP fragments.
-<br/>
-However, when you run the second one, only `#` will show up.
-
----
-
-# Updating services
-
-- We want to make changes to the web UI
-
-- The process is as follows:
-
-  - edit code
-
-  - build new image
-
-  - ship new image
-
-  - run new image
-
----
-
-class: extra-details
-
-## But first...
-
-- Restart the workers
-
-.exercise[
-
-- Just scale back to 10 replicas:
-  ```bash
-  docker service update dockercoins_worker --replicas 10
-  ```
-
-- Check that they're running:
-  ```bash
-  docker service ps dockercoins_worker
-  ```
-
-]
-
----
-
-## Updating a single service the hard way
-
-- To update a single service, we could do the following:
-  ```bash
-  REGISTRY=localhost:5000 TAG=v0.2
-  IMAGE=$REGISTRY/dockercoins_webui:$TAG
-  docker build -t $IMAGE webui/
-  docker push $IMAGE
-  docker service update dockercoins_webui --image $IMAGE
-  ```
-
-- Make sure to tag properly your images: update the `TAG` at each iteration
-
-  (When you check which images are running, you want these tags to be uniquely identifiable)
-
----
-
-## Updating services the easy way
-
-- With the Compose inbtegration, all we have to do is:
-  ```bash
-  export TAG=v0.2
-  docker-compose -f composefile.yml build
-  docker-compose -f composefile.yml push
-  docker stack deploy -c composefile.yml nameofstack
-  ```
-
---
-
-- That's exactly what we used earlier to deploy the app
-
-- We don't need to learn new commands!
-
----
-
-## Updating the web UI
-
-- Let's make the numbers on the Y axis bigger!
-
-.exercise[
-
-- Edit the file `webui/files/index.html`
-
-- Locate the `font-size` CSS attribute and increase it (at least double it)
-
-- Save and exit
-
-- Build, ship, and run:
-  ```bash
-  export TAG=v0.2
-  docker-compose -f dockercoins.yml build
-  docker-compose -f dockercoins.yml push
-  docker stack deploy -c dockercoins.yml dockercoins
-  ```
-
-]
-
----
-
-## Viewing our changes
-
-- Wait at least 10 seconds (for the new version to be deployed)
-
-- Then reload the web UI
-
-- Or just mash "reload" frantically
-
-- ... Eventually the legend on the left will be bigger!
-
----
-
-## Making changes
-
-.exercise[
-
-- Edit `~/orchestration-workshop/dockercoins/worker/worker.py`
-
-- Locate the line that has a `sleep` instruction
-
-- Increase the `sleep` from `0.1` to `1.0`
-
-- Save your changes and exit
-
-]
-
----
-
-## Rolling updates
-
-- Let's change a scaled service: `worker`
-
-.exercise[
-
-- Edit `worker/worker.py`
-
-- Locate the `sleep` instruction and change the delay
-
-- Build, ship, and run our changes:
-  ```bash
-  export TAG=v0.3
-  docker-compose -f dockercoins.yml build
-  docker-compose -f dockercoins.yml push
-  docker stack deploy -c dockercoins.yml dockercoins
-  ```
-
-]
-
----
-
-## Viewing our update as it rolls out
-
-.exercise[
-
-- Check the status of the `dockercoins_worker` service:
-  ```bash
-  watch docker service ps dockercoins_worker
-  ```
-
-- Hide the tasks that are shutdown:
-  ```bash
-  watch -n1 "docker service ps dockercoins_worker | grep -v Shutdown.*Shutdown"
-  ```
-
-]
-
-If you had stopped the workers earlier, this will automatically restart them.
-
-By default, SwarmKit does a rolling upgrade, one instance at a time.
-
-We should therefore see the workers being updated one my one.
-
----
-
-## Changing the upgrade policy
-
-- We can set upgrade parallelism (how many instances to update at the same time)
-
-- And upgrade delay (how long to wait between two batches of instances)
-
-.exercise[
-
-- Change the parallelism to 2 and the delay to 5 seconds:
-  ```bash
-    docker service update dockercoins_worker \
-      --update-parallelism 2 --update-delay 5s
-  ```
-
-]
-
-The current upgrade will continue at a faster pace.
-
----
-
-## Changing the policy in the Compose file
-
-- The policy can also be updated in the Compose file
-
-- This is done by adding an `update_config` key under the `deploy` key:
-
-  ```yaml
-    deploy:
-      replicas: 10
-      update_config:
-        parallelism: 2
-        delay: 10s
-  ```
-
----
-
-## Rolling back
-
-- At any time (e.g. before the upgrade is complete), we can rollback:
-
-  - by editing the Compose file and redeploying;
-
-  - or with the special `--rollback` flag
-
-.exercise[
-
-- Try to rollback the service:
-  ```bash
-  docker service update dockercoins_worker --rollback
-  ```
-
-]
-
-What happens with the web UI graph?
-
----
-
-## The fine print with rollback
-
-- Rollback reverts to the previous service definition
-
-- If we visualize successive updates as a stack:
-
-  - it doesn't "pop" the latest update
-
-  - it "pushes" a copy of the previous update on top
-
-  - ergo, rolling back twice does nothing
-
-- "Service definition" includes rollout cadence
-
-- Each `docker service update` command = a new service definition
-
----
-
-class: extra-details
-
-## Timeline of an upgrade
-
-- SwarmKit will upgrade N instances at a time
-  <br/>(following the `update-parallelism` parameter)
-
-- New tasks are created, and their desired state is set to `Ready`
-  <br/>.small[(this pulls the image if necessary, ensures resource availability, creates the container ... without starting it)]
-
-- If the new tasks fail to get to `Ready` state, go back to the previous step
-  <br/>.small[(SwarmKit will try again and again, until the situation is addressed or desired state is updated)]
-
-- When the new tasks are `Ready`, it sets the old tasks desired state to `Shutdown`
-
-- When the old tasks are `Shutdown`, it starts the new tasks
-
-- Then it waits for the `update-delay`, and continues with the next batch of instances
-
----
-
-name: healthchecks
-
-class: healthchecks
-
-# Health checks
-
-(New in Docker Engine 1.12)
-
-- Commands that are executed on regular intervals in a container
-
-- Must return 0 or 1 to indicate "all is good" or "something's wrong"
-
-- Must execute quickly (timeouts = failures)
-
-- Example:
-  ```bash
-  curl -f http://localhost/_ping || false
-  ```
-  - the `-f` flag ensures that `curl` returns non-zero for 404 and similar errors
-  - `|| false` ensures that any non-zero exit status gets mapped to 1
-  - `curl` must be installed in the container that is being checked
-
----
-
-class: healthchecks
-
-## Defining health checks
-
-- In a Dockerfile, with the [HEALTHCHECK](https://docs.docker.com/engine/reference/builder/#healthcheck) instruction
-  ```
-  HEALTHCHECK --interval=1s --timeout=3s CMD curl -f http://localhost/ || false
-  ```
-
-- From the command line, when running containers or services
-  ```
-  docker run --health-cmd "curl -f http://localhost/ || false" ...
-  docker service create --health-cmd "curl -f http://localhost/ || false" ...
-  ```
-
-- In Compose files, with a per-service [healthcheck](https://docs.docker.com/compose/compose-file/#healthcheck) section
-  ```yaml
-    www:
-      image: hellowebapp
-      healthcheck:
-        test: "curl -f https://localhost/ || false"
-        timeout: 3s
-  ```
-
----
-
-class: healthcheck
-
-## Using health checks
-
-- With `docker run`, health checks are purely informative
-
-  - `docker ps` shows health status
-
-  - `docker inspect` has extra details (including health check command output)
-
-- With `docker service`:
-
-  - unhealthy tasks are terminated (i.e. the service is restarted)
-
-  - failed deployments can be rolled back automatically
-    <br/>(by setting *at least* the flag `--update-failure action rollback`)
-
----
-
-class: healthcheck
-
-## Automated rollbacks
-
-Here is a comprehensive example using the CLI:
-
-```bash
-docker service update \
-  --update-delay 5s \
-  --update-failure-action rollback \
-  --update-max-failure-ratio .25 \
-  --update-monitor 5s \
-  --update-parallelism 1 \
-  --rollback-delay 5s \
-  --rollback-failure-action pause \
-  --rollback-max-failure-ratio .5 \
-  --rollback-monitor 5s \
-  --rollback-parallelism 0 \
-  --health-cmd "curl -f http://localhost/ || exit 1" \
-  --health-interval 2s \
-  --health-retries 1 \
-  --image yourimage:newversion \
-  yourservice
-```
-
----
-
-class: healthcheck
-
-## Implementing auto-rollback in practice
-
-We will use the following Compose file (`stacks/dockercoins+healthchecks.yml`):
-
-```yaml
-...
-  hasher:
-    build: dockercoins/hasher
-    image: ${REGISTRY-127.0.0.1:5000}/hasher:${TAG-latest}
-    deploy:
-      replicas: 7
-      update_config:
-        delay: 5s
-        failure_action: rollback
-        max_failure_ratio: .5
-        monitor: 5s
-        parallelism: 1
-...
-```
-
----
-
-class: healthcheck
-
-## Enabling auto-rollback
-
-.exercise[
-
-- Go to the `stacks` directory:
-  ```bash
-  cd ~/orchestration-workshop/
-  ```
-
-- Deploy the updated stack:
-  ```bash
-  docker deploy dockercoins --compose-file dockercoins+healthchecks.yml
-  ```
-
-]
-
-This will also scale the `hasher` service to 7 instances.
-
----
-
-class: healthcheck
-
-## Visualizing a rolling update
-
-First, let's make an "innocent" change and deploy it.
-
-.exercise[
-
-- Update the `sleep` delay in the code:
-  ```bash
-  sed -i "s/sleep 0.1/sleep 0.2/" dockercoins/hasher/hasher.rb
-  ```
-
-- Build, ship, and run the new image:
-  ```bash
-  docker-compose -f dockercoins+healthchecks.yml build
-  docker-compose -f dockercoins+healthchecks.yml push
-  docker service update dockercoins_hasher \
-           --detach=false --image=127.0.0.1:5000/hasher:latest
-  ```
-
-]
-
----
-
-class: healthcheck
-
-## Visualizing an automated rollback
-
-And now, a breaking change that will cause the health check to fail:
-
-.exercise[
-
-- Change the HTTP listening port:
-  ```bash
-  sed -i "s/80/81/" dockercoins/hasher/hasher.rb
-  ```
-
-- Build, ship, and run the new image:
-  ```bash
-  docker-compose -f dockercoins+healthchecks.yml build
-  docker-compose -f dockercoins+healthchecks.yml push
-  docker service update dockercoins_hasher \
-           --detach=false --image=127.0.0.1:5000/hasher:latest
-  ```
-
-]
-
----
-
-class: healthcheck
-
-## Command-line options available for health checks, rollbacks, etc.
-
-Batteries included, but swappable
-
-.small[
-```
---health-cmd string                  Command to run to check health
---health-interval duration           Time between running the check (ms|s|m|h)
---health-retries int                 Consecutive failures needed to report unhealthy
---health-start-period duration       Start period for the container to initialize before counting retries towards unstable (ms|s|m|h)
---health-timeout duration            Maximum time to allow one check to run (ms|s|m|h)
---no-healthcheck                     Disable any container-specified HEALTHCHECK
---restart-condition string           Restart when condition is met ("none"|"on-failure"|"any")
---restart-delay duration             Delay between restart attempts (ns|us|ms|s|m|h)
---restart-max-attempts uint          Maximum number of restarts before giving up
---restart-window duration            Window used to evaluate the restart policy (ns|us|ms|s|m|h)
---rollback                           Rollback to previous specification
---rollback-delay duration            Delay between task rollbacks (ns|us|ms|s|m|h)
---rollback-failure-action string     Action on rollback failure ("pause"|"continue")
---rollback-max-failure-ratio float   Failure rate to tolerate during a rollback
---rollback-monitor duration          Duration after each task rollback to monitor for failure (ns|us|ms|s|m|h)
---rollback-order string              Rollback order ("start-first"|"stop-first")
---rollback-parallelism uint          Maximum number of tasks rolled back simultaneously (0 to roll back all at once)
---update-delay duration              Delay between updates (ns|us|ms|s|m|h)
---update-failure-action string       Action on update failure ("pause"|"continue"|"rollback")
---update-max-failure-ratio float     Failure rate to tolerate during an update
---update-monitor duration            Duration after each task update to monitor for failure (ns|us|ms|s|m|h)
---update-order string                Update order ("start-first"|"stop-first")
---update-parallelism uint            Maximum number of tasks updated simultaneously (0 to update all at once)
-```
-]
-
-Yup ... That's a lot of batteries!
-
----
-
-class: node-info
-
-## Getting task information for a given node
-
-- You can see all the tasks assigned to a node with `docker node ps`
-
-- It shows the *desired state* and *current state* of each task
-
-- `docker node ps` shows info about the current node
-
-- `docker node ps <node_name_or_id>` shows info for another node
-
-- `docker node ps -a` includes stopped and failed tasks
-
----
-
-class: swarmtools
-
-# SwarmKit debugging tools
-
-- The SwarmKit repository comes with debugging tools
-
-- They are *low level* tools; not for general use
-
-- We are going to see two of these tools:
-
-  - `swarmctl`, to communicate directly with the SwarmKit API
-
-  - `swarm-rafttool`, to inspect the content of the Raft log
-
----
-
-class: swarmtools
-
-## Building the SwarmKit tools
-
-- We are going to install a Go compiler, then download SwarmKit source and build it
-
-.exercise[
-- Download, compile, and install SwarmKit with this one-liner:
-  ```bash
-  docker run -v /usr/local/bin:/go/bin golang \
-         go get `-v` github.com/docker/swarmkit/...
-  ```
-
-]
-
-Remove `-v` if you don't like verbose things.
-
-Shameless promo: for more Go and Docker love, check
-[this blog post](http://jpetazzo.github.io/2016/09/09/go-docker/)!
-
-Note: in the unfortunate event of SwarmKit *master* branch being broken,
-the build might fail. In that case, just skip the Swarm tools section.
-
----
-
-class: swarmtools
-
-## Getting cluster-wide task information
-
-- The Docker API doesn't expose this directly (yet)
-
-- But the SwarmKit API does
-
-- We are going to query it with `swarmctl`
-
-- `swarmctl` is an example program showing how to
-  interact with the SwarmKit API
-
----
-
-class: swarmtools
-
-## Using `swarmctl`
-
-- The Docker Engine places the SwarmKit control socket in a special path
-
-- You need root privileges to access it
-
-.exercise[
-
-- If you are using Play-With-Docker, set the following alias:
-  ```bash
-    alias swarmctl='/lib/ld-musl-x86_64.so.1 /usr/local/bin/swarmctl \
-                    --socket /var/run/docker/swarm/control.sock'
-  ```
-
-- Otherwise, set the following alias:
-  ```bash
-    alias swarmctl='sudo swarmctl \
-                    --socket /var/run/docker/swarm/control.sock'
-  ```
-
-]
-
----
-
-class: swarmtools
-
-## `swarmctl` in action
-
-- Let's review a few useful `swarmctl` commands
-
-.exercise[
-
-- List cluster nodes (that's equivalent to `docker node ls`):
-  ```bash
-  swarmctl node ls
-  ```
-
-- View all tasks across all services:
-  ```bash
-  swarmctl task ls
-  ```
-
-]
-
----
-
-class: swarmtools
-
-## `swarmctl` notes
-
-- SwarmKit is vendored into the Docker Engine
-
-- If you want to use `swarmctl`, you need the exact version of
-  SwarmKit that was used in your Docker Engine
-
-- Otherwise, you might get some errors like:
-
-  ```
-  Error: grpc: failed to unmarshal the received message proto: wrong wireType = 0
-  ```
-
-- With Docker 1.12, the control socket was in `/var/lib/docker/swarm/control.sock`
-
----
-
-class: swarmtools
-
-## `swarm-rafttool`
-
-- SwarmKit stores all its important data in a distributed log using the Raft protocol
-
-  (This log is also simply called the "Raft log")
-
-- You can decode that log with `swarm-rafttool`
-
-- This is a great tool to understand how SwarmKit works
-
-- It can also be used in forensics or troubleshooting
-
-  (But consider it as a *very low level* tool!)
-
----
-
-class: swarmtools
-
-## The powers of `swarm-rafttool`
-
-With `swarm-rafttool`, you can:
-
-- view the latest snapshot of the cluster state;
-
-- view the Raft log (i.e. changes to the cluster state);
-
-- view specific objects from the log or snapshot;
-
-- decrypt the Raft data (to analyze it with other tools).
-
-It *cannot* work on live files, so you must stop Docker or make a copy first.
-
----
-
-class: swarmtools
-
-## Using `swarm-rafttool`
-
-- First, let's make a copy of the current Swarm data
-
-.exercise[
-
-- If you are using Play-With-Docker, the Docker data directory is `/graph`:
-  ```bash
-  cp -r /graph/swarm /swarmdata
-  ```
-
-- Otherwise, it is in the default `/var/lib/docker`:
-  ```bash
-  sudo cp -r /var/lib/docker/swarm /swarmdata
-  ```
-
-]
-
----
-
-class: swarmtools
-
-## Dumping the Raft log
-
-- We have to indicate the path holding the Swarm data
-
-  (Otherwise `swarm-rafttool` will try to use the live data, and complain that it's locked!)
-
-.exercise[
-
-- If you are using Play-With-Docker, you must use the musl linker:
-  ```bash
-  /lib/ld-musl-x86_64.so.1 /usr/local/bin/swarm-rafttool -d /swarmdata/ dump-wal
-  ```
-
-- Otherwise, you don't need the musl linker but you need to get root:
-  ```bash
-  sudo swarm-rafttool -d /swarmdata/ dump-wal
-  ```
-
-]
-
-Reminder: this is a very low-level tool, requiring a knowledge of SwarmKit's internals!
-
----
-
-# Secrets management and encryption at rest
-
-(New in Docker Engine 1.13)
-
-- Secrets management = selectively and securely bring secrets to services
-
-- Encryption at rest = protect against storage theft or prying
-
-- Remember:
-
-  - control plane is authenticated through mutual TLS, certs rotated every 90 days
-
-  - control plane is encrypted with AES-GCM, keys rotated every 12 hours
-
-  - data plane is not encrypted by default (for performance reasons),
-    <br/>but we saw earlier how to enable that with a single flag
-
----
-
-class: secrets
-
-## Secret management
-
-- Docker has a "secret safe" (secure key→value store)
-
-- You can create as many secrets as you like
-
-- You can associate secrets to services
-
-- Secrets are exposed as plain text files, but kept in memory only (using `tmpfs`)
-
-- Secrets are immutable (at least in Engine 1.13)
-
-- Secrets have a max size of 500 KB
-
----
-
-class: secrets
-
-## Creating secrets
-
-- Must specify a name for the secret; and the secret itself
-
-.exercise[
-
-- Assign [one of the four most commonly used passwords](https://www.youtube.com/watch?v=0Jx8Eay5fWQ) to a secret called `hackme`:
-  ```bash
-  echo love | docker secret create hackme -
-  ```
-
-]
-
-If the secret is in a file, you can simply pass the path to the file.
-
-(The special path `-` indicates to read from the standard input.)
-
----
-
-class: secrets
-
-## Creating better secrets
-
-- Picking lousy passwords always leads to security breaches
-
-.exercise[
-
-- Let's craft a better password, and assign it to another secret:
-  ```bash
-  base64 /dev/urandom | head -c16 | docker secret create arewesecureyet -
-  ```
-
-]
-
-Note: in the latter case, we don't even know the secret at this point. But Swarm does.
-
----
-
-class: secrets
-
-## Using secrets
-
-- Secrets must be handed explicitly to services
-
-.exercise[
-
-- Create a dummy service with both secrets:
-  ```bash
-    docker service create \
-           --secret hackme --secret arewesecureyet \
-           --name dummyservice --mode global \
-           alpine sleep 1000000000
-  ```
-
-]
-
-We use a global service to make sure that there will be an instance on the local node.
-
----
-
-class: secrets
-
-## Accessing secrets
-
-- Secrets are materialized on `/run/secrets` (which is an in-memory filesystem)
-
-.exercise[
-
-- Find the ID of the container for the dummy service:
-  ```bash
-  CID=$(docker ps -q --filter label=com.docker.swarm.service.name=dummyservice)
-  ```
-
-- Enter the container:
-  ```bash
-  docker exec -ti $CID sh
-  ```
-
-- Check the files in `/run/secrets`
-
-]
-
----
-
-class: secrets
-
-## Rotating secrets
-
-- You can't change a secret
-
-  (Sounds annoying at first; but allows clean rollbacks if a secret update goes wrong)
-
-- You can add a secret to a service with `docker service update --secret-add`
-
-  (This will redeploy the service; it won't add the secret on the fly)
-
-- You can remove a secret with `docker service update --secret-rm`
-
-- Secrets can be mapped to different names by expressing them with a micro-format:
-  ```bash
-  docker service create --secret source=secretname,target=filename
-  ```
-
----
-
-class: secrets
-
-## Changing our insecure password
-
-- We want to replace our `hackme` secret with a better one
-
-.exercise[
-
-- Remove the insecure `hackme` secret:
-  ```bash
-  docker service update dummyservice --secret-rm hackme
-  ```
-
-- Add our better secret instead:
-  ```bash
-  docker service update dummyservice \
-         --secret-add source=arewesecureyet,target=hackme
-  ```
-
-]
-
-Wait for the service to be fully updated with e.g. `watch docker service ps dummyservice`.
-
----
-
-class: secrets
-
-## Checking that our password is now stronger
-
-- We will use the power of `docker exec`!
-
-.exercise[
-
-- Get the ID of the new container:
-  ```bash
-  CID=$(docker ps -q --filter label=com.docker.swarm.service.name=dummyservice)
-  ```
-
-- Check the contents of the secret files:
-  ```bash
-  docker exec $CID grep -r . /run/secrets
-  ```
-
-]
-
----
-
-class: secrets
-
-## Secrets in practice
-
-- Can be (ab)used to hold whole configuration files if needed
-
-- If you intend to rotate secret `foo`, call it `foo.N` instead, and map it to `foo`
-
-  (N can be a serial, a timestamp...)
-
-  ```bash
-  docker service create --secret source=foo.N,target=foo ...
-  ```
-
-- You can update (remove+add) a secret in a single command:
-
-  ```bash
-  docker service update ... --secret-rm foo.M --secret-add source=foo.N,target=foo
-  ```
-
-- For more details and examples, [check the documentation](https://docs.docker.com/engine/swarm/secrets/)
-
----
-
-# Least privilege model
-
-- All the important data is stored in the "Raft log"
-
-- Managers nodes have read/write access to this data
-
-- Workers nodes have no access to this data
-
-- Workers only receive the minimum amount of data that they need:
-
-  - which services to run
-  - network configuration information for these services
-  - credentials for these services
-
-- Compromising a worker node does not give access to the full cluster
-
----
-
-## What can I do if I compromise a worker node?
-
-- I can enter the containers running on that node
-
-- I can access the configuration and credentials used by these containers
-
-- I can inspect the network traffic of these containers
-
-- I cannot inspect or disrupt the network traffic of other containers
-
-  (network information is provided by manager nodes; ARP spoofing is not possible)
-
-- I cannot infer the topology of the cluster and its number of nodes
-
-- I can only learn the IP addresses of the manager nodes
-
----
-
-## Guidelines for workload isolation leveraging least privilege model
-
-- Define security levels
-
-- Define security zones
-
-- Put managers in the highest security zone
-
-- Enforce workloads of a given security level to run in a given zone
-
-- Enforcement can be done with [Authorization Plugins](https://docs.docker.com/engine/extend/plugins_authorization/)
-
----
-
-class: namespaces
-name: namespaces
-
-# Improving isolation with User Namespaces
-
-- *Namespaces* are kernel mechanisms to compartimetalize the system
-
-- There are different kind of namespaces: `pid`, `net`, `mnt`, `ipc`, `uts`, and `user`
-
-- For a primer, see "Anatomy of a Container"
-  ([video](https://www.youtube.com/watch?v=sK5i-N34im8))
-  ([slides](https://www.slideshare.net/jpetazzo/cgroups-namespaces-and-beyond-what-are-containers-made-from-dockercon-europe-2015))
-
-- The *user namespace* allows to map UIDs between the containers and the host
-
-- As a result, `root` in a container can map to a non-privileged user on the host
-
-Note: even without user namespaces, `root` in a container cannot go wild on the host.
-<br/>
-It is mediated by capabilities, cgroups, namespaces, seccomp, LSMs...
-
----
-
-class: namespaces
-
-## User Namespaces in Docker
-
-- Optional feature added in Docker Engine 1.10
-
-- Not enabled by default
-
-- Has to be enabled at Engine startup, and affects all containers
-
-- When enabled, `UID:GID` in containers are mapped to a different range on the host
-
-- Safer than switching to a non-root user (with `-u` or `USER`) in the container
-  <br/>
-  (Since with user namespaces, root escalation maps to a non-privileged user)
-
-- Can be selectively disabled per container by starting them with `--userns=host`
-
----
-
-class: namespaces
-
-## User Namespaces Caveats
-
-When user namespaces are enabled, containers cannot:
-
-- Use the host's network namespace (with `docker run --network=host`)
-
-- Use the host's PID namespace (with `docker run --pid=host`)
-
-- Run in privileged mode (with `docker run --privileged`)
-
-... Unless user namespaces are disabled for the container, with flag `--userns=host`
-
-External volume and graph drivers that don't support user mapping might not work.
-
-All containers are currently mapped to the same UID:GID range.
-
-Some of these limitations might be lifted in the future!
-
----
-
-class: namespaces
-
-## Filesystem ownership details
-
-When enabling user namespaces:
-
-- the UID:GID on disk (in the images and containers) has to match the *mapped* UID:GID
-
-- existing images and containers cannot work (their UID:GID would have to be changed)
-
-For practical reasons, when enabling user namespaces, the Docker Engine places containers and images (and everything else) in a different directory.
-
-As a resut, if you enable user namespaces on an existing installation:
-
--  all containers and images (and e.g. Swarm data) disappear
-
-- *if a node is a member of a Swarm, it is then kicked out of the Swarm*
-
--  everything will re-appear if you disable user namespaces again
-
----
-
-class: namespaces
-
-## Picking a node
-
-- We will select a node where we will enable user namespaces
-
-- This node will have to be re-added to the Swarm
-
-- All containers and services running on this node will be rescheduled
-
-- Let's make sure that we do not pick the node running the registry!
-
-.exercise[
-
-- Check on which node the registry is running:
-  ```bash
-  docker service ps registry
-  ```
-
-]
-
-Pick any other node (noted `nodeX` in the next slides).
-
----
-
-class: namespaces
-
-## Logging into the right Engine
-
-.exercise[
-
-- Log into the right node:
-  ```bash
-  ssh node`X`
-  ```
-
-]
-
----
-
-class: namespaces
-
-## Configuring the Engine
-
-.exercise[
-
-- Create a configuration file for the Engine:
-  ```bash
-  echo '{"userns-remap": "default"}' | sudo tee /etc/docker/daemon.json
-  ```
-
-- Restart the Engine:
-  ```bash
-  kill $(pidof dockerd)
-  ```
-
-]
-
----
-
-class: namespaces 
-
-## Checking that User Namespaces are enabled
-
-.exercise[
-  - Notice the new Docker path:
-  ```bash
-  docker info | grep var/lib
-  ```
-
-  - Notice the new UID:GID permissions:
-  ```bash
-  sudo ls -l /var/lib/docker
-  ```
-
-]
-
-You should see a line like the following:
-```
-drwx------ 11 296608 296608 4096 Aug  3 05:11 296608.296608
-```
-
----
-
-class: namespaces
-
-## Add the node back to the Swarm
-
-.exercise[
-
-- Get our manager token from another node:
-  ```bash
-  ssh node`Y` docker swarm join-token manager
-  ```
-
-- Copy-paste the join command to the node
-
-]
-
----
-
-class: namespaces
-
-## Check the new UID:GID
-
-.exercise[
-
-- Run a background container on the node:
-  ```bash
-  docker run -d --name lockdown alpine sleep 1000000
-  ```
-
-- Look at the processes in this container:
-  ```bash
-  docker top lockdown
-  ps faux
-  ```
-
-]
-
----
-
-class: namespaces
-
-## Comparing on-disk ownership with/without User Namespaces
-
-.exercise[
-
-- Compare the output of the two following commands:
-  ```bash
-  docker run alpine ls -l /
-  docker run --userns=host alpine ls -l /
-  ```
-
-]
-
---
-
-class: namespaces
-
-In the first case, it looks like things belong to `root:root`.
-
-In the second case, we will see the "real" (on-disk) ownership.
-
---
-
-class: namespaces
-
-Remember to get back to `node1` when finished!
-
----
-
-## A reminder about *scope*
-
-- Out of the box, Docker API access is "all or nothing"
-
-- When someone has access to the Docker API, they can access *everything*
-
-- If your developers are using the Docker API to deploy on the dev cluster ...
-
-  ... and the dev cluster is the same as the prod cluster ...
-
-  ... it means that your devs have access to your production data, passwords, etc.
-
-- This can easily be avoided
-
----
-
-## Fine-grained API access control
-
-A few solutions, by increasing order of flexibility:
-
-- Use separate clusters for different security perimeters
-
-  (And different credentials for each cluster)
-
---
-
-- Add an extra layer of abstraction (sudo scripts, hooks, or full-blown PAAS)
-
---
-
-- Enable [authorization plugins]
-
-  - each API request is vetted by your plugin(s)
-
-  - by default, the *subject name* in the client TLS certificate is used as user name
-
-  - example: [user and permission management] in [UCP]
-
-[authorization plugins]: https://docs.docker.com/engine/extend/plugins_authorization/
-[UCP]: https://docs.docker.com/datacenter/ucp/2.1/guides/
-[user and permission management]: https://docs.docker.com/datacenter/ucp/2.1/guides/admin/manage-users/
-
-
----
-
-class: encryption-at-rest
-
-## Encryption at rest
-
-- Swarm data is always encrypted
-
-- A Swarm cluster can be "locked"
-
-- When a cluster is "locked", the encryption key is protected with a passphrase
-
-- Starting or restarting a locked manager requires the passphrase
-
-- This protects against:
-
-  - theft (stealing a physical machine, a disk, a backup tape...)
-
-  - unauthorized access (to e.g. a remote or virtual volume)
-
-  - some vulnerabilities (like path traversal)
-
----
-
-class: encryption-at-rest
-
-## Locking a Swarm cluster
-
-- This is achieved through the `docker swarm update` command
-
-.exercise[
-
-- Lock our cluster:
-  ```bash
-  docker swarm update --autolock=true
-  ```
-
-]
-
-This will display the unlock key. Copy-paste it somewhere safe.
-
----
-
-class: encryption-at-rest
-
-## Locked state
-
-- If we restart a manager, it will now be locked
-
-.exercise[
-
-- Restart the local Engine:
-  ```bash
-  sudo systemctl restart docker
-  ```
-
-]
-
-Note: if you are doing the workshop on your own, using nodes
-that you [provisioned yourself](https://github.com/jpetazzo/orchestration-workshop/tree/master/prepare-machine) or with [Play-With-Docker](http://play-with-docker.com/), you might have to use a different method to restart the Engine.
-
----
-
-class: encryption-at-rest
-
-## Checking that our node is locked
-
-- Manager commands (requiring access to crypted data) will fail
-
-- Other commands are OK
-
-.exercise[
-
-- Try a few basic commands:
-  ```bash
-  docker ps
-  docker run alpine echo ♥
-  docker node ls
-  ```
-
-]
-
-(The last command should fail, and it will tell you how to unlock this node.)
-
----
-
-class: encryption-at-rest
-
-## Checking the state of the node programmatically
-
-- The state of the node shows up in the output of `docker info`
-
-.exercise[
-
-- Check the output of `docker info`:
-  ```bash
-  docker info
-  ```
-
-- Can't see it? Too verbose? Grep to the rescue!
-  ```bash
-  docker info | grep ^Swarm
-  ```
-
-]
-
----
-
-class: encryption-at-rest
-
-## Unlocking a node
-
-- You will need the secret token that we obtained when enabling auto-lock earlier
-
-.exercise[
-
-- Unlock the node:
-  ```bash
-  docker swarm unlock
-  ```
-
-- Copy-paste the secret token that we got earlier
-
-- Check that manager commands now work correctly:
-  ```bash
-  docker node ls
-  ```
-
-]
-
----
-
-class: encryption-at-rest
-
-## Managing the secret key
-
-- If the key is compromised, you can change it and re-encrypt with a new key:
-  ```bash
-  docker swarm unlock-key --rotate
-  ```
-
-- If you lost the key, you can get it as long as you have at least one unlocked node:
-  ```bash
-  docker swarm unlock-key -q
-  ```
-
-Note: if you rotate the key while some nodes are locked, without saving the previous key, those nodes won't be able to rejoin.
-
-Note: if somebody steals both your disks and your key, .strike[you're doomed! Doooooomed!]
-<br/>you can block the compromised node with `docker node demote` and `docker node rm`.
-
----
-
-class: encryption-at-rest
-
-## Unlocking the cluster permanently
-
-- If you want to remove the secret key, disable auto-lock
-
-.exercise[
-
-- Permanently unlock the cluster:
-  ```bash
-  docker swarm update --autolock=false
-  ```
-
-]
-
-Note: if some nodes are in locked state at that moment (or if they are offline/restarting
-while you disabled autolock), they still need the previous unlock key to get back online.
-
-For more information about locking, you can check the [upcoming documentation](https://github.com/docker/docker.github.io/pull/694).
-
----
-
-name: logging
-
-# Centralized logging
-
-- We want to send all our container logs to a central place
-
-- If that place could offer a nice web dashboard too, that'd be nice
-
---
-
-- We are going to deploy an ELK stack
-
-- It will accept logs over a GELF socket
-
-- We will update our services to send logs through the GELF logging driver
-
----
-
-# Setting up ELK to store container logs
-
-*Important foreword: this is not an "official" or "recommended"
-setup; it is just an example. We used ELK in this demo because
-it's a popular setup and we keep being asked about it; but you
-will have equal success with Fluent or other logging stacks!*
-
-What we will do:
-
-- Spin up an ELK stack with services
-
-- Gaze at the spiffy Kibana web UI
-
-- Manually send a few log entries using one-shot containers
-
-- Set our containers up to send their logs to Logstash
-
----
-
-## What's in an ELK stack?
-
-- ELK is three components:
-
-  - ElasticSearch (to store and index log entries)
-
-  - Logstash (to receive log entries from various
-    sources, process them, and forward them to various
-    destinations)
-
-  - Kibana (to view/search log entries with a nice UI)
-
-- The only component that we will configure is Logstash
-
-- We will accept log entries using the GELF protocol
-
-- Log entries will be stored in ElasticSearch,
-  <br/>and displayed on Logstash's stdout for debugging
-
----
-
-class: elk-manual
-
-## Setting up ELK
-
-- We need three containers: ElasticSearch, Logstash, Kibana
-
-- We will place them on a common network, `logging`
-
-.exercise[
-
-- Create the network:
-  ```bash
-  docker network create --driver overlay logging
-  ```
-
-- Create the ElasticSearch service:
-  ```bash
-  docker service create --network logging --name elasticsearch elasticsearch:2.4
-  ```
-
-]
-
----
-
-class: elk-manual
-
-## Setting up Kibana
-
-- Kibana exposes the web UI
-
-- Its default port (5601) needs to be published
-
-- It needs a tiny bit of configuration: the address of the ElasticSearch service
-
-- We don't want Kibana logs to show up in Kibana (it would create clutter)
-  <br/>so we tell Logspout to ignore them
-
-.exercise[
-
-- Create the Kibana service:
-  ```bash
-  docker service create --network logging --name kibana --publish 5601:5601 \
-         -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana:4.6
-  ```
-
-]
-
----
-
-class: elk-manual
-
-## Setting up Logstash
-
-- Logstash needs some configuration to listen to GELF messages and send them to ElasticSearch
-
-- We could author a custom image bundling this configuration
-
-- We can also pass the [configuration](https://github.com/jpetazzo/orchestration-workshop/blob/master/elk/logstash.conf) on the command line
-
-.exercise[
-
-- Create the Logstash service:
-  ```bash
-    docker service create --network logging --name logstash -p 12201:12201/udp \
-           logstash:2.4 -e "$(cat ~/orchestration-workshop/elk/logstash.conf)"
-  ```
-
-]
-
----
-
-class: elk-manual
-
-## Checking Logstash
-
-- Before proceeding, let's make sure that Logstash started properly
-
-.exercise[
-
-- Lookup the node running the Logstash container:
-  ```bash
-  docker service ps logstash
-  ```
-
-- Connect to that node
-
-]
-
----
-
-class: elk-manual
-
-## View Logstash logs
-
-.exercise[
-
-- Get the ID of the Logstash container:
-  ```bash
-  CID=$(docker ps -q --filter label=com.docker.swarm.service.name=logstash)
-  ```
-
-- View the logs:
-  ```bash
-  docker logs --follow $CID
-  ```
-
-]
-
-You should see the heartbeat messages:
-.small[
-```json
-{      "message" => "ok",
-          "host" => "1a4cfb063d13",
-      "@version" => "1",
-    "@timestamp" => "2016-06-19T00:45:45.273Z"
-}
-```
-]
-
----
-
-class: elk-auto
-
-## Deploying our ELK cluster
-
-- We will use a stack file
-
-.exercise[
-
-- Build, ship, and run our ELK stack:
-  ```bash
-  docker-compose -f elk.yml build
-  docker-compose -f elk.yml push
-  docker stack deploy elk -c elk.yml
-  ```
-
-]
-
-Note: the *build* and *push* steps are not strictly necessary, but they don't hurt!
-
-Let's have a look at the [Compose file](
-https://github.com/jpetazzo/orchestration-workshop/blob/master/stacks/elk.yml).
-
----
-
-class: elk-auto
-
-## Checking that our ELK stack works correctly
-
-- Let's view the logs of logstash
-
-  (Who logs the loggers?)
-
-.exercise[
-
-- Stream logstash's logs:
-  ```bash
-  docker service logs --follow --tail 1 elk_logstash
-  ```
-
-]
-
-You should see the heartbeat messages:
-
-.small[
-```json
-{      "message" => "ok",
-          "host" => "1a4cfb063d13",
-      "@version" => "1",
-    "@timestamp" => "2016-06-19T00:45:45.273Z"
-}
-```
-]
-
----
-
-## Testing the GELF receiver
-
-- In a new window, we will generate a logging message
-
-- We will use a one-off container, and Docker's GELF logging driver
-
-.exercise[
-
-- Send a test message:
-  ```bash
-    docker run --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:12201 \
-           --rm alpine echo hello
-  ```
-]
-
-The test message should show up in the logstash container logs.
-
----
-
-## Sending logs from a service
-
-- We were sending from a "classic" container so far; let's send logs from a service instead
-
-- We're lucky: the parameters (`--log-driver` and `--log-opt`) are exactly the same!
-
-
-.exercise[
-
-- Send a test message:
-  ```bash
-    docker service create \
-           --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:12201 \
-           alpine echo hello
-  ```
-
-]
-
-The test message should show up as well in the logstash container logs.
-
---
-
-In fact, *multiple messages will show up, and continue to show up every few seconds!*
-
----
-
-## Restart conditions
-
-- By default, if a container exits (or is killed with `docker kill`, or runs out of memory ...),
-  the Swarm will restart it (possibly on a different machine)
-
-- This behavior can be changed by setting the *restart condition* parameter
-
-.exercise[
-
-- Change the restart condition so that Swarm doesn't try to restart our container forever:
-  ```bash
-  docker service update `xxx` --restart-condition none
-  ```
-]
-
-Available restart conditions are `none`, `any`, and `on-error`.
-
-You can also set `--restart-delay`, `--restart-max-attempts`, and `--restart-window`.
-
----
-
-## Connect to Kibana
-
-- The Kibana web UI is exposed on cluster port 5601
-
-.exercise[
-
-- Connect to port 5601 of your cluster
-
-  - if you're using Play-With-Docker, click on the (5601) badge above the terminal
-
-  - otherwise, open http://(any-node-address):5601/ with your browser
-
-]
-
----
-
-## "Configuring" Kibana
-
-- If you see a status page with a yellow item, wait a minute and reload
-  (Kibana is probably still initializing)
-
-- Kibana should offer you to "Configure an index pattern":
-  <br/>in the "Time-field name" drop down, select "@timestamp", and hit the
-  "Create" button
-
-- Then:
-
-  - click "Discover" (in the top-left corner)
-  - click "Last 15 minutes" (in the top-right corner)
-  - click "Last 1 hour" (in the list in the middle)
-  - click "Auto-refresh" (top-right corner)
-  - click "5 seconds" (top-left of the list)
-
-- You should see a series of green bars (with one new green bar every minute)
-
----
-
-## Updating our services to use GELF
-
-- We will now inform our Swarm to add GELF logging to all our services
-
-- This is done with the `docker service update` command
-
-- The logging flags are the same as before
-
-.exercise[
-
-<!--
-
-- Enable GELF logging for all our *stateless* services:
-  ```bash
-    for SERVICE in hasher rng webui worker; do
-      docker service update dockercoins_$SERVICE \
-             --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:12201
-    done
-  ```
-
--->
-
-- Enable GELF logging for the `rng` service:
-  ```bash
-    docker service update dockercoins_rng \
-           --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:12201
-  ```
-
-]
-
-After ~15 seconds, you should see the log messages in Kibana.
-
----
-
-## Viewing container logs
-
-- Go back to Kibana
-
-- Container logs should be showing up!
-
-- We can customize the web UI to be more readable
-
-.exercise[
-
-- In the left column, move the mouse over the following
-  columns, and click the "Add" button that appears:
-
-  - host
-  - container_name
-  - message
-
-<!--
-  - logsource
-  - program
-  - message
--->
-
-]
-
----
-
-## .warning[Don't update stateful services!]
-
-- What would have happened if we had updated the Redis service?
-
-- When a service changes, SwarmKit replaces existing container with new ones
-
-- This is fine for stateless services
-
-- But if you update a stateful service, its data will be lost in the process
-
-- If we updated our Redis service, all our DockerCoins would be lost
-
----
-
-## Important afterword
-
-**This is not a "production-grade" setup.**
-
-It is just an educational example. We did set up a single
-ElasticSearch instance and a single Logstash instance.
-
-In a production setup, you need an ElasticSearch cluster
-(both for capacity and availability reasons). You also
-need multiple Logstash instances.
-
-And if you want to withstand
-bursts of logs, you need some kind of message queue:
-Redis if you're cheap, Kafka if you want to make sure
-that you don't drop messages on the floor. Good luck.
-
-If you want to learn more about the GELF driver,
-have a look at [this blog post](
-http://jpetazzo.github.io/2017/01/20/docker-logging-gelf/).
-
----
-
-# Metrics collection
-
-- We want to gather metrics in a central place
-
-- We will gather node metrics and container metrics
-
-- We want a nice interface to view them (graphs)
-
----
-
-## Node metrics
-
-- CPU, RAM, disk usage on the whole node
-
-- Total number of processes running, and their states
-
-- Number of open files, sockets, and their states
-
-- I/O activity (disk, network), per operation or volume
-
-- Physical/hardware (when applicable): temperature, fan speed ...
-
-- ... and much more!
-
----
-
-## Container metrics
-
-- Similar to node metrics, but not totally identical
-
-- RAM breakdown will be different
-
-  - active vs inactive memory
-  - some memory is *shared* between containers, and accounted specially
-
-- I/O activity is also harder to track
-
-  - async writes can cause deferred "charges"
-  - some page-ins are also shared between containers
-
-For details about container metrics, see:
-<br/>
-http://jpetazzo.github.io/2013/10/08/docker-containers-metrics/
-
----
-
-class: snap, prom
-
-## Tools
-
-We will build *two* different metrics pipelines:
-
-- One based on Intel Snap,
-
-- Another based on Prometheus.
-
-If you're using Play-With-Docker, skip the exercises
-relevant to Intel Snap (we rely on a SSH server to deploy,
-and PWD doesn't have that yet).
-
----
-
-class: snap
-
-## First metrics pipeline
-
-We will use three open source Go projects for our first metrics pipeline:
-
-- Intel Snap
-
-  Collects, processes, and publishes metrics
-
-- InfluxDB
-
-  Stores metrics
-
-- Grafana
-
-  Displays metrics visually
-
----
-
-class: snap
-
-## Snap
-
-- [github.com/intelsdi-x/snap](https://github.com/intelsdi-x/snap)
-
-- Can collect, process, and publish metric data
-
-- Doesn’t store metrics
-
-- Works as a daemon (snapd) controlled by a CLI (snapctl)
-
-- Offloads collecting, processing, and publishing to plugins
-
-- Does nothing out of the box; configuration required!
-
-- Docs: https://github.com/intelsdi-x/snap/blob/master/docs/
-
----
-
-class: snap
-
-## InfluxDB
-
-- Snap doesn't store metrics data
-
-- InfluxDB is specifically designed for time-series data
-
-  - CRud vs. CRUD (you rarely if ever update/delete data)
-
-  - orthogonal read and write patterns
-
-  - storage format optimization is key (for disk usage and performance)
-
-- Snap has a plugin allowing to *publish* to InfluxDB
-
----
-
-class: snap
-
-## Grafana
-
-- Snap cannot show graphs
-
-- InfluxDB cannot show graphs
-
-- Grafana will take care of that
-
-- Grafana can read data from InfluxDB and display it as graphs
-
----
-
-class: snap
-
-## Getting and setting up Snap
-
-- We will install Snap directly on the nodes
-
-- Release tarballs are available from GitHub
-
-- We will use a *global service*
-  <br/>(started on all nodes, including nodes added later)
-
-- This service will download and unpack Snap in /opt and /usr/local
-
-- /opt and /usr/local will be bind-mounted from the host
-
-- This service will effectively install Snap on the hosts
-
----
-
-class: snap
-
-## The Snap installer service
-
-- This will get Snap on all nodes
-
-.exercise[
-
-```bash
-docker service create --restart-condition=none --mode global \
-       --mount type=bind,source=/usr/local/bin,target=/usr/local/bin \
-       --mount type=bind,source=/opt,target=/opt centos sh -c '
-SNAPVER=v0.16.1-beta
-RELEASEURL=https://github.com/intelsdi-x/snap/releases/download/$SNAPVER
-curl -sSL $RELEASEURL/snap-$SNAPVER-linux-amd64.tar.gz |
-     tar -C /opt -zxf-
-curl -sSL $RELEASEURL/snap-plugins-$SNAPVER-linux-amd64.tar.gz |
-     tar -C /opt -zxf-
-ln -s snap-$SNAPVER /opt/snap
-for BIN in snapd snapctl; do ln -s /opt/snap/bin/$BIN /usr/local/bin/$BIN; done
-' # If you copy-paste that block, do not forget that final quote ☺
-```
-
-]
-
----
-
-class: snap
-
-## First contact with `snapd`
-
-- The core of Snap is `snapd`, the Snap daemon
-
-- Application made up of a REST API, control module, and scheduler module
-
-.exercise[
-
-- Start `snapd` with plugin trust disabled and log level set to debug:
-  ```bash
-  snapd -t 0 -l 1
-  ```
-
-]
-
-- More resources:
-
-  https://github.com/intelsdi-x/snap/blob/master/docs/SNAPD.md
-  https://github.com/intelsdi-x/snap/blob/master/docs/SNAPD_CONFIGURATION.md
-
----
-
-class: snap
-
-## Using `snapctl` to interact with `snapd`
-
-- Let's load a *collector* and a *publisher* plugins
-
-.exercise[
-
-- Open a new terminal
-
-- Load the psutil collector plugin:
-  ```bash
-  snapctl plugin load /opt/snap/plugin/snap-plugin-collector-psutil
-  ```
-
-- Load the file publisher plugin:
-  ```bash
-  snapctl plugin load /opt/snap/plugin/snap-plugin-publisher-mock-file
-  ```
-
-]
-
----
-
-class: snap
-
-## Checking what we've done
-
-- Good to know: Docker CLI uses `ls`, Snap CLI uses `list`
-
-.exercise[
-
-- See your loaded plugins:
-  ```bash
-  snapctl plugin list
-  ```
-
-- See the metrics you can collect:
-  ```bash
-  snapctl metric list
-  ```
-
-]
-
----
-
-class: snap
-
-## Actually collecting metrics: introducing *tasks*
-
-- To start collecting/processing/publishing metric data, you need to create a *task*
-
-- A *task* indicates:
-
-  - *what* to collect (which metrics)
-  - *when* to collect it (e.g. how often)
-  - *how* to process it (e.g. use it directly, or compute moving averages)
-  - *where* to publish it
-
-- Tasks can be defined with manifests written in JSON or YAML
-
-- Some plugins, such as the Docker collector, allow for wildcards (\*) in the metrics "path"
-  <br/>(see snap/docker-influxdb.json)
-
-- More resources:
-  https://github.com/intelsdi-x/snap/blob/master/docs/TASKS.md
-
----
-
-class: snap
-
-## Our first task manifest
-
-```yaml
-  version: 1
-  schedule:
-    type: "simple" # collect on a set interval
-    interval: "1s" # of every 1s
-  max-failures: 10
-  workflow:
-    collect: # first collect
-      metrics: # metrics to collect
-        /intel/psutil/load/load1: {}
-      config: # there is no configuration
-      publish: # after collecting, publish
-        -
-            plugin_name: "file" # use the file publisher
-            config:
-                file: "/tmp/snap-psutil-file.log" # write to this file
-```
-
----
-
-class: snap
-
-## Creating our first task
-
-- The task manifest shown on the previous slide is stored in `snap/psutil-file.yml`.
-
-.exercise[
-
-- Create a task using the manifest:
-
-  ```bash
-  cd ~/orchestration-workshop/snap
-  snapctl task create -t psutil-file.yml
-  ```
-
-]
-
-  The output should look like the following:
-  ```
-    Using task manifest to create task
-    Task created
-    ID: 240435e8-a250-4782-80d0-6fff541facba
-    Name: Task-240435e8-a250-4782-80d0-6fff541facba
-    State: Running
-  ```
-
----
-
-class: snap
-
-## Checking existing tasks
-
-.exercise[
-
-- This will confirm that our task is running correctly, and remind us of its task ID
-
-  ```bash
-  snapctl task list
-  ```
-
-]
-
-The output should look like the following:
-  ```
-    ID           NAME              STATE     HIT MISS FAIL CREATED
-    24043...acba Task-24043...acba Running   4   0    0    2:34PM   8-13-2016
-  ```
----
-
-class: snap
-
-## Viewing our task dollars at work
-
-- The task is using a very simple publisher, `mock-file`
-
-- That publisher just writes text lines in a file (one line per data point)
-
-.exercise[
-
-- Check that the data is flowing indeed:
-  ```bash
-  tail -f /tmp/snap-psutil-file.log
-  ```
-
-]
-
-To exit, hit `^C`
-
----
-
-class: snap
-
-## Debugging tasks
-
-- When a task is not directly writing to a local file, use `snapctl task watch`
-
-- `snapctl task watch` will stream the metrics you are collecting to STDOUT
-
-.exercise[
-
-```bash
-snapctl task watch <ID>
-```
-
-]
-
-To exit, hit `^C`
-
----
-
-class: snap
-
-## Stopping snap
-
-- Our Snap deployment has a few flaws:
-
-  - snapd was started manually
-
-  - it is running on a single node
-
-  - the configuration is purely local
-
---
-
-class: snap
-
-- We want to change that!
-
---
-
-class: snap
-
-- But first, go back to the terminal where `snapd` is running, and hit `^C`
-
-- All tasks will be stopped; all plugins will be unloaded; Snap will exit
-
----
-
-class: snap
-
-## Snap Tribe Mode
-
-- Tribe is Snap's clustering mechanism
-
-- When tribe mode is enabled, nodes can join *agreements*
-
-- When a node in an *agreement* does something (e.g. load a plugin or run a task),
-  <br/>other nodes of that agreement do the same thing
-
-- We will use it to load the Docker collector and InfluxDB publisher on all nodes,
-  <br/>and run a task to use them
-
-- Without tribe mode, we would have to load plugins and run tasks manually on every node
-
-- More resources:
-  https://github.com/intelsdi-x/snap/blob/master/docs/TRIBE.md
-
----
-
-class: snap
-
-## Running Snap itself on every node
-
-- Snap runs in the foreground, so you need to use `&` or start it in tmux
-
-.exercise[
-
-- Run the following command *on every node:*
-  ```bash
-  snapd -t 0 -l 1 --tribe --tribe-seed node1:6000
-  ```
-
-]
-
-If you're *not* using Play-With-Docker, there is another way to start Snap!
-
----
-
-class: snap
-
-## Starting a daemon through SSH
-
-.warning[Hackety hack ahead!]
-
-- We will create a *global service*
-
-- That global service will install a SSH client
-
-- With that SSH client, the service will connect back to its local node
-  <br/>(i.e. "break out" of the container, using the SSH key that we provide)
-
-- Once logged on the node, the service starts snapd with Tribe Mode enabled
-
----
-
-class: snap
-
-## Running Snap itself on every node
-
-- I might go to hell for showing you this, but here it goes ...
-
-.exercise[
-
-- Start Snap all over the place:
-  ```bash
-    docker service create --name snapd --mode global \
-           --mount type=bind,source=$HOME/.ssh/id_rsa,target=/sshkey \
-           alpine sh -c "
-                  apk add --no-cache openssh-client &&
-                  ssh -o StrictHostKeyChecking=no -i /sshkey docker@172.17.0.1 \
-                      sudo snapd -t 0 -l 1 --tribe --tribe-seed node1:6000
-           " # If you copy-paste that block, don't forget that final quote :-)
-   ```
-
-]
-
-Remember: this *does not work* with Play-With-Docker (which doesn't have SSH).
-
----
-
-class: snap
-
-## Viewing the members of our tribe
-
-- If everything went fine, Snap is now running in tribe mode
-
-.exercise[
-
-- View the members of our tribe:
-  ```bash
-  snapctl member list
-  ```
-
-]
-
-This should show the 5 nodes with their hostnames.
-
----
-
-class: snap
-
-## Create an agreement
-
-- We can now create an *agreement* for our plugins and tasks
-
-.exercise[
-
-- Create an agreement; make sure to use the same name all along:
-  ```bash
-  snapctl agreement create docker-influxdb
-  ```
-
-]
-
-The output should look like the following:
-
-```
-  Name             Number of Members       plugins      tasks
-  docker-influxdb  0                       0            0
-```
-
----
-
-class: snap
-
-## Instruct all nodes to join the agreement
-
-- We dont need another fancy global service!
-
-- We can join nodes from any existing node of the cluster
-
-.exercise[
-
-- Add all nodes to the agreement:
-  ```bash
-    snapctl member list | tail -n +2 |
-      xargs -n1 snapctl agreement join docker-influxdb
-  ```
-
-]
-
-The last bit of output should look like the following:
-```
-  Name             Number of Members       plugins         tasks
-  docker-influxdb  5                       0               0
-```
-
----
-
-class: snap
-
-## Start a container on every node
-
-- The Docker plugin requires at least one container to be started
-
-- Normally, at this point, you will have at least one container on each node
-
-- But just in case you did things differently, let's create a dummy global service
-
-.exercise[
-
-- Create an alpine container on the whole cluster:
-  ```bash
-    docker service create --name ping --mode global alpine ping 8.8.8.8
-  ```
-
-]
-
----
-
-class: snap
-
-## Running InfluxDB
-
-- We will create a service for InfluxDB
-
-- We will use the official image
-
-- InfluxDB uses multiple ports:
-
-  - 8086 (HTTP API; we need this)
-
-  - 8083 (admin interface; we need this)
-
-  - 8088 (cluster communication; not needed here)
-
-  - more ports for other protocols (graphite, collectd...)
-
-- We will just publish the first two
-
----
-
-class: snap
-
-## Creating the InfluxDB service
-
-.exercise[
-
-- Start an InfluxDB service, publishing ports 8083 and 8086:
-  ```bash
-    docker service create --name influxdb \
-           --publish 8083:8083 \
-           --publish 8086:8086 \
-           influxdb:0.13
-  ```
-
-]
-
-Note: this will allow any node to publish metrics data to `localhost:8086`,
-and it will allows us to access the admin interface by connecting to any node
-on port 8083.
-
-.warning[Make sure to use InfluxDB 0.13; a few things changed in 1.0
-(like, the name of the default retention policy is now "autogen") and
-this breaks a few things.]
-
----
-
-class: snap
-
-## Setting up InfluxDB
-
-- We need to create the "snap" database
-
-.exercise[
-
-- Open port 8083 with your browser
-
-- Enter the following query in the query box:
-  ```
-  CREATE DATABASE "snap"
-  ```
-
-- In the top-right corner, select "Database: snap"
-
-]
-
-Note: the InfluxDB query language *looks like* SQL but it's not.
-
-???
-
-## Setting a retention policy
-
-- When graduating to 1.0, InfluxDB changed the name of the default policy
-
-- It used to be "default" and it is now "autogen"
-
-- Snap still uses "default" and this results in errors
-
-.exercise[
-
-- Create a "default" retention policy by entering the following query in the box:
-  ```
-  CREATE RETENTION POLICY "default" ON "snap" DURATION 1w REPLICATION 1
-  ```
-
-]
-
----
-
-class: snap
-
-## Load Docker collector and InfluxDB publisher
-
-- We will load plugins on the local node
-
-- Since our local node is a member of the agreement, all other
-  nodes in the agreement will also load these plugins
-
-.exercise[
-
-- Load Docker collector:
-
-  ```bash
-  snapctl plugin load /opt/snap/plugin/snap-plugin-collector-docker
-  ```
-
-- Load InfluxDB publisher:
-
-  ```bash
-  snapctl plugin load /opt/snap/plugin/snap-plugin-publisher-influxdb
-  ```
-
-]
-
----
-
-class: snap
-
-## Start a simple collection task
-
-- Again, we will create a task on the local node
-
-- The task will be replicated on other nodes members of the same agreement
-
-.exercise[
-
-- Load a task manifest file collecting a couple of metrics on all containers,
-  <br/>and sending them to InfluxDB:
-  ```bash
-  cd ~/orchestration-workshop/snap
-  snapctl task create -t docker-influxdb.json
-  ```
-
-]
-
-Note: the task description sends metrics to the InfluxDB API endpoint
-located at 127.0.0.1:8086. Since the InfluxDB container is published
-on port 8086, 127.0.0.1:8086 always routes traffic to the InfluxDB
-container.
-
----
-
-class: snap
-
-## If things go wrong...
-
-Note: if a task runs into a problem (e.g. it's trying to publish
-to a metrics database, but the database is unreachable), the task
-will be stopped.
-
-You will have to restart it manually by running:
-
-```bash
-snapctl task enable <ID>
-snapctl task start <ID>
-```
-
-This must be done *per node*. Alternatively, you can delete+re-create
-the task (it will delete+re-create on all nodes).
-
----
-
-class: snap
-
-## Check that metric data shows up in InfluxDB
-
-- Let's check existing data with a few manual queries in the InfluxDB admin interface
-
-.exercise[
-
-- List "measurements":
-  ```
-  SHOW MEASUREMENTS
-  ```
-  (This should show two generic entries corresponding to the two collected metrics.)
-
-- View time series data for one of the metrics:
-  ```
-  SELECT * FROM "intel/docker/stats/cgroups/cpu_stats/cpu_usage/total_usage"
-  ```
-  (This should show a list of data points with **time**, **docker_id**, **source**, and **value**.)
-
-]
-
----
-
-class: snap
-
-## Deploy Grafana
-
-- We will use an almost-official image, `grafana/grafana`
-
-- We will publish Grafana's web interface on its default port (3000)
-
-.exercise[
-
-- Create the Grafana service:
-  ```bash
-  docker service create --name grafana --publish 3000:3000 grafana/grafana:3.1.1
-  ```
-
-]
-
----
-
-class: snap
-
-## Set up Grafana
-
-.exercise[
-
-- Open port 3000 with your browser
-
-- Identify with "admin" as the username and password
-
-- Click on the Grafana logo (the orange spiral in the top left corner)
-
-- Click on "Data Sources"
-
-- Click on "Add data source" (green button on the right)
-
-]
-
----
-
-class: snap
-
-## Add InfluxDB as a data source for Grafana
-
-.small[
-
-Fill the form exactly as follows:
-- Name = "snap"
-- Type = "InfluxDB"
-
-In HTTP settings, fill as follows:
-- Url = "http://(IP.address.of.any.node):8086"
-- Access = "direct"
-- Leave HTTP Auth untouched
-
-In InfluxDB details, fill as follows:
-- Database = "snap"
-- Leave user and password blank
-
-Finally, click on "add", you should see a green message saying "Success - Data source is working".
-If you see an orange box (sometimes without a message), it means that you got something wrong. Triple check everything again.
-
-]
-
----
-
-class: snap
-
-![Screenshot showing how to fill the form](grafana-add-source.png)
-
----
-
-class: snap
-
-## Create a dashboard in Grafana
-
-.exercise[
-
-- Click on the Grafana logo again (the orange spiral in the top left corner)
-
-- Hover over "Dashboards"
-
-- Click "+ New"
-
-- Click on the little green rectangle that appeared in the top left
-
-- Hover over "Add Panel"
-
-- Click on "Graph"
-
-]
-
-At this point, you should see a sample graph showing up.
-
----
-
-class: snap
-
-## Setting up a graph in Grafana
-
-.exercise[
-
-- Panel data source: select snap
-- Click on the SELECT metrics query to expand it
-- Click on "select measurement" and pick CPU usage
-- Click on the "+" right next to "WHERE"
-- Select "docker_id"
-- Select the ID of a container of your choice (e.g. the one running InfluxDB)
-- Click on the "+" on the right of the "SELECT" line
-- Add "derivative"
-- In the "derivative" option, select "1s"
-- In the top right corner, click on the clock, and pick "last 5 minutes"
-
-]
-
-Congratulations, you are viewing the CPU usage of a single container!
-
----
-
-class: snap
-
-![Screenshot showing the end result](grafana-add-graph.png)
-
----
-
-class: snap, prom
-
-## Before moving on ...
-
-- Leave that tab open!
-
-- We are going to setup *another* metrics system
-
-- ... And then compare both graphs side by side
-
----
-
-class: snap, prom
-
-## Prometheus vs. Snap
-
-- Prometheus is another metrics collection system
-
-- Snap *pushes* metrics; Prometheus *pulls* them
-
----
-
-class: prom
-
-## Prometheus components
-
-- The *Prometheus server* pulls, stores, and displays metrics
-
-- Its configuration defines a list of *exporter* endpoints
-  <br/>(that list can be dynamic, using e.g. Consul, DNS, Etcd...)
-
-- The exporters expose metrics over HTTP using a simple line-oriented format
-
-  (An optimized format using protobuf is also possible)
-
----
-
-class: prom
-
-## It's all about the `/metrics`
-
-- This is was the *node exporter* looks like:
-
-  http://demo.robustperception.io:9100/metrics
-
-- Prometheus itself exposes its own internal metrics, too:
-
-  http://demo.robustperception.io:9090/metrics
-
-- A *Prometheus server* will *scrape* URLs like these
-
-  (It can also use protobuf to avoid the overhead of parsing line-oriented formats!)
-
----
-
-class: prom-manual
-
-## Collecting metrics with Prometheus on Swarm
-
-- We will run two *global services* (i.e. scheduled on all our nodes):
-
-  - the Prometheus *node exporter* to get node metrics
-
-  - Google's cAdvisor to get container metrics
-
-- We will run a Prometheus server to scrape these exporters
-
-- The Prometheus server will be configured to use DNS service discovery
-
-- We will use `tasks.<servicename>` for service discovery
-
-- All these services will be placed on a private internal network
-
----
-
-class: prom-manual
-
-## Creating an overlay network for Prometheus
-
-- This is the easiest step ☺
-
-.exercise[
-
-- Create an overlay network:
-  ```bash
-  docker network create --driver overlay prom
-  ```
-
-]
-
----
-
-class: prom-manual
-
-## Running the node exporter
-
-- The node exporter *should* run directly on the hosts
-- However, it can run from a container, if configured properly
-  <br/>
-  (it needs to access the host's filesystems, in particular /proc and /sys)
-
-.exercise[
-
-- Start the node exporter:
-  ```bash
-    docker service create --name node --mode global --network prom \
-     --mount type=bind,source=/proc,target=/host/proc \
-     --mount type=bind,source=/sys,target=/host/sys \
-     --mount type=bind,source=/,target=/rootfs \
-     prom/node-exporter \
-      -collector.procfs /host/proc \
-      -collector.sysfs /host/proc \
-      -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"
-   ```
-
-]
-
----
-
-class: prom-manual
-
-## Running cAdvisor
-
-- Likewise, cAdvisor *should* run directly on the hosts
-
-- But it can run in containers, if configured properly
-
-.exercise[
-
-- Start the cAdvisor collector:
-  ```bash
-    docker service create --name cadvisor --network prom --mode global \
-      --mount type=bind,source=/,target=/rootfs \
-      --mount type=bind,source=/var/run,target=/var/run \
-      --mount type=bind,source=/sys,target=/sys \
-      --mount type=bind,source=/var/lib/docker,target=/var/lib/docker \
-      google/cadvisor:latest
-  ```
-
-]
-
----
-
-class: prom-manual
-
-## Configuring the Prometheus server
-
-This will be our configuration file for Prometheus:
-
-```yaml
-global:
-  scrape_interval: 10s
-scrape_configs:
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['localhost:9090']
-  - job_name: 'node'
-    dns_sd_configs:
-      - names: ['tasks.node']
-        type: 'A'
-        port: 9100
-  - job_name: 'cadvisor'
-    dns_sd_configs:
-      - names: ['tasks.cadvisor']
-        type: 'A'
-        port: 8080
-```
-
----
-
-class: prom-manual
-
-## Passing the configuration to the Prometheus server
-
-- We need to provide our custom configuration to the Prometheus server
-
-- The easiest solution is to create a custom image bundling this configuration
-
-- We will use a very simple Dockerfile:
-  ```dockerfile
-  FROM prom/prometheus:v1.4.1
-  COPY prometheus.yml /etc/prometheus/prometheus.yml
-  ```
-
-  (The configuration file, and the Dockerfile, are in the `prom` subdirectory)
-
-- We will build this image, and push it to our local registry
-
-- Then we will create a service using this image
-
-Note: it is also possible to use a `config` to inject that configuration file
-without having to create this ad-hoc image.
-
----
-
-class: prom-manual
-
-## Building our custom Prometheus image
-
-- We will use the local registry started previously on 127.0.0.1:5000
-
-.exercise[
-
-- Build the image using the provided Dockerfile:
-  ```bash
-  docker build -t 127.0.0.1:5000/prometheus ~/orchestration-workshop/prom
-  ```
-
-- Push the image to our local registry:
-  ```bash
-  docker push 127.0.0.1:5000/prometheus
-  ```
-
-]
-
----
-
-class: prom-manual
-
-## Running our custom Prometheus image
-
-- That's the only service that needs to be published
-
-  (If we want to access Prometheus from outside!)
-
-.exercise[
-
-- Start the Prometheus server:
-  ```bash
-    docker service create --network prom --name prom \
-           --publish 9090:9090 127.0.0.1:5000/prometheus
-  ```
-
-]
-
----
-
-class: prom-auto
-
-## Deploying Prometheus on our cluster
-
-- We will use a stack definition (once again)
-
-.exercise[
-
-- Make sure we are in the stacks directory:
-  ```bash
-  cd ~/orchestration-workshop/stacks
-  ```
-
-- Build, ship, and run the Prometheus stack:
-  ```bash
-  docker-compose -f prometheus.yml build
-  docker-compose -f prometheus.yml push
-  docker stack deploy -c prometheus.yml prometheus
-  ```
-
-]
-
----
-
-class: prom
-
-## Checking our Prometheus server
-
-- First, let's make sure that Prometheus is correctly scraping all metrics
-
-.exercise[
-
-- Open port 9090 with your browser
-
-- Click on "status", then "targets"
-
-]
-
-You should see 11 endpoints (5 cadvisor, 5 node, 1 prometheus).
-
-Their state should be "UP".
-
----
-
-class: prom-auto, config
-
-## Injecting a configuration file
-
-(New in Docker Engine 17.06)
-
-- We are creating a custom image *just to inject a configuration*
-
-- Instead, we could use the base Prometheus image + a `config` 
-
-- A `config` is a blob (usually, a configuration file) that:
-
-  - is created and managed through the Docker API (and CLI)
-
-  - gets persisted into the Raft log (i.e. safely)
-
-  - can be associated to a service
-    <br/>
-    (this injects the blob as a plain file in the service's containers)
-
----
-
-class: prom-auto, config
-
-## Differences between `config` and `secret`
-
-The two are very similar, but ...
-
-- `configs`:
-
-  - can be injected to any filesystem location
-
-  - can be viewed and extracted using the Docker API or CLI
-
-- `secrets`:
-
-  - can only be injected into `/run/secrets`
-
-  - are never stored in clear text on disk
-
-  - cannot be viewed or extracted with the Docker API or CLI
-
----
-
-class: prom-auto, config
-
-## Deploying Prometheus with a `config`
-
-- The `config` can be created manually or declared in the Compose file
-
-- This is what our new Compose file looks like:
-
-.small[
-```yaml
-version: "3.3"
-
-services:
-
-prometheus:
-  image: prom/prometheus:v1.4.1 
-  ports:
-    - "9090:9090"
-  configs:
-    - source: prometheus
-      target: /etc/prometheus/prometheus.yml
-
-...
-
-configs:
-  prometheus:
-    file: ../prom/prometheus.yml
-```
-]
-
-(This is from `prometheus+config.yml`)
-
----
-
-class: prom-auto, config
-
-## Specifying a `config` in a Compose file
-
-- In each service, an optional `configs` section can list as many configs as you want
-
-- Each config can specify:
-
-  - an optional `target` (path to inject the configuration; by default: root of the container)
-
-  - ownership and permissions (by default, the file will be owned by UID 0, i.e. `root`)
-
-- These configs reference top-level `configs` elements
-
-- The top-level configs can be declared as:
-
-  - *external*, meaning that it is supposed to be created before you deploy the stack
-
-  - referencing a file, whose content is used to initialize the config
-
----
-
-class: prom-auto, config
-
-## Re-deploying Prometheus with a config
-
-- We will update the existing stack using `prometheus+config.yml`
-
-.exercise[
-
-- Redeploy the `prometheus` stack:
-  ```bash
-  docker stack deploy -c prometheus+config.yml prometheus
-  ```
-
-- Check that Prometheus still works as intended
-
-  (By connecting to any node of the cluster, on port 9090)
-
-]
-
----
-
-class: prom-auto, config
-
-## Accessing the config object from the Docker CLI
-
-- Config objects can be viewed from the CLI (or API)
-
-.exercise[
-
-- List existing config objects:
-  ```bash
-  docker config ls
-  ```
-
-- View details about our config object:
-  ```bash
-  docker config inspect prometheus_prometheus
-  ```
-
-]
-
-Note: the content of the config blob is shown with BASE64 encoding.
-<br/>
-(It doesn't have to be text; it could be an image or any kind of binary content!)
-
----
-
-class: prom-auto, config
-
-## Extracting a config blob
-
-- Let's retrieve that Prometheus configuration!
-
-.exercise[
-
-- Extract the BASE64 payload with `jq`:
-  ```bash
-  docker config inspect prometheus_prometheus | jq -r .[0].Spec.Data
-  ```
-
-- Decode it with `base64 -d`:
-  ```bash
-  docker config inspect prometheus_prometheus | jq -r .[0].Spec.Data | base64 -d
-  ```
-
-]
-
----
-
-class: prom
-
-## Displaying metrics directly from Prometheus
-
-- This is easy ... if you are familiar with PromQL
-
-.exercise[
-
-- Click on "Graph", and in "expression", paste the following:
-  ```
-    sum by (container_label_com_docker_swarm_node_id) (
-      irate(
-        container_cpu_usage_seconds_total{
-          container_label_com_docker_swarm_service_name="dockercoins_worker"
-          }[1m]
-      )
-    )
-  ```
-
-- Click on the blue "Execute" button and on the "Graph" tab just below
-
-]
-
----
-
-class: prom
-
-## Building the query from scratch
-
-- We are going to build the same query from scratch
-
-- This doesn't intend to be a detailed PromQL course
-
-- This is merely so that you (I) can pretend to know how the previous query works
-  <br/>so that your coworkers (you) can be suitably impressed (or not)
-
-  (Or, so that we can build other queries if necessary, or adapt if cAdvisor,
-  Prometheus, or anything else changes and requires editing the query!)
-
----
-
-class: prom
-
-## Displaying a raw metric for *all* containers
-
-- Click on the "Graph" tab on top
-
-  *This takes us to a blank dashboard*
-
-- Click on the "Insert metric at cursor" drop down, and select `container_cpu_usage_seconds_total`
-
-  *This puts the metric name in the query box*
-
-- Click on "Execute"
-
-  *This fills a table of measurements below*
-
-- Click on "Graph" (next to "Console")
-
-  *This replaces the table of measurements with a series of graphs (after a few seconds)*
-
----
-
-class: prom
-
-## Selecting metrics for a specific service
-
-- Hover over the lines in the graph
-
-  (Look for the ones that have labels like `container_label_com_docker_...`)
-
-- Edit the query, adding a condition between curly braces:
-
-  .small[`container_cpu_usage_seconds_total{container_label_com_docker_swarm_service_name="dockercoins_worker"}`]
-
-- Click on "Execute"
-
-  *Now we should see one line per CPU per container*
-
-- If you want to select by container ID, you can use a regex match: `id=~"/docker/c4bf.*"`
-
-- You can also specify multiple conditions by separating them with commas
-
----
-
-class: prom
-
-## Turn counters into rates
-
-- What we see is the total amount of CPU used (in seconds)
-
-- We want to see a *rate* (CPU time used / real time)
-
-- To get a moving average over 1 minute periods, enclose the current expression within:
-
-  ```
-  rate ( ... { ... } [1m] )
-  ```
-
-  *This should turn our steadily-increasing CPU counter into a wavy graph*
-
-- To get an instantaneous rate, use `irate` instead of `rate`
-
-  (The time window is then used to limit how far behind to look for data if data points
-  are missing in case of scrape failure; see [here](https://www.robustperception.io/irate-graphs-are-better-graphs/) for more details!)
-
-  *This should show spikes that were previously invisible because they were smoothed out*
-
----
-
-class: prom
-
-## Aggregate multiple data series
-
-- We have one graph per CPU per container; we want to sum them
-
-- Enclose the whole expression within:
-
-  ```
-  sum ( ... )
-  ```
-
-  *We now see a single graph*
-
----
-
-class: prom
-
-## Collapse dimensions
-
-- If we have multiple containers we can also collapse just the CPU dimension:
-
-  ```
-  sum without (cpu) ( ... )
-  ```
-
-  *This shows the same graph, but preserves the other labels*
-
-- Congratulations, you wrote your first PromQL expression from scratch!
-
-  (I'd like to thank [Johannes Ziemke](https://twitter.com/discordianfish) and
-  [Julius Volz](https://twitter.com/juliusvolz) for their help with Prometheus!)
-
----
-
-class: prom, snap
-
-## Comparing Snap and Prometheus data
-
-- If you haven't setup Snap, InfluxDB, and Grafana, skip this section
-
-- If you have closed the Grafana tab, you might have to re-setup a new dashboard
-
-  (Unless you saved it before navigating it away)
-
-- To re-do the setup, just follow again the instructions from the previous chapter
-
----
-
-class: prom, snap
-
-## Add Prometheus as a data source in Grafana
-
-.exercise[
-
-- In a new tab, connect to Grafana (port 3000)
-
-- Click on the Grafana logo (the orange spiral in the top-left corner)
-
-- Click on "Data Sources"
-
-- Click on the green "Add data source" button
-
-]
-
-We see the same input form that we filled earlier to connect to InfluxDB.
-
----
-
-class: prom, snap
-
-## Connecting to Prometheus from Grafana
-
-.exercise[
-
-- Enter "prom" in the name field
-
-- Select "Prometheus" as the source type
-
-- Enter http://(IP.address.of.any.node):9090 in the Url field
-
-- Select "direct" as the access method
-
-- Click on "Save and test"
-
-]
-
-Again, we should see a green box telling us "Data source is working."
-
-Otherwise, double-check every field and try again!
-
----
-
-class: prom, snap
-
-## Adding the Prometheus data to our dashboard
-
-.exercise[
-
-- Go back to the the tab where we had our first Grafana dashboard
-
-- Click on the blue "Add row" button in the lower right corner
-
-- Click on the green tab on the left; select "Add panel" and "Graph"
-
-]
-
-This takes us to the graph editor that we used earlier.
-
----
-
-class: prom, snap
-
-## Querying Prometheus data from Grafana
-
-The editor is a bit less friendly than the one we used for InfluxDB.
-
-.exercise[
-
-- Select "prom" as Panel data source
-
-- Paste the query in the query field:
-  ```
-    sum without (cpu, id) ( irate (
-      container_cpu_usage_seconds_total{
-        container_label_com_docker_swarm_service_name="influxdb"}[1m] ) )
-  ```
-
-- Click outside of the query field to confirm
-
-- Close the row editor by clicking the "X" in the top right area
-
-]
-
----
-
-class: prom, snap
-
-## Interpreting results
-
-- The two graphs *should* be similar
-
-- Protip: align the time references!
-
-.exercise[
-
-- Click on the clock in the top right corner
-
-- Select "last 30 minutes"
-
-- Click on "Zoom out"
-
-- Now press the right arrow key (hold it down and watch the CPU usage increase!)
-
-]
-
-*Adjusting units is left as an exercise for the reader.*
-
----
-
-## More resources on container metrics
-
-- [Prometheus, a Whirlwind Tour](https://speakerdeck.com/copyconstructor/prometheus-a-whirlwind-tour),
-  an original overview of Prometheus
-
-- [Docker Swarm & Container Overview](https://grafana.net/dashboards/609),
-  a custom dashboard for Grafana
-
-- [Gathering Container Metrics](http://jpetazzo.github.io/2013/10/08/docker-containers-metrics/),
-  a blog post about cgroups
-
-- [The Prometheus Time Series Database](https://www.youtube.com/watch?v=HbnGSNEjhUc),
-  a talk explaining why custom data storage is necessary for metrics
-
----
-
-# Dealing with stateful services
-
-- First of all, you need to make sure that the data files are on a *volume*
-
-- Volumes are host directories that are mounted to the container's filesystem
-
-- These host directories can be backed by the ordinary, plain host filesystem ...
-
-- ... Or by distributed/networked filesystems
-
-- In the latter scenario, in case of node failure, the data is safe elsewhere ...
-
-- ... And the container can be restarted on another node without data loss
-
----
-
-## Building a stateful service experiment
-
-- We will use Redis for this example
-
-- We will expose it on port 10000 to access it easily
-
-.exercise[
-
-- Start the Redis service:
-  ```bash
-  docker service create --name stateful -p 10000:6379 redis
-  ```
-
-- Check that we can connect to it:
-  ```bash
-  docker run --net host --rm redis redis-cli -p 10000 info server
-  ```
-
-]
-
----
-
-## Accessing our Redis service easily
-
-- Typing that whole command is going to be tedious
-
-.exercise[
-
-- Define a shell alias to make our lives easier:
-  ```bash
-  alias redis='docker run --net host --rm redis redis-cli -p 10000'
-  ```
-
-- Try it:
-  ```bash
-  redis info server
-  ```
-
-]
-
----
-
-## Basic Redis commands
-
-.exercise[
-
-- Check that the `foo` key doesn't exist:
-  ```bash
-  redis get foo
-  ```
-
-- Set it to `bar`:
-  ```bash
-  redis set foo bar
-  ```
-
-- Check that it exists now:
-  ```bash
-  redis get foo
-  ```
-
-]
-
----
-
-## Local volumes vs. global volumes
-
-- Global volumes exist in a single namespace
-
-- A global volume can be mounted on any node
-  <br/>.small[(bar some restrictions specific to the volume driver in use; e.g. using an EBS-backed volume on a GCE/EC2 mixed cluster)]
-
-- Attaching a global volume to a container allows to start the container anywhere
-  <br/>(and retain its data wherever you start it!)
-
-- Global volumes require extra *plugins* (Flocker, Portworx...)
-
-- Docker doesn't come with a default global volume driver at this point
-
-- Therefore, we will fall back on *local volumes*
-
----
-
-## Local volumes
-
-- We will use the default volume driver, `local`
-
-- As the name implies, the `local` volume driver manages *local* volumes
-
-- Since local volumes are (duh!) *local*, we need to pin our container to a specific host
-
-- We will do that with a *constraint*
-
-.exercise[
-
-- Add a placement constraint to our service:
-  ```bash
-  docker service update stateful --constraint-add node.hostname==$HOSTNAME
-  ```
-
-]
-
----
-
-## Where is our data?
-
-- If we look for our `foo` key, it's gone!
-
-.exercise[
-
-- Check the `foo` key:
-  ```bash
-  redis get foo
-  ```
-
-- Adding a constraint caused the service to be redeployed:
-  ```bash
-  docker service ps stateful
-  ```
-
-]
-
-Note: even if the constraint ends up being a no-op (i.e. not
-moving the service), the service gets redeployed.
-This ensures consistent behavior.
-
----
-
-## Setting the key again
-
-- Since our database was wiped out, let's populate it again
-
-.exercise[
-
-- Set `foo` again:
-  ```bash
-  redis set foo bar
-  ```
-
-- Check that it's there:
-  ```bash
-  redis get foo
-  ```
-
-]
-
----
-
-## Service updates cause containers to be replaced
-
-- Let's try to make a trivial update to the service and see what happens
-
-.exercise[
-
-- Set a memory limit to our Redis service:
-  ```bash
-  docker service update stateful --limit-memory 100M
-  ```
-
-- Try to get the `foo` key one more time:
-  ```bash
-  redis get foo
-  ```
-
-]
-
-The key is blank again!
-
----
-
-## Service volumes are ephemeral by default
-
-- Let's highlight what's going on with volumes!
-
-.exercise[
-
-- Check the current list of volumes:
-  ```bash
-  docker volume ls
-  ```
-
-- Carry a minor update to our Redis service:
-  ```bash
-  docker service update stateful --limit-memory 200M
-  ```
-
-]
-
-Again: all changes trigger the creation of a new task, and therefore a replacement of the existing container;
-even when it is not strictly technically necessary.
-
----
-
-## The data is gone again
-
-- What happened to our data?
-
-.exercise[
-
-- The list of volumes is slightly different:
-  ```bash
-  docker volume ls
-  ```
-
-]
-
-(You should see one extra volume.)
-
----
-
-## Assigning a persistent volume to the container
-
-- Let's add an explicit volume mount to our service, referencing a named volume
-
-.exercise[
-
-- Update the service with a volume mount:
-  ```bash
-    docker service update stateful \
-           --mount-add type=volume,source=foobarstore,target=/data
-  ```
-
-- Check the new volume list:
-  ```bash
-  docker volume ls
-  ```
-
-]
-
-Note: the `local` volume driver automatically creates volumes.
-
----
-
-## Checking that persistence actually works across service updates
-
-.exercise[
-
-- Store something in the `foo` key:
-  ```bash
-  redis set foo barbar
-  ```
-
-- Update the service with yet another trivial change:
-  ```bash
-  docker service update stateful --limit-memory 300M
-  ```
-
-- Check that `foo` is still set:
-  ```bash
-  redis get foo
-  ```
-
-]
-
----
-
-## Recap
-
-- The service must commit its state to disk when being shutdown.red[*]
-
-  (Shutdown = being sent a `TERM` signal)
-
-- The state must be written on files located on a volume
-
-- That volume must be specified to be persistent
-
-- If using a local volume, the service must also be pinned to a specific node
-
-  (And losing that node means losing the data, unless there are other backups)
-
-.footnote[<br/>
-.red[*]If you customize Redis configuration, make sure you
-persist data correctly!
-<br/>
-It's easy to make that mistake — __Trust me!__]
-
----
-
-## Cleaning up
-
-.exercise[
-
-- Remove the stateful service:
-  ```bash
-  docker service rm stateful
-  ```
-
-- Remove the associated volume:
-  ```bash
-  docker volume rm foobarstore
-  ```
-
-]
-
-Note: we could keep the volume around if we wanted.
-
----
-
-## Should I run stateful services in containers?
-
---
-
-Depending whom you ask, they'll tell you:
-
---
-
-- certainly not, heathen!
-
---
-
-- we've been running a few thousands PostgreSQL instances in containers ...
-  <br/>for a few years now ... in production ... is that bad?
-
---
-
-- what's a container?
-
---
-
-Perhaps a better question would be:
-
-*"Should I run stateful services?"*
-
---
-
-- is it critical for my business?
-- is it my value-add?
-- or should I find somebody else to run them for me?
-
----
-
-class: extra-details
-
-# Controlling Docker from a container
-
-- In a local environment, just bind-mount the Docker control socket:
-  ```bash
-  docker run -ti -v /var/run/docker.sock:/var/run/docker.sock docker
-  ```
-
-- Otherwise, you have to:
-
-  - set `DOCKER_HOST`,
-  - set `DOCKER_TLS_VERIFY` and `DOCKER_CERT_PATH` (if you use TLS),
-  - copy certificates to the container that will need API access.
-
-More resources on this topic:
-
-- [Do not use Docker-in-Docker for CI](
-  http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/)
-- [One container to rule them all](
-  http://jpetazzo.github.io/2016/04/03/one-container-to-rule-them-all/)
-
----
-
-class: extra-details
-
-## Bind-mounting the Docker control socket
-
-- In Swarm mode, bind-mounting the control socket gives you access to the whole cluster
-
-- You can tell Docker to place a given service on a manager node, using constraints:
-  ```bash
-    docker service create \
-      --mount source=/var/run/docker.sock,type=bind,target=/var/run/docker.sock \
-      --name autoscaler --constraint node.role==manager ...
-  ```
-
----
-
-class: extra-details
-
-## Constraints and global services
-
-(New in Docker Engine 1.13)
-
-- By default, global services run on *all* nodes
-  ```bash
-  docker service create --mode global ...
-  ```
-
-- You can specify constraints for global services
-
-- These services will run only on the node satisfying the constraints
-
-- For instance, this service will run on all manager nodes:
-  ```bash
-  docker service create --mode global --constraint node.role==manager ...
-  ```
-
----
-
-class: extra-details
-
-## Constraints and dynamic scheduling
-
-(New in Docker Engine 1.13)
-
-- If constraints change, services are started/stopped accordingly
-
-  (e.g., `--constraint node.role==manager` and nodes are promoted/demoted)
-
-- This is particularly useful with labels:
-  ```bash
-  docker node update node1 --label-add defcon=five
-  docker service create --constraint node.labels.defcon==five ...
-  docker node update node2 --label-add defcon=five
-  docker node update node1 --label-rm defcon=five
-  ```
-
----
-
-class: extra-details
-
-## Shortcomings of dynamic scheduling
-
-.warning[If a service becomes "unschedulable" (constraints can't be satisfied):]
-
-- It won't be scheduled automatically when constraints are satisfiable again
-
-- You will have to update the service; you can do a no-op udate with:
-  ```bash
-  docker service update ... --force
-  ```
-
-.warning[Docker will silently ignore attempts to remove a non-existent label or constraint]
-
-- It won't warn you if you typo when removing a label or constraint!
-
----
-
-class: extra-details
-
-# Node management
-
-- SwarmKit allows to change (almost?) everything on-the-fly
-
-- Nothing should require a global restart
-
----
-
-class: extra-details
-
-## Node availability
-
-```bash
-docker node update <node-name> --availability <active|pause|drain>
-```
-
-- Active = schedule tasks on this node (default)
-
-- Pause = don't schedule new tasks on this node; existing tasks are not affected
-
-  You can use it to troubleshoot a node without disrupting existing tasks
-
-  It can also be used (in conjunction with labels) to reserve resources
-
-- Drain = don't schedule new tasks on this node; existing tasks are moved away
-
-  This is just like crashing the node, but containers get a chance to shutdown cleanly
-
----
-
-class: extra-details
-
-## Managers and workers
-
-- Nodes can be promoted to manager with `docker node promote`
-
-- Nodes can be demoted to worker with `docker node demote`
-
-- This can also be done with `docker node update <node> --role <manager|worker>`
-
-- Reminder: this has to be done from a manager node
-  <br/>(workers cannot promote themselves)
-
----
-
-class: extra-details
-
-## Removing nodes
-
-- You can leave Swarm mode with `docker swarm leave`
-
-- Nodes are drained before being removed (i.e. all tasks are rescheduled somewhere else)
-
-- Managers cannot leave (they have to be demoted first)
-
-- After leaving, a node still shows up in `docker node ls` (in `Down` state)
-
-- When a node is `Down`, you can remove it with `docker node rm` (from a manager node)
-
----
-
-class: extra-details
-
-## Join tokens and automation
-
-- If you have used Docker 1.12-RC: join tokens are now mandatory!
-
-- You cannot specify your own token (SwarmKit generates it)
-
-- If you need to change the token: `docker swarm join-token --rotate ...`
-
-- To automate cluster deployment:
-
-  - have a seed node do `docker swarm init` if it's not already in Swarm mode
-
-  - propagate the token to the other nodes (secure bucket, facter, ohai...)
-
----
-
-class: extra-details
-
-## Disk space management: `docker system df`
-
-- Shows disk usage for images, containers, and volumes
-
-- Breaks down between *active* and *reclaimable* categories
-
-.exercise[
-
-- Check how much disk space is used at the end of the workshop:
-  ```bash
-  docker system df
-  ```
-
-]
-
-Note: `docker system` is new in Docker Engine 1.13.
-
----
-
-class: extra-details
-
-## Reclaiming unused resources: `docker system prune`
-
-- Removes stopped containers
-
-- Removes dangling images (that don't have a tag associated anymore)
-
-- Removes orphaned volumes
-
-- Removes empty networks
-
-.exercise[
-
-- Try it:
-  ```bash
-  docker system prune -f
-  ```
-
-]
-
-Note: `docker system prune -a` will also remove *unused* images.
-
----
-
-class: extra-details
-
-## Events
-
-- You can get a real-time stream of events with `docker events`
-
-- This will report *local events* and *cluster events*
-
-- Local events =
-  <br/>
-  all activity related to containers, images, plugins, volumes, networks, *on this node*
-
-- Cluster events =
-  <br/>Swarm Mode activity related to services, nodes, secrets, configs, *on the whole cluster*
-
-- `docker events` doesn't report *local events happening on other nodes*
-
-- Events can be filtered (by type, target, labels...)
-
-- Events can be formatted with Go's `text/template` or in JSON
-
----
-
-class: extra-details
-
-## Getting *all the events*
-
-- There is no built-in to get a stream of *all the events* on *all the nodes*
-
-- This can be achieved with (for instance) the four following services working together:
-
-  - a Redis container (used as a stateless, fan-in message queue)
-
-  - a global service bind-mounting the Docker socket, pushing local events to the queue
-
-  - a similar singleton service to push global events to the queue
-
-  - a queue consumer fetching events and processing them as you please
-
-I'm not saying that you should implement it with Shell scripts, but you totally could.
-
-.small[
-(It might or might not be one of the initiating rites of the
-[House of Bash](https://twitter.com/carmatrocity/status/676559402787282944))
-]
-
-For more information about event filters and types, check [the documentation](https://docs.docker.com/engine/reference/commandline/events/).
-
----
-
-
-class: title, extra-details
-
-# What's next?
-
-## (What to expect in future versions of this workshop)
-
----
-
-class: extra-details
-
-## Implemented and stable, but out of scope
-
-- [Docker Content Trust](https://docs.docker.com/engine/security/trust/content_trust/) and
-  [Notary](https://github.com/docker/notary) (image signature and verification)
-
-- Image security scanning (many products available, Docker Inc. and 3rd party)
-
-- [Docker Cloud](https://cloud.docker.com/) and
-  [Docker Datacenter](https://www.docker.com/products/docker-datacenter)
-  (commercial offering with node management, secure registry, CI/CD pipelines, all the bells and whistles)
-
-- Network and storage plugins
-
----
-
-class: extra-details
-
-## Work in progress
-
-- Demo at least one volume plugin
-  <br/>(bonus points if it's a distributed storage system)
-
-- ..................................... (your favorite feature here)
-
-Reminder: there is a tag for each iteration of the content
-in the Github repository.
-
-It makes it easy to come back later and check what has changed since you did it!
-
----
-
-class: title, self-paced
-
-Thank you!
-
----
-
-class: title, in-person
-
-That's all folks! <br/> Questions?
-
-.small[.small[
-
-Jérôme ([@jpetazzo](https://twitter.com/jpetazzo)) — [@docker](https://twitter.com/docker)
-
-AJ ([@s0ulshake](https://twitter.com/s0ulshake)) — *For hire!*
-<br/>
-`curl cv.soulshake.net`
-
-]]
-
-<!--
-<br/> Jérôme ([@jpetazzo](https://twitter.com/jpetazzo)) <br/> Tiffany ([@tiffanyfayj](https://twitter.com/tiffanyfayj))
--->
-
-    </textarea>
-    <script src="remark.min.js" type="text/javascript">
-    </script>
-    <script type="text/javascript">
-      var slideshow = remark.create({
-        ratio: '16:9',
-        highlightSpans: true,
-        excludedClasses: ["in-person", "elk-auto", "prom-auto"]
-        //excludedClasses: ["self-paced", "extra-details", "advertise-addr", "docker-machine", "netshoot", "sbt", "ipsec", "node-info", "swarmtools", "secrets", "encryption-at-rest", "elk-manual", "snap", "prom-manual"]
-      });
-    </script>
-  </body>
-</html>
