class: title # Docker <br/> Orchestration <br/> Workshop --- ## Logistics - Hello! We're `jerome at docker dot com` and `aj at soulshake dot net` <!-- Reminder, when updating the agenda: when people are told to show up at 9am, they usually trickle in until 9:30am (except for paid training sessions). If you're not sure that people will be there on time, it's a good idea to have a breakfast with the attendees at e.g. 9am, and start at 9:30. --> - Agenda: .small[ - 08:00-09:00 hello and breakfast - 09:00:10:25 part 1 - 10:25-10:35 coffee break - 10:35-12:00 part 2 - 12:00-13:00 lunch break - 13:00-14:25 part 3 - 14:25-14:35 coffee break - 14:35-16:00 part 4 ] <!-- - This will be FAST PACED, but DON'T PANIC! --> - All the content is publicly available (slides, code samples, scripts) <!-- Remember to change: - the Gitter link below - the "tweet my speed" hashtag in DockerCoins HTML --> - Experimental chat support on [Gitter](http://container.training/chat) --- <!-- grep '^# ' index.html | grep -v '<br' | tr '#' '-'^C --> ## Outline (1/4) - Pre-requirements - VM environment - Our sample application - Running the application - Container port mapping - Identifying bottlenecks - Scaling HTTP on a single node - Put a load balancer on it - Connecting to containers on other hosts - Abstracting remote services with ambassadors - Various considerations about ambassadors --- ## Outline (2/4) - Docker for ops - Backups - Docker events stream - Attaching labels - Logs - Storing container logs in an ELK stack - Security upgrades - Network traffic analysis --- ## Outline (3/4) - Dynamic orchestration - Hands-on Swarm - Deploying Swarm - Cluster discovery - Building our app on Swarm - Connecting containers with ambassadors - Setting up Consul and overlay networks - Multi-host networking - Using overlay networks with Compose - Load balancing with overlay networks --- ## Outline (4/4) - Here be dragons - Distributing Machine credentials - Highly available Swarm managers - Highly available containers - Conclusions --- # Pre-requirements - Computer with network connection and SSH client - on Linux, OS X, FreeBSD... you are probably all set - on Windows, get [putty](http://www.putty.org/), [Git BASH](https://msysgit.github.io/), or [MobaXterm](http://mobaxterm.mobatek.net/) - Basic Docker knowledge <br/>(but that's OK if you're not a Docker expert!) --- ## Nice-to-haves - [GitHub](https://github.com/join) account <br/>(if you want to fork the repo; also used to join Gitter) - [Gitter](https://gitter.im/) account <br/>(to join the conversation during the workshop) - [Docker Hub](https://hub.docker.com) account <br/>(it's one way to distribute images on your Swarm cluster) --- ## Hands-on sections - The whole workshop is hands-on - I will show Docker in action - I invite you to reproduce what I do - All hands-on sections are clearly identified, like the gray rectangle below .exercise[ - This is the stuff you're supposed to do! - Go to [container.training](http://container.training/) to view these slides - Join the chat room on [Gitter](http://container.training/chat) ] --- # VM environment - Each person gets 5 private VMs (not shared with anybody else) - They'll be up until tomorrow - You have a little card with login+password+IP addresses - You can automatically SSH from one VM to another .exercise[ <!-- ```bash for N in $(seq 1 5); do ssh -o StrictHostKeyChecking=no node$N true done for N in $(seq 1 5); do (. docker-machine rm -f node$N ssh node$N "docker ps -aq | xargs -r docker rm -f" ssh node$N sudo rm -f /etc/systemd/system/docker.service ssh node$N sudo systemctl daemon-reload echo Restarting node$N. ssh node$N sudo systemctl restart docker echo Restarted node$N. ) & done wait ``` --> - Log into the first VM (`node1`) - Check that you can SSH (without password) to `node2`: ```bash ssh node2 ``` - Type `exit` or `^D` to come back to node1 <!-- ```meta ^D ``` --> ] --- ## We will (mostly) interact with node1 only - Unless instructed, **all commands must be run from the first VM, `node1`** - We will only checkout/copy the code on `node1` - When we will use the other nodes, we will do it mostly through the Docker API - We will use SSH only for a few "out of band" operations (mass-removing containers...) --- ## Terminals Once in a while, the instructions will say: <br/>"Open a new terminal." There are multiple ways to do this: - create a new window or tab on your machine, and SSH into the VM; - use screen or tmux on the VM and open a new window from there. You are welcome to use the method that you feel the most comfortable with. --- ## Tmux cheatsheet - Ctrl-b c → creates a new window - Ctrl-b n → go to next window - Ctrl-b p → go to previous window - Ctrl-b " → split window top/bottom - Ctrl-b % → split window left/right - Ctrl-b Alt-1 → rearrange windows in columns - Ctrl-b Alt-2 → rearrange windows in rows - Ctrl-b arrows → navigate to other windows - Ctrl-b d → detach session - tmux attach → reattach to session --- ## Brand new versions! - Engine 1.12 - Compose 1.7 - Swarm 1.2 - Machine 0.6 .exercise[ - Check all installed versions: ```bash docker version docker-compose -v docker run --rm swarm -version docker-machine -v ``` ] --- ## Why are we not using the latest version of Machine? - The latest version of Machine is 0.7 - The way it deploys Swarm is different from 0.6 - This causes a regression in the strategy that we will use later - More details later! --- # Our sample application - Visit the GitHub repository with all the materials of this workshop: <br/>https://github.com/jpetazzo/orchestration-workshop - The application is in the [dockercoins]( https://github.com/jpetazzo/orchestration-workshop/tree/master/dockercoins) subdirectory - Let's look at the general layout of the source code: there is a Compose file [docker-compose.yml]( https://github.com/jpetazzo/orchestration-workshop/blob/master/dockercoins/docker-compose.yml) ... ... and 4 other services, each in its own directory: - `rng` = web service generating random bytes - `hasher` = web service computing hash of POSTed data - `worker` = background process using `rng` and `hasher` - `webui` = web interface to watch progress --- ## What's this application? --- class: pic  (DockerCoins logo courtesy of @jonasrosland. Thanks!) --- ## What's this application? - It is a DockerCoin miner! 💰🐳📦🚢 - No, you can't buy coffee with DockerCoins - How DockerCoins works: - `worker` asks to `rng` to give it random bytes - `worker` feeds those random bytes into `hasher` - each hash starting with `0` is a DockerCoin - DockerCoins are stored in `redis` - `redis` is also updated every second to track speed - you can see the progress with the `webui` --- ## Getting the application source code - We will clone the GitHub repository - The repository also contains scripts and tools that we will use through the workshop .exercise[ <!-- ```bash [ -d orchestration-workshop ] && mv orchestration-workshop orchestration-workshop.$$ ``` --> - Clone the repository on `node1`: ```bash git clone git://github.com/jpetazzo/orchestration-workshop ``` ] (You can also fork the repository on GitHub and clone your fork if you prefer that.) --- # Running the application Without further ado, let's start our application. .exercise[ - Go to the `dockercoins` directory, in the cloned repo: ```bash cd ~/orchestration-workshop/dockercoins ``` - Use Compose to build and run all containers: ```bash docker-compose up ``` ] Compose tells Docker to build all container images (pulling the corresponding base images), then starts all containers, and displays aggregated logs. --- ## Lots of logs - The application continuously generates logs - We can see the `worker` service making requests to `rng` and `hasher` - Let's put that in the background .exercise[ - Stop the application by hitting `^C` <!-- ```meta ^C ``` --> ] - `^C` stops all containers by sending them the `TERM` signal - Some containers exit immediately, others take longer <br/>(because they don't handle `SIGTERM` and end up being killed after a 10s timeout) --- ## Restarting in the background - Many flags and commands of Compose are modeled after those of `docker` .exercise[ - Start the app in the background with the `-d` option: ```bash docker-compose up -d ``` - Check that our app is running with the `ps` command: ```bash docker-compose ps ``` ] `docker-compose ps` also shows the ports exposed by the application. --- ## Connecting to the web UI - The `webui` container exposes a web dashboard; let's view it .exercise[ - Open http://[yourVMaddr]:8000/ (from a browser) ] - The app actually has a constant, steady speed (3.33 coins/second) - The speed seems not-so-steady because: - the worker doesn't update the counter after every loop, but up to once per second - the speed is computed by the browser, checking the counter about once per second - between two consecutive updates, the counter will increase either by 4, or by 0 --- ## Viewing logs - The `docker-compose logs` command works like `docker logs` .exercise[ - View all logs since container creation and exit when done: ```bash docker-compose logs ``` - Stream container logs, starting at the last 10 lines for each container: ```bash docker-compose logs --tail 10 --follow ``` <!-- ```meta ^C ``` --> ] Tip: use `^S` and `^Q` to pause/resume log output. --- ## Upgrading from Compose 1.6 .warning[The `logs` command has changed between Compose 1.6 and 1.7!] - Up to 1.6 - `docker-compose logs` is the equivalent of `logs --follow` - `docker-compose logs` must be restarted if containers are added - Since 1.7 - `--follow` must be specified explicitly - new containers are automatically picked up by `docker-compose logs` --- ## Compose file format version *Particularly relevant if you have used Compose before...* - Compose 1.6 introduced support for a new Compose file format (aka "v2") - Services are no longer at the top level, but under a `services` section - There has to be a `version` key at the top level, with value `"2"` (as a string, not an integer) - Containers are placed on a dedicated network, making links unnecessary - There are other minor differences, but upgrade is easy and straightforward --- ## Links, naming, and service discovery - Containers can have network aliases (resolvable through DNS) - Compose file version 2 makes each container reachable through its service name - Compose file version 1 requires "links" sections - Our code can connect to services using their short name (instead of e.g. IP address or FQDN) --- ## Example in `worker/worker.py`  --- ## Testing services in isolation - We will stop the `worker` service, and test `rng` and `hasher` alone .exercise[ - Stop the `worker` service: ```bash docker-compose stop worker ``` - Look at the logs of `rng` and `hasher`: ```bash docker-compose logs --tail 10 --follow ``` <!-- ```meta ^Z ``` --> ] We will let Compose stream the logs, and open a new terminal. --- ## Compose file location - What happens if you try to run Compose in `$HOME`? -- - Compose complains that it cannot find `docker-compose.yml` - You need to go to the directory containing `docker-compose.yml` <br/>...or use the `-f` option to give the path to the Compose file <br/>...or set the `COMPOSE_FILE` environment variable .exercise[ - Go to the application directory: ```bash cd ~/orchestration-workshop/dockercoins ``` ] --- ## Locating port numbers for `rng` and `hasher` - We can see the port mapping for our services with: ```bash docker-compose ps ``` - Both services run on port 80 *in their respective container* - On the Docker host, they are mapped to ports 8001 and 8002 - They are not exposed on port 80, because the Docker host has only one port 80 - The mapping is done in `docker-compose.yml` --- # Container port mapping - `node1`, the Docker host, has only one port 80 - If we give the one and only port 80 to the first container who asks for it, we are in trouble when another container needs it - Default behavior: containers are not "exposed" <br/>(only reachable by the Docker host and other containers, through their private address) - Container network services can be exposed: - statically (you decide which host port to use) - dynamically (Docker allocates a host port) --- ## Declaring port mapping - Directly with the Docker Engine: ```bash docker run -d -p 8000:80 nginx docker run -d -p 80 nginx docker run -d -P nginx ``` - With Docker Compose, in the `docker-compose.yml` file: ```yaml rng: … ports: - "8001:80" ``` → port 8001 *on the host* maps to port 80 *in the container* --- ## Testing the `rng` service Let's get random bytes of data! .exercise[ - Check that the service is alive: <br/>`curl localhost:8001` - Get 10 bytes of random data: <br/>`curl localhost:8001/10` ] If the binary data output messed up your terminal, fix it with `reset`. --- ## Testing the `hasher` service .exercise[ - Check that the `hasher` service is alive: ```bash curl localhost:8002 ``` - Posting binary data requires some extra flags: ```bash curl -H "Content-type: application/octet-stream" \ --data-binary hello localhost:8002 ``` ] --- ## Verifying the computed hash - This web service is essentially implementing `sha256sum` over HTTP .exercise[ - Check that `sha256sum` yields the same result: ```bash echo -n hello | sha256sum ``` ] - We specify `-n` to prevent `echo` from adding a trailing `\n` <br/>(which would change the SHA digest) --- ## Checking logs - The tests that we made should show up in the first window <br/>(where `docker-compose logs` is still running) - We can now restart the `worker` service .exercise[ - Start the `worker` container: ```bash docker-compose start worker ``` ] In the web UI, the graph should go up again. How can we get that graph to go further up? --- ## Looking at resource usage - Let's look at CPU, memory, and I/O usage .exercise[ - run `top` to see CPU and memory usage (you should see idle cycles) - run `vmstat 3` to see I/O usage (si/so/bi/bo) <br/>(the 4 numbers should be almost zero, except `bo` for logging) ] We have available resources. - Why? - How can we use them? --- ## Scaling workers on a single node - Docker Compose supports scaling.red[*] - Let's scale `worker` and see what happens! .exercise[ - Start 9 more `worker` containers: ```bash docker-compose scale worker=10 ``` - Check the aggregated logs of those containers - See the impact on CPU load (with top/htop), and on compute speed (with web UI) <!-- ```bash sleep 5 killall docker-compose ``` --> ] .red[*]With some limitations, as we'll see later. --- # Identifying bottlenecks - You should have seen a 3x speed bump (not 10x) - Adding workers didn't result in linear improvement - *Something else* is slowing us down -- - ... But what? -- - The code doesn't have instrumentation - Let's use state-of-the-art HTTP performance analysis! <br/>(i.e. good old tools like `ab`, `httping`...) --- ## Measuring latency under load We will use `httping`. .exercise[ - Check the latency of `rng`: ```bash httping -c 10 localhost:8001 ``` - Check the latency of `hasher`: ```bash httping -c 10 localhost:8002 ``` ] `rng` has a much higher latency than `hasher`. --- ## Benchmarking in isolation We will now use `ab`. .exercise[ - Stop the `worker` containers: ```bash docker-compose kill worker ``` ] --- ## Benchmarking `rng` We will send 50 requests, but with various levels of concurrency. .exercise[ - Send 50 requests, with a single sequential client: ```bash ab -c 1 -n 50 localhost:8001/10 ``` - Send 50 requests, with ten parallel clients: ```bash ab -c 10 -n 50 localhost:8001/10 ``` ] --- ## Benchmark results for `rng` - In both cases, the benchmark takes ~5 seconds to complete - When serving requests sequentially, they each take 100ms - In the parallel scenario, the latency increased dramatically: - one request is served in 100ms - another is served in 200ms - another is served in 300ms - ... - another is served in 1000ms - What about `hasher`? --- ## Benchmarking `hasher` We will do the same tests for `hasher`. The command is slightly more complex, since we need to post random data. First, we need to put the POST payload in a temporary file. .exercise[ - Generate 10 bytes of random data: ```bash curl localhost:8001/10 >/tmp/random ``` ] --- ## Benchmarking `hasher` Once again, we will send 50 requests, with different levels of concurrency. .exercise[ - Send 50 requests with a sequential client: ```bash ab -c 1 -n 50 -T application/octet-stream \ -p /tmp/random localhost:8002/ ``` - Send 50 requests with 10 parallel clients: ```bash ab -c 10 -n 50 -T application/octet-stream \ -p /tmp/random localhost:8002/ ``` ] --- ## Benchmark results for `hasher` - The sequential benchmarks takes ~5 seconds to complete - The parallel benchmark takes less than 1 second to complete - In both cases, each request takes a bit more than 100ms to complete - Requests are a bit slower in the parallel benchmark - It looks like `hasher` is better equiped to deal with concurrency than `rng` --- class: title Why? --- ## Why does everything take (at least) 100ms? -- `rng` code:  -- `hasher` code:  --- class: title But ... WHY?!? --- ## Why did we sprinkle this sample app with sleeps? - Deterministic performance <br/>(regardless of instance speed, CPUs, I/O...) -- - Actual code sleeps all the time anyway -- - When your code makes a remote API call: - it sends a request; - it sleeps until it gets the response; - it processes the response. --- ## Why do `rng` and `hasher` behave differently?  -- (Synchronous vs. asynchronous event processing) --- ## How to make `rng` go faster - Obvious solution: comment out the `sleep` instruction -- - Unfortunately, in the real world, network latency exists -- - More realistic solution: use an asynchronous framework <br/>(e.g. use gunicorn with gevent) -- - New rule: we can't change the code! -- - Solution: scale out `rng` <br/>(dispatch `rng` requests on multiple instances) --- # Scaling HTTP on a single node - We can try to scale with Compose: ```bash docker-compose scale rng=3 ``` - This will result into an error, because our Compose file uses an explicit port - We cannot have multiple containers bound to the same port (here, 8001) - Compose *tries* to scale anyway (because on a cluster, you *can* have multiple containers on the same public port) - Let's remove the explicit port mapping, and see what happens when we scale! --- ## Removing explicit port from the Compose file - We will replace `8001:80` with just `80` - This will continue to make the service publicly available, <br/>but on a port dynamically allocated by Docker .exercise[ - Edit the `docker-compose.yml` file to change the ports section for `rng`: ```yaml rng: ... ports: - "80" ``` <!-- ```edit cp docker-compose.yml-scaled docker-compose.yml ``` --> ] Shortcut: `docker-compose.yml-portmap` --- ## Scaling up a network service with Compose - We changed the definition of the service (by removing the static port mapping), <br/>so we must execute `docker-compose up` before scaling .exercise[ - Refresh the `rng` service: ```bash docker-compose up -d ``` - Scale the `rng` service: ```bash docker-compose scale rng=3 ``` ] --- ## Results - In the web UI, you might see a performance increase ... or maybe not -- - Since Engine 1.11, we get round-robin DNS records (i.e. resolving `rng` will yield the IP addresses of all 3 containers) - Docker randomizes the records it sends - But many resolvers will sort them in unexpected ways - Depending on various factors, you could get: - all traffic on a single container - traffic perfectly balanced on all containers - traffic unevenly balanced across containers --- ## Assessing DNS randomness - Let's see how our containers resolve DNS requests .exercise[ - On each of our 10 scaled workers, execute 5 ping requests: ```bash for N in $(seq 1 10); do echo PING__________$N for I in $(seq 1 5); do docker exec -ti dockercoins_worker_$N ping -c1 rng done done | grep PING ``` ] (The 7th Might Surprise You!) --- ## DNS randomness - Other programs can yield different results - Same program on another distro can yield different results - Same source code with another libc or resolver can yield different results - Running the same test at different times can yield different results - Did I mention that Your Results May Vary? --- ## Implementing fair load balancing - Instead of relying on DNS round robin, let's use a proper load balancer - Ceate multiple identical `rng` services - Put a load balancer in front of them - Point other services to the load balancer --- ## Scaling `rng` .exercise[ - Replace the `rng` service with multiple copies of it: ```yaml rng1: build: rng rng2: build: rng rng3: build: rng ``` ] That's all! Shortcut: `docker-compose.yml-scaled-rng` --- ## Introduction to `jpetazzo/hamba` - General purpose load balancer and traffic director - [Source code is available on GitHub]( https://github.com/jpetazzo/hamba) - [Public image is available on the Docker Hub]( https://hub.docker.com/r/jpetazzo/hamba/) - Generates a configuration file for HAProxy, then starts HAProxy - Parameters are provided on the command line; for instance: ```bash docker run -d -p 80 jpetazzo/hamba 80 www1:1234 www2:2345 docker run -d -p 80 jpetazzo/hamba 80 www1 1234 www2 2345 ``` Those two commands do the same thing: they start a load balancer listening on port 80, and balancing traffic across www1:1234 and www2:2345 --- # Put a load balancer on it Let's add our load balancer to the Compose file. .exercise[ - Add the following section to the Compose file: ```yaml rng: image: jpetazzo/hamba command: 80 rng1 80 rng2 80 rng3 80 ports: - "8001:80" ``` <!-- ```edit cp docker-compose.yml-scaled-rng docker-compose.yml ``` --> ] Shortcut: `docker-compose.yml-scaled-rng` --- ## Start the whole stack .exercise[ - Scale back `rng` to a single instance: <br/>`docker-compose scale rng=1` - Start the new services: <br/>`docker-compose up -d` - Check worker logs: <br/>`docker-compose logs worker` - Check load balancer logs: <br/>`docker-compose logs rng` ] .warning[If you get errors, see next slide.] --- ## Recovering from errors - If you scale a service that has an explicit port mapping, you can end up with extra containers that are *created*, but not *started*, and will tie up resources - Those containers can prevent other containers from starting - If that happens, just remove those containers; for instance: ```bash docker-compose kill rng && docker-compose rm -f rng ``` - After removing the containers, you can `docker-compose up -d` again and everything should work fine. --- ## Results - Check the latency of `rng` <br/>(it should have improved significantly!) - Check the application performance in the web UI <br/>(it should improve if you have enough workers) --- ## The good, the bad, the ugly -- - The good We scaled a service, added a load balancer - without changing a single line of code. -- - The bad We manually copy-pasted sections in `docker-compose.yml`. Improvement: write scripts to transform the YAML file. -- - The ugly If we scale up/down, we have to restart everything. Improvement: reconfigure the load balancer dynamically. --- ## Cleaning up - Before moving forward, let's clean up all those containers .exercise[ - Use the `down` command to stop and remove containers: ```bash docker-compose down ``` ] --- # Connecting to containers on other hosts - So far, our whole stack is on a single machine - We want to scale out (across multiple nodes) - We will deploy the same stack multiple times - But we want every stack to use the same Redis <br/>(in other words: Redis is our only *stateful* service here) -- - And remember: we're not allowed to change the code! - the code connects to host `redis` - `redis` must resolve to the address of our Redis service - the Redis service must listen on the default port (6379) --- # Using custom DNS mapping - We could setup a Redis server on its default port - And add a DNS entry mapping `redis` to this server .exercise[ - See what happens if we run: ```bash docker run --add-host redis:1.2.3.4 alpine ping redis ``` <!-- ```meta ^C ``` --> ] There is a Compose file option for that: `extra_hosts`. --- # Abstracting remote services with ambassadors - What if we can't/won't run Redis on its default port? - What if we want to be able to move it easily? -- - We will use an ambassador - Redis will be started independently of our stack - It will run at an arbitrary location (host+port) - In our stack, we replace `redis` with an ambassador - The ambassador will connect to Redis - The ambassador will "act as" Redis in the stack --- class: pic  --- class: pic  --- ## Start redis - Start a standalone Redis container - Let Docker expose it on a random port .exercise[ - Run redis with a random public port: <br/>`docker run -d -P --name myredis redis` - Check which port was allocated: <br/>`docker port myredis 6379` ] - Note the IP address of the machine, and this port --- ## Update `docker-compose.yml` .exercise[ - Replace `redis` with an ambassador using `jpetazzo/hamba`: ```yaml redis: image: jpetazzo/hamba command: 6379 `AA.BB.CC.DD:EEEEE` ``` <!-- ```edit cat docker-compose.yml-ambassador | sed "s/AA.BB.CC.DD/$(curl myip.enix.org/REMOTE_ADDR)/" | sed "s/EEEEE/$(docker port myredis 6379 | cut -d: -f2)/" > docker-compose.yml ``` --> ] Shortcut: `docker-compose.yml-ambassador` <br/>(But you still have to update `AA.BB.CC.DD:EEEEE`!) --- ## Start the stack on the first machine - Compose will detect the change in the `redis` service - It will replace `redis` with a `jpetazzo/hamba` instance .exercise[ - Just tell Compose to do its thing: <br/>`docker-compose up -d` - Check that the stack is up and running: <br/>`docker-compose ps` - Look at the web UI to make sure that it works fine ] --- ## Controlling other Docker Engines - Many tools in the ecosystem will honor the `DOCKER_HOST` environment variable - Those tools include (obviously!) the Docker CLI and Docker Compose - Our training VMs have been setup to accept API requests on port 55555 <br/>(without authentication - this is very insecure, by the way!) - We will see later how to setup mutual authentication with certificates --- ## Setting the `DOCKER_HOST` environment variable .exercise[ - Check how many containers are running on `node1`: ```bash docker ps ``` - Set the `DOCKER_HOST` variable to control `node2`, and compare: ```bash export DOCKER_HOST=tcp://node2:55555 docker ps ``` ] You shouldn't see any container running on `node2` at this point. --- ## Start the stack on another machine - We will tell Compose to bring up our stack on the other node - It will use the local code (we don't need to checkout the code on `node2`) .exercise[ - Start the stack: ```bash docker-compose up -d ``` - Check that it's running: ```bash docker-compose ps ``` ] --- ## Run the application on every node - We will repeat the previous step with a little shell loop ... but introduce parallelism to save some time .exercise[ - Deploy one instance of the stack on each node: ```bash for N in 3 4 5; do DOCKER_HOST=tcp://node$N:55555 docker-compose up -d & done wait ``` ] Note: building the stack everywhere is not optimal. We will see later how to build once, and deploy the same build everywhere. --- ## Scale! - The app is built (and running!) everywhere - Scaling can be done very quickly .exercise[ - Add a bunch of workers all over the place: ```bash for N in 1 2 3 4 5; do DOCKER_HOST=tcp://node$N:55555 docker-compose scale worker=10 done ``` - Admire the result in the web UI! ] --- ## Social Media Moment Let's celebrate our success! (And the fact that we're just 2498349893849283948982 DockerCoins away from being able to afford a cup of coffee!) .exercise[ - If you have a Twitter account, tweet your mining speed! </br>(use the "Tweet this!" link below the graph☺) ] --- ## A few words about development volumes - Try to access the web UI on another node -- - It doesn't work! Why? -- - Static assets are masked by an empty volume -- - We need to comment out the `volumes` section --- ## Why must we comment out the `volumes` section? - Volumes have multiple uses: - storing persistent stuff (database files...) - sharing files between containers (logs, configuration...) - sharing files between host and containers (source...) - The `volumes` directive expands to an host path: `/home/docker/orchestration-workshop/dockercoins/webui/files` - This host path exists on the local machine (not on the others) - This specific volume is used in development (not in production) --- ## Stop the app (but leave Redis running) - Let's use `docker-compose down` - It will stop and remove the DockerCoins app (but leave other containers running) .exercise[ - We can do another simple parallel shell loop: ```bash for N in $(seq 1 5); do export DOCKER_HOST=tcp://node$N:55555 docker-compose down & done wait ``` ] (We need to keep the `myredis` container for our next section, which will be about backups!) --- # Various considerations about ambassadors - "But, ambassadors are adding an extra hop!" -- - Yes, but if you need load balancing, you need that hop - Ambassadors actually *save* one hop <br/>(they act as local load balancers) - traditional load balancer: <br/>client ⇒ external LB ⇒ server (2 physical hops) - ambassadors: <br/>client → ambassador ⇒ server (1 physical hop) -- - Ambassadors are more reliable than traditional LBs <br/>(they are colocated with their clients) --- ## Inconvenients of ambassadors - Generic issues <br/>(shared with any kind of load balancing / HA setup) - extra logical hop (not transparent to the client) - must assess backend health - one more thing to worry about (!) - Specific issues - load balancing fairness High-end load balancing solutions will rely on back pressure from the backends. This addresses the fairness issue. --- ## There are many ways to deploy ambassadors "Ambassador" is a design pattern. There are many ways to implement it. We will present three increasingly complex (but also powerful) ways to deploy ambassadors. --- ## Single-tier ambassador deployment - One-shot configuration process - Must be executed manually after each scaling operation - Scans current state, updates load balancer configuration - Pros: <br/>- simple, robust, no extra moving part <br/>- easy to customize (thanks to simple design) <br/>- can deal efficiently with large changes - Cons: <br/>- must be executed after each scaling operation <br/>- harder to compose different strategies - Example: this workshop --- ## Two-tier ambassador deployment - Daemon listens to Docker events API - Reacts to container start/stop events - Adds/removes back-ends to load balancers configuration - Pros: <br/>- no extra step required when scaling up/down - Cons: <br/>- extra process to run and maintain <br/>- deals with one event at a time (ordering matters) - Hidden gotcha: load balancer creation - Example: interlock --- ## Three-tier ambassador deployment - Daemon listens to Docker events API - Reacts to container start/stop events - Adds/removes scaled services in distributed config DB (zookeeper, etcd, consul…) - Another daemon listens to config DB events, <br/>adds/removes backends to load balancers configuration - Pros: <br/>- more flexibility - Cons: <br/>- three extra services to run and maintain - Example: registrator --- ## Ambassadors and overlay networks - Overlay networks allow direct multi-host communication - Ambassadors are still useful to implement other tasks: - load balancing; - credentials injection; - instrumentation; - fail-over; - etc. --- class: title # Interlude <br/> # Docker for ops --- # Backups - Redis is still running (with name `myredis`) - We want to enable backups without touching it - We will use a special backup container: - sharing the same volumes - linked to it (to connect to it easily) - possibly containing our backup tools - This works because the `redis` container image stores its data on a volume --- ## Starting the backup container .exercise[ - Make sure you're talking to the initial host: ```bash unset DOCKER_HOST ``` <!-- ```meta ^{ ``` --> - Start the container: ```bash docker run --link myredis:redis \ --volumes-from myredis \ -v /tmp/myredis:/output \ -ti alpine sh ``` - Look in `/data` in the container (that's where Redis puts its data dumps) ] --- ## Connecting to Redis - We need to tell Redis to perform a data dump *now* .exercise[ - Connect to Redis: ```bash telnet redis 6379 ``` - Issue commands `SAVE` then `QUIT` - Look at `/data` again (notice the time stamps) ] - There should be a recent dump file now! --- ## Getting the dump out of the container - We could use many things: - s3cmd to copy to S3 - SSH to copy to a remote host - gzip/bzip/etc before copying - We'll just copy it to the Docker host .exercise[ - Copy the file from `/data` to `/output` - Exit the container - Look into `/tmp/myredis` (on the host) <!-- ```meta ^} ``` --> ] --- ## Scheduling backups In the "old world," we (generally) use cron. With containers, what are our options? -- - run `cron` on the Docker host, and put `docker run` in the crontab -- - run `cron` in the backup container, and make sure it keeps running <br/>(e.g. with `docker run --restart=…`) -- - run `cron` in a container, and start backup containers from there -- - listen to the Docker events stream, automatically scheduling backups <br/>when database containers are started --- # Starting more containers from your container - In a local environment, just bind-mount the Docker control socket: ```bash docker run -ti -v /var/run/docker.sock:/var/run/docker.sock docker ``` - Otherwise, you have to: - set `DOCKER_HOST`, - set `DOCKER_TLS_VERIFY` and `DOCKER_CERT_PATH` (if you use TLS), - copy certificates to the container that will need API access. More resources on this topic: - [Do not use Docker-in-Docker for CI]( http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/) - [One container to rule them all]( http://jpetazzo.github.io/2016/04/03/one-container-to-rule-them-all/) --- # Docker events stream - Using the Docker API, we can get real-time notifications of everything happening in the Engine: - container creation/destruction - container start/stop - container exit/signal/out of memory - container attach/detach - volume creation/destruction - network creation/destruction - connection/disconnection of containers (Networks will be covered a bit later!) --- ## Subscribing to the events stream - This is done with `docker events` .exercise[ - Get a stream of events: ```bash docker events ``` <!-- ```meta ^Z ``` --> - In a new terminal, do *anything*: ```bash docker run --rm alpine sleep 10 ``` ] You should see events for the lifecycle of the container, as well as its connection/disconnection to the default `bridge` network. --- # Attaching labels - You can attach arbitrary labels to engines and containers - You can read the value of those labels - You can use those labels as filters in some commands .exercise[ - Start two containers, with and without a `backup` label: ```bash docker run -d --name leweb nginx docker run -d --name ledata --label backup=hourly redis ``` <!-- ```meta ^K ``` --> ] --- ## Using labels as filters - `docker ps` can take a `--filter` argument .exercise[ - List only containers that have a `backup` label: ```bash docker ps --filter label=backup ``` - List only containers where the `backup` label has a specific value: ```bash docker ps --filter label=backup=hourly ``` ] --- ## Filtering events - On a large cluster, there will be *lots* of events <br/>(especially when using short-lived containers) - `docker events` can also take a `--filter` argument .exercise[ - Show events only for containers with a "backup" label: ```bash docker events --filter label=backup ``` <!-- ```meta ^Z ``` --> - In a different terminal, terminate our containers: ```bash docker kill leweb ledata ``` <!-- ```meta ^K ``` --> ] --- ## Using `docker ps` in scripts - The default output of `docker ps` has two flaws: - it is not machine-readable - some information is not shown - This can be changed with the `--format` flag .exercise[ - List containers that have a `backup` label; <br/>show their container ID, image, and the label: ```bash docker ps --filter label=backup \ --format '{{ .ID }} {{ .Image }} {{ .Label "backup" }}' ``` ] --- # Logs - Two strategies: - log to plain files on volumes - log to stdout <br/>(and use a logging driver) --- ## Logging to plain files on volumes (Sorry, that part won't be hands-on!) - Start a container with `-v /logs` - Make sure that all log files are in `/logs` - To check logs, run e.g. ```bash docker run --volumes-from ... ubuntu sh -c "grep WARN /logs/*.log" ``` - Or just go interactive: ```bash docker run --volumes-from ... -ti ubuntu ``` - You can (should) start a log shipper that way --- ## Logging to stdout - All containers should write to stdout/stderr - Docker will collect logs and pass them to a logging driver - Logging driver can specified globally, and per container <br/>(changing it for a container overrides the global setting) - To change the global logging driver, pass extra flags to the daemon <br/>(requires a daemon restart) - To override the logging driver for a container, pass extra flags to `docker run` --- ## Specifying logging flags - `--log-driver` *selects the driver* - `--log-opt key=val` *adds driver-specific options* <br/>*(can be repeated multiple times)* - The flags are identical for `docker daemon` and `docker run` --- ## Logging flags in practice - If you provision your nodes with Docker Machine, you can set global logging flags (which will apply to all containers started by a given Engine) like this: ```bash docker-machine create ... --engine-opt log-driver=... ``` - Otherwise, use your favorite method to edit or manage configuration files - You can set per-container logging options in Compose files --- ## Available drivers - json-file (default) - syslog (can send to UDP, TCP, TCP+TLS, UNIX sockets) - awslogs (AWS CloudWatch) - journald - gelf - fluentd - splunk --- ## About json-file ... - It doesn't rotate logs by default, so your disks will fill up (Unless you set `maxsize` *and* `maxfile` log options.) - It's the only one supporting logs retrieval (If you want to use `docker logs`, `docker-compose logs`, or fetch logs from the Docker API, you need json-file!) - This might change in the future (But it's complex since there is no standard protocol to *retrieve* log entries.) All about logging in the documentation: https://docs.docker.com/reference/logging/overview/ --- # Storing container logs in an ELK stack *Important foreword: this is not an "official" or "recommended" setup; it is just an example. We do not endorse ELK, GELF, or the other elements of the stack more than others!* What we will do: - Spin up an ELK stack, with Compose - Gaze at the spiffy Kibana web UI - Manually send a few log entries over GELF - Reconfigure our DockerCoins app to send logs to ELK --- ## What's in an ELK stack? - ELK is three components: - ElasticSearch (to store and index log entries) - Logstash (to receive log entries from various sources, process them, and forward them to various destinations) - Kibana (to view/search log entries with a nice UI) - The only component that we will configure is Logstash - We will accept log entries using the GELF protocol - Log entries will be stored in ElasticSearch, <br/>and displayed on Logstash's stdout for debugging --- ## Starting our ELK stack - We will use a *separate* Compose file - The Compose file is in the `elk` directory .exercise[ - Go to the `elk` directory: ```bash cd ~/orchestration-workshop/elk ``` - Start the ELK stack: ```bash docker-compose up -d ``` ] --- ## Checking that our ELK stack works - Our default Logstash configuration sends a test message every minute - All messages are stored into ElasticSearch, but also shown on Logstash stdout .exercise[ - Look at Logstash stdout: ```bash docker-compose logs logstash ``` ] After less than one minute, you should see a `"message" => "ok"` in the output. --- ## Connect to Kibana - Our ELK stack exposes two public services: <br/>the Kibana web server, and the GELF UDP socket - They are both exposed on their default port numbers <br/>(5601 for Kibana, 12201 for GELF) .exercise[ - Open the UI in your browser: http://instance-address:5601/ ] --- ## "Configuring" Kibana - If you see a status page with a yellow item, wait a minute and reload (Kibana is probably still initializing) - Kibana should offer you to "Configure an index pattern", just click the "Create" button - Then: - click "Discover" (in the top-left corner) - click "Last 15 minutes" (in the top-right corner) - click "Last 1 hour" (in the list in the middle) - click "Auto-refresh" (top-right corner) - click "5 seconds" (top-left of the list) - You should see a series of green bars (with one new green bar every minute) ---  --- ## Sending container output to Kibana - We will create a simple container displaying "hello world" - We will override the container logging driver - The GELF address is `127.0.0.1:12201`, because the Compose file explicitly exposes the GELF socket on port 12201 .exercise[ - Start our one-off container: ```bash docker run --rm --log-driver gelf \ --log-opt gelf-address=udp://127.0.0.1:12201 \ alpine echo hello world ``` ] --- ## Visualizing container logs in Kibana - Less than 5 seconds later (the refresh rate of the UI), the log line should be visible in the web UI - We can customize the web UI to be more readable .exercise[ - In the left column, move the mouse over the following columns, and click the "Add" button that appears: - host - container_name - message ] --- ## Removing the old deployment of DockerCoins - Before redeploying DockerCoins, remove everything .exercise[ - Go back to the dockercoins directory: <br/>`cd ~/orchestration-workshop/dockercoins` - Stop and remove all DockerCoins containers: <br/>`docker-compose kill && docker-compose down --remove-orphans` - Reset the Compose file: <br/>`git checkout docker-compose.yml` - Point the Docker API to a single node: <br/>`unset DOCKER_HOST` ] --- ## Add the logging driver to the Compose file - We need to add the logging section to each container - We need the GELF endpoint that we got earlier with `docker-compose ps logstash` .exercise[ - Edit the `docker-compose.yml` file, adding the following lines **to each container**: ```yaml logging: driver: gelf options: gelf-address: "udp://127.0.0.1:12201" ``` <!-- ```edit cp docker-compose.yml-logging docker-compose.yml ``` --> ] Shortcut: `docker-compose.yml-logging` --- ## Start the DockerCoins app .exercise[ - Use Compose normally: ```bash docker-compose up -d ``` ] If you look in the Kibana web UI, you will see log lines refreshed every 5 seconds. Note: to do interesting things (graphs, searches...) we would need to create indexes. This is beyond the scope of this workshop. --- ## Logging in production - If we were using an ELK stack: - scale ElasticSearch - scale Logstash - move away from UDP *or* put one Logstash per node - interpose a Redis or Kafka queue - Configure your Engines to send all logs to ELK by default - Start the logging containers with a different logging system <br/>(to avoid a logging loop) - Make sure you don't end up writing *all logs* on the nodes running Logstash! --- # Security upgrades - This section is not hands-on - Public Service Announcement - We'll discuss: - how to upgrade the Docker daemon - how to upgrade container images --- ## Upgrading the Docker daemon - Stop all containers cleanly - Stop the Docker daemon - Upgrade the Docker daemon - Start the Docker daemon - Start all containers - This is like upgrading your Linux kernel, but it will get better (Docker Engine 1.11 is using containerd, which will ultimately allow seamless upgrades.) --- ## In practice - Keep track of running containers before stopping the Engine: ```bash docker ps --no-trunc -q | tee /tmp/running | xargs -n1 -P10 docker stop ``` - Restart those containers after the Engine is running again: ```bash xargs docker start < /tmp/running ``` <br/>(Run this multiple times if you have linked containers!) --- ## Upgrading container images - When a vulnerability is announced: - if it affects your base images: make sure they are fixed first - if it affects downloaded packages: make sure they are fixed first - re-pull base images - rebuild - restart containers --- ## How do we know when to upgrade? - Subscribe to CVE notifications - https://cve.mitre.org/ - your distros' security announcements - Check CVE status in official images <br/>(tag [cve-tracker]( https://github.com/docker-library/official-images/labels/cve-tracker) in [docker-library/official-images]( https://github.com/docker-library/official-images/labels/cve-tracker) repo) - Coming soon: Project Nautilus <br/>(see [this DC15EU presentation]( http://www.slideshare.net/Docker/official-repos-and-project-nautilus/26)) --- ## Upgrading with Compose Compose makes this particularly easy: ```bash docker-compose build --pull --no-cache docker-compose up -d ``` This will automatically: - pull base images; - rebuild all container images; - bring up the new containers. Remember: Compose will automatically move our volumes to the new containers, so data is preserved. --- # Network traffic analysis - We still have `myredis` running - We will use *shared network namespaces* to perform network analysis - Two containers sharing the same network namespace... - have the same IP addresses - have the same network interfaces - `eth0` is therefore the same in both containers --- ## Install and start `ngrep` Ngrep uses libpcap (like tcpdump) to sniff network traffic. .exercise[ <!-- ```meta ^{ ``` --> - Start a container with the same network namespace: <br/>`docker run --net container:dockercoins_redis_1 -ti alpine sh` - Install ngrep: <br/>`apk update && apk add ngrep` - Run ngrep: <br/>`ngrep -tpd eth0 -Wbyline . tcp` <!-- ```meta ^} ``` --> ] You should see a stream of Redis requests and responses. --- class: title # Dynamic orchestration --- ## Static vs Dynamic - Static - you decide what goes where - simple to describe and implement - seems easy at first but doesn't scale efficiently - Dynamic - the system decides what goes where - requires extra components (HA KV...) - scaling can be finer-grained, more efficient --- ## Mesos (overview) - First presented in 2009 - Initial goal: resource scheduler (two-level/pessimistic) - top-level "master" knows the global cluster state - "slave" nodes report status and resources to master - master allocates resources to "frameworks" - Container support added recently (had to fit existing model) - Network and service discovery is complex --- ## Mesos (in practice) - Super easy to setup a test cluster <br/>(e.g. [minimesos](https://minimesos.org/) put all Mesos components in container) - Great to accommodate mixed workloads <br/>(see Marathon, Chronos, Aurora, and many more) - "Meh" if you only want to run Docker containers - In production on clusters of thousands of nodes - Open source project; commercial support available --- ## Kubernetes (overview) - Started in June 2014 - Designed specifically as a platform for containers ("greenfield" design) - "pods" = groups of containers sharing network/storage - Scaling and HA managed by "replication controllers" - extensive use of "tags" instead of e.g. tree hierarchy - Initially designed around Docker, but doesn't hesitate to diverge in a few places --- ## Kubernetes (in practice) - Network and service discovery is powerful, but complex <br/>.small[(different mechanisms within pod, between pods, for inbound traffic...)] - Initially designed around GCE <br/>.small[(networking and persistence work better on GCE, but this is getting better)] - Tends to be loved by ops more than devs <br/>.small[(but keep in mind that it's evolving quite as fast as Docker)] - Setting up a local development cluster is non-trivial <br/>.small[(the easiest way is to stage it up on e.g. GKE)] - Bottom line: Kubernetes is not Docker! <br/>.small[(different concepts, APIs, tooling, concepts, configuration files...)] --- ## ECS (overview) - Amazon EC2 Container Service - "Bring your own instance" - "Native" container scheduler on AWS - Some integration with other AWS products - No extra component to operate - Defines new concepts: - task - task definition - service --- ## ECS (in practice) - Task definitions look like Compose files, but are significantly different - Integration with e.g. ELB is suboptimal <br/>(ELB requires all backends to run on the same port) - Cluster deployment is made easier thanks to ECS CLI - Docker API gets partially exposed through ECS API, with some features lagging behind - Service discovery is painful --- ## Nomad (overview) - Generic job scheduler <br/>(not only for containers) - Desired state is stored in Consul - Nodes pull jobs from Consul - Scheduling happens in parallel --- ## Nomad (in practice) *Disclaimer: I have little first-hand experience with Nomad!* - Does only one thing, but does it really well - Works with jobs, not applications, services, etc. - As I understand it: Nomad is an excellent building block, <br/>but you need to add other components to deploy your apps --- ## Swarm (in theory) - Consolidates multiple Docker hosts into a single one - You talk to Swarm using the Docker API → you can use all existing tools: Docker CLI, Docker Compose, etc. - Swarm talks to your Docker Engines using the Docker API too → you can use existing Engines without modification - Dispatches (schedules) your containers across the cluster, transparently - Open source and written in Go (like Docker) - Started by two of the original Docker authors ([@aluzzardi](https://twitter.com/aluzzardi) and [@vieux](https://twitter.com/vieux)) --- ## Swarm (in practice) - Stable since November 2015 - Initially for small setups, easy to setup - Tested with 1000 nodes + 50000 containers <br/>.small[(without particular tuning; see DockerCon EU opening keynotes!)] - Still easy to setup, but scales perfectly to larger deployments too - Requires a key/value store for advanced features - We'll see it in action! ??? ## PAAS on Docker - The PAAS workflow: *just push code* <br/>(inspired by Heroku, dotCloud...) - TL,DR: easier for devs, harder for ops, <br/>some very opinionated choices - A few examples: <br/>(Non-exhaustive list!!!) - Cloud Foundry - Deis - Dokku - Flynn - Tsuru *Docker made it very easy to cobble your own PAAS.* ??? ## A few other tools - Volume plugins (Convoy, Flocker...) - manage/migrate stateful containers (and more) - Network plugins (Contiv, Weave...) - overlay network so that containers can ping each other - Docker Cloud (Tutum), Docker UCP (Universal Control Plane) - dashboards to manage fleets of Docker hosts ... And many more! --- class: title # Hands-on Swarm --- class: pic  --- ## Setting up our Swarm cluster - This can be done manually or with **Docker Machine** - Manual deployment: - with TLS: certificate generation is painful <br/>(needs dual-use certs) - without TLS: easier, but insecure <br/>(unless you run on your internal/private network) - Docker Machine deployment: - generates keys, certificates, and deploys them for you - can also create VMs --- ## The Way Of The Machine - Install `docker-machine` (single binary download) - Set a few environment variables (cloud credentials) - Create one or more machines: <br/>`docker-machine create -d digitalocean node42` - List machines and their status: <br/>`docker-machine ls` - Select a machine for use: <br/>`eval $(docker-machine env node42)` <br/>(this will set a few environment variables) - Execute regular commands with Docker, Compose, etc. <br/>(they will pick up remote host address from environment) --- ## Docker Machine `generic` driver - Most drivers work the same way: - use cloud API to create instance - connect to instance over SSH - install Docker - The `generic` driver skips the first step - It can install Docker on any machine, as long as you have SSH access - We will use that! --- # Deploying Swarm - Components involved: - cluster discovery mechanism <br/>(so that the manager can learn about the nodes) - Swarm manager <br/>(your frontend to the cluster) - Swarm agent <br/>(runs on each node, registers it with service discovery) --- # Cluster discovery - Possible backends: - dynamic, self-hosted <br/>(requires to run a Consul/Etcd/Zookeeper cluster) - static, through command-line or file <br/>(great for testing, or for private subnets, see [this article]( https://medium.com/on-docker/docker-swarm-flat-file-engine-discovery-2b23516c71d4#.6vp94h5wn) - external, token-based <br/>(dynamic; nothing to operate; relies on external service operated by Docker Inc.) - We will use the token mechanism --- ## Generating our Swarm discovery token The token is a unique identifier, corresponding to a bucket in the discovery service hosted by Docker Inc. (You can consider it as a rendez-vous point for your cluster.) .exercise[ - Create your token, saving it preciously to disk as well: ```bash TOKEN=$(docker run swarm create | tee ~/token) echo $TOKEN ``` ] --- ## Swarm agent - Used only for dynamic discovery (zk, etcd, consul, token) - Must run on each node - Every 20s (by default), tells to the discovery system: </br>"Hello, there is a Swarm node at A.B.C.D:EFGH" - Must know the node's IP address <br/>(sorry, it can't figure it out by itself, because <br/>it doesn't know whether to use public or private addresses) - The node continues to work even if the agent dies - Automatically started by Docker Machine (when the `--swarm` option is passed) --- ## Swarm manager - Accepts Docker API requests - Communicates with the cluster nodes - Performs healthchecks, scheduling... --- ## Pre-flight checks - According to data gathered during previous workshops, the most common ways to get the next part wrong are: - provisioning from the wrong node - provisioning from a terminal where `$TOKEN` is not set .exercise[ - Double-check that you are on `node1` - Double-check that the `$TOKEN` environment variable is set: ```bash echo $TOKEN ``` ] --- ## Provision our first node with Docker Machine - If `$TOKEN` is set properly, we can move forward! .exercise[ <!-- ```placeholder AA.BB.CC.DD $(getent hosts node1 | awk '{print $1}') ``` --> - Invoke Docker Machine with *all the flags*: ```bash echo $TOKEN docker-machine create --driver generic \ --swarm --swarm-master --swarm-discovery token://$TOKEN \ --generic-ssh-user docker --generic-ip-address `AA.BB.CC.DD` node1 ``` ] (Don't forget to replace AA.BB.CC.DD with the node IP address!) --- ## Redundancy - The manager is a SPOF - If you lose the manager: - you can't control the cluster anymore - you can still control individual nodes - you can start a new manager (at this point, it is stateless) - We'll setup active/passive redundancy later --- ## Check our node Let's connect to the node *individually*. .exercise[ - Select the node with Machine ```bash eval $(docker-machine env node1) ``` - Execute some Docker commands ```bash docker version docker info docker ps ``` ] Two containers should show up: the agent and the manager. --- ## Check our (single-node) Swarm cluster Let's connect to the manager instead. .exercise[ - Select the Swarm manager with Machine ```bash eval $(docker-machine env node1 --swarm) ``` <!-- ```bash while ! docker info | grep -q "^Nodes: 1"; do sleep 1 done ``` --> - Execute some Docker commands ```bash docker version docker info docker ps ``` ] The output is different! Let's review this. --- ## `docker version` Swarm identifies itself clearly: ``` Client: Version: 1.10.2 API version: 1.22 Go version: go1.5.3 Git commit: c3959b1 Built: Mon Feb 22 21:40:35 2016 OS/Arch: linux/amd64 Server: Version: swarm/1.1.3 API version: 1.22 Go version: go1.5.3 Git commit: 7e9c6bd Built: Wed Mar 2 00:15:12 UTC 2016 OS/Arch: linux/amd64 ``` --- ## `docker info` Swarm gives cluster information, showing all nodes: .small[ ``` Containers: 3 Images: 6 Role: primary Strategy: spread Filters: affinity, health, constraint, port, dependency Nodes: 1 node1: 52.58.50.15:2376 └ Status: Healthy └ Containers: 3 └ Reserved CPUs: 0 / 2 └ Reserved Memory: 0 B / 3.859 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.2.0-30-generic, operatingsystem=Ubuntu 15.10, provider=generic, storagedriver=aufs └ Error: (none) └ UpdatedAt: 2016-03-09T14:01:43Z Kernel Version: 4.2.0-30-generic Operating System: linux Architecture: amd64 CPUs: 2 Total Memory: 3.86 GiB Name: node1 ``` ] --- ## `docker ps` - This one should show nothing at this point - The Swarm containers are hidden - This avoids unneeded pollution - This also avoids killing them by mistake - We can still see them with `docker ps -a`, though --- ## Add other nodes to the cluster - Let's use *almost* the same command line (but without `--swarm-master`) .exercise[ - Stay on `node1` (it has keys and certificates now!) - Add another node with Docker Machine <!-- ```placeholder AA.BB.CC.DD $(getent hosts node2 | awk '{print $1}') ``` --> ```bash TOKEN=$(cat ~/token) docker-machine create --driver generic \ --swarm --swarm-discovery token://$TOKEN \ --generic-ssh-user docker --generic-ip-address `AA.BB.CC.DD` node2 ``` ] Remember to replace AA.BB.CC.DD with the correct IP address. --- ## Scripting - We can automate the remaining nodes with a little shell script .exercise[ - Deploy nodes 3, 4, and 5: ```bash TOKEN=$(cat ~/token) grep node[345] /etc/hosts | grep -v ^127 | while read IPADDR NODENAME do docker-machine create --driver generic \ --swarm --swarm-discovery token://$TOKEN \ --generic-ssh-user docker \ --generic-ip-address $IPADDR $NODENAME done ``` ] --- ## Checking the state of our cluster .exercise[ <!-- ```bash while ! docker info | grep -q "^Nodes: 5"; do sleep 1 done ``` --> - Check that all nodes are here: ```bash docker info ``` ] If less than 5 nodes show up, just retry every few seconds: the nodes need some time to register themselves with the service discovery mechanism. --- ## Running containers on Swarm Try to run a few `busybox` containers. Then, let's get serious: .exercise[ - Start a Redis service: <br/>`docker run -dP redis` - See the service address: <br/>`docker port $(docker ps -lq) 6379` ] This can be any of your five nodes. --- ## Scheduling strategies - Random: pick a node at random <br/>(but honor resource constraints) - Spread: pick the node with the least containers <br/>(including stopped containers) - Binpack: try to maximize resource usage <br/>(in other words: use as few hosts as possible) --- # Resource allocation - Swarm can honor resource reservations - This requires containers to be started with resource limits - Swarm refuses to schedule a container if it cannot honor a reservation .exercise[ - Start Redis containers with 1 GB of RAM until Swarm refuses to start more: ```bash docker run -d -m 1G redis ``` ] On a cluster of 5 nodes with ~3.8 GB of RAM per node, Swarm will refuse to start the 16th container. --- ## Things to know about resource allocation - `docker info` shows resource allocation for each node - Swarm allows a 5% resource overcommit (tunable) - Containers without resource reservation can always be started - Resources of stopped containers are still counted as being reserved - this guarantees that it will be possible to restart a stopped container - containers have to be deleted to free up their resources - `docker update` can be used to change resource allocation on the fly --- ## Connecting containers with Swarm (1/2) - Implement service discovery in the application - requires extensive code changes - doesn't require extra services or containers - provides load balancing and failover - Inject service addresses in environment variables - requires minimal code changes - doesn't require extra services or containers - doesn't provide load balancing and failover --- ## Connecting containers with Swarm (2/2) - Ambassadors - don't require code changes - require additional containers - provide load balancing and failover - Overlay networks - don't require code changes - don't require extra services or containers - doesn't provide load balancing and failover (yet) --- # Connecting containers with ambassadors - Hands off section (for information only!) - We will use one-tier, dynamic ambassadors - Each link to a service will be replaced by an ambassador - Each ambassador will be placed in the network namespace of the service using the ambassador - Ambassadors will be dynamically reconfigured when linked services are updated --- ## Revisiting `jpetazzo/hamba` - Configuration is stored in a *volume* - A watcher process looks for configuration updates, <br/>and restarts HAProxy when needed - It can be started without configuration: ```bash docker run --name amba jpetazzo/hamba run ``` - There is a helper to inject a new configuration: ```bash docker run --rm --volumes-from amba jpetazzo/hamba \ reconfigure 80 backend1 port1 backend2 port2 ... ``` --- ## Network namespaces and `extra_hosts` This is our plan: - Replace each `link` with an `extra_host`, pointing to the `127.127.X.X` address space - Start app containers normally (`docker-compose up`, `docker-compose scale`) - Start ambassadors after app containers are up: - ambassadors bind to `127.127.X.X` - they share their client's network namespace - Reconfigure ambassadors each time something changes --- ## Our plan for service discovery - Replace all `links` with static `/etc/hosts` entries - Those entries will map to `127.127.0.X` (with different `X` for each service) - Example: `redis` will point to `127.127.0.2` (instead of a container address) - Start all services; scale them if we want (at this point, they will all fail to connect) - Start ambassadors in the services' namespace; <br/>each ambassador will listen on the right `127.127.0.X` - Gather all backend addresses and configure ambassadors .warning[Services should try to reconnect!] --- ## "Design for failure," they said - When the containers are started, the network is not ready - First connection attempts **will fail** - App should try to reconnect - It is OK to crash and restart - Exponential back-off is nice --- ## Pictures worth 1000 words - In the following diagrams, we are connecting a `www` service to a `redis` service through an ambassador. --- class: pic  --- class: pic  --- class: pic  --- class: pic  --- class: pic  --- class: pic  --- ## Our tools - `link-to-ambassadors.py` - replaces all `links` with `extra_hosts` entries - `create-ambassadors.py` - scans running containers - allocates `127.127.X.X` addresses - starts (unconfigured) ambassadors - `configure-ambassadors.py` - scans running containers - gathers backend addresses - sends configuration to ambassadors --- ## Convert links to ambassadors - When we ran `build-tag-push.py` earlier, <br/>it generated a new `docker-compose.yml-NNN` file. .notexercise[ - Run the first script to create a new YAML file: <br/>`../bin/link-to-ambassadors.py` ] In the Compose file, all links have been replaced by `extra_hosts` sections. --- class: pic ## Current state  --- ## Bring up the application The application can now be started and scaled. .notexercise[ - Start the application: <br/>`docker-compose up -d` ] Note: you can scale everything as you like, *except Redis*, because it is stateful. --- class: pic ## Current state  --- ## Create the ambassadors This has to be executed each time you create new services or scale up existing ones. After reading `$COMPOSE_FILE`, it will scan running containers, and compare: - the list of app containers, - the list of ambassadors. It will create missing ambassadors. .notexercise[ - Run the script! <br/>`../bin/create-ambassadors.py` ] --- class: pic ## Current state  --- ## Configure the ambassadors All ambassadors are created but they still need configuration. That's the purpose of the last script. It will read `$COMPOSE_FILE` and gather: - the list of app backends, - the list of ambassadors. Then it configures all ambassadors with all found backends. .notexercise[ - Run it! <br/>`../bin/configure-ambassadors.py` ] --- class: pic ## Current state  --- ## Check what we did .notexercise[ - Find out the address of the web UI: <br/>`docker-compose ps webui` - Point your browser to it ] --- ## Scale - We will now add more containers. .notexercise[ - Scale worker and rng: ```bash docker-compose scale worker=5 rng=10 ``` ] The performance graph stays at the same level. If we look at the logs of the added workers, we will see screenfuls of "connection refused" exceptions. --- class: pic ## Current state  --- ## Add ambassadors - The new containers don't have ambassadors at this point. .notexercise[ - Create the missing ambassadors with the script: ```bash ../bin/create-ambassadors.py ``` ] The performance graph stays at the same level. If we look at the logs of the added workers, we will now see timeout errors instead of "connection refused." --- class: pic ## Current state  --- ## Configure ambassadors - The last step is to inject the updated configuration. .notexercise[ - Run the last script one more time: ```bash ../bin/configure-ambassadors.py ``` ] Now the performance graph climbs up, and the worker logs show normal operation. --- class: pic ## Current state  --- ## A few words about those ambassadors - There is "a lot" of added complexity here <br/>(5 scripts of almost 50 lines each!) - Snark aside, those scripts tap into those concepts: - network namespaces - dynamic load balancer reconfiguration - sidekick containers that are *mandatory* - ... and have to be managed manually - We are going to see an easier way to manage this! --- # Setting up Consul and overlay networks - We will reconfigure our Swarm cluster to enable overlays - We will deploy a Consul cluster - We will connect containers running on different machines --- ## First, let's Clean All The Things! - We need to remove the old containers <br/>(in particular the `swarm` agents and managers) .exercise[ - The following snippet will nuke all containers on all hosts: ```bash for N in 1 2 3 4 5 do ssh node$N "docker ps -qa | xargs -r docker rm -f" done ``` (If it asks you to confirm SSH keys, just do it!) ] Note: our Swarm cluster is now broken. --- ## Remove old Machine information - We will use `docker-machine rm` - With the `generic` driver, this doesn't do anything (it just deletes local configuration) - With cloud/VM drivers, this would actually delete VMs .exercise[ - Remove our nodes from Docker Machine config database: ```bash for N in 1 2 3 4 5 do docker-machine rm -f node$N done ``` ] --- ## Add extra options to our Engines - We need two new options for our engines: - `cluster-store` (to indicate which key/value store to use) - `cluster-advertise` (to indicate which IP address to register) - `cluster-store` will be `consul://localhost:8500` <br/>(we will run one Consul node on each machine) - `cluster-advertise` will be `eth0:2376` <br/>(Engine will automatically pick up eth0's IP address) --- ## Reconfiguring Swarm clusters, the Docker way - The traditional way to reconfigure a service is to edit its configuration (or init script), then restart - Instead, we can use Machine's `generic` driver to redeploy with new parameters .exercise[ - Re-provision the manager node: <!-- ```placeholder AA.BB.CC.DD $(getent hosts node1 | awk '{print $1}') ``` --> ```bash docker-machine create --driver generic \ --engine-opt cluster-store=consul://localhost:8500 \ --engine-opt cluster-advertise=eth0:2376 \ --swarm --swarm-master --swarm-discovery consul://localhost:8500 \ --generic-ssh-user docker --generic-ip-address `AA.BB.CC.DD` node1 ``` ] --- ## Reconfigure the other nodes - Once again, scripting to the rescue! .exercise[ ```bash grep node[2345] /etc/hosts | grep -v ^127 | while read IPADDR NODENAME do docker-machine create --driver generic \ --engine-opt cluster-store=consul://localhost:8500 \ --engine-opt cluster-advertise=eth0:2376 \ --swarm --swarm-discovery consul://localhost:8500 \ --generic-ssh-user docker \ --generic-ip-address $IPADDR $NODENAME done ``` ] --- ## Checking what we did .exercise[ - Directly point the CLI to a node and check configuration: ```bash eval $(docker-machine env node1) docker info ``` (should show `Cluster store` and `Cluster advertise`) - Try to talk to the Swarm cluster: ```bash eval $(docker-machine env node1 --swarm) docker info ``` (should show zero node) ] --- ## Why zero node? - We haven't started Consul yet - Swarm discovery is not operationl - Swarm can't discover the nodes Note: Docker will start (and be functional) without a K/V store. This lets us run Consul itself in a container. --- ## Adding Consul - We will run Consul in containers - We will use the [Consul official image]( https://hub.docker.com/_/consul/) that was released *very recently* - We will tell Docker to automatically restart it on reboots - To simplify network setup, we will use `host` networking --- ## A few words about `host` networking - Consul needs to be aware of its actual IP address (seen by other nodes) - It also binds a bunch of different ports - It makes sense (from a security point of view) to have Consul listening on localhost only (and have "users", i.e. Engine, Swarm, etc. connect over localhost) - Therefore, we will use `host` networking! - Also: Docker Machine 0.6 starts the Swarm containers in `host` networking ... - ... but Docker Machine 0.7 doesn't (which is why we stick to 0.6 for now) --- ## Consul fundamentals (if I must give you just one slide...) - Consul nodes can be "just an agent" or "server" - From the client's perspective, they behave the same - Only servers are members in the Raft consensus / leader election / etc (non-server agents forward requests to a server) - All nodes (except maybe one) must be told the address of at least another node to join - At least the first nodes must know how many nodes to expect to have quorum - Consul can have only one "truth" at a time (hence the importance of quorum) --- ## Starting our Consul cluster .exercise[ - Make sure you're logged into `node1`, and: ```bash IPADDR=$(ip a ls dev eth0 | sed -n 's,.*inet \(.*\)/.*,\1,p') for N in 1 2 3 4 5; do ssh node$N -- docker run -d --restart=always --name consul_node$N \ -e CONSUL_BIND_INTERFACE=eth0 --net host consul \ agent -server -retry-join $IPADDR -bootstrap-expect 3 \ -ui -client 0.0.0.0 done ``` ] Note: in production, you probably want to remove `-client 0.0.0.0` since it gives public access to your cluster! Also adapt `-bootstrap-expect` to your quorum. --- ## Check that our Consul cluster is up - With your browser, navigate to any instance on port 8500 <br/>(in "NODES" you should see the five nodes) - Let's run a couple of useful Consul commands .exercise[ - Ask Consul the list of members it knows: ```bash docker run --net host --rm consul members ``` - Ask Consul which node is the current leader: ```bash curl localhost:8500/v1/status/leader ``` ] --- ## Check that our Swarm cluster is up .exercise[ - Try again the `docker info` from earlier: ```bash eval $(docker-machine env --swarm node1) docker info ``` - Now all nodes should be visible <br/>(Give them a minute or two to register) ] --- # Multi-host networking - Docker 1.9 has the concept of *networks* - By default, containers are on the default "bridge" network - You can create additional networks - Containers can be on multiple networks - Containers can dynamically join/leave networks - The "overlay" driver lets networks span multiple hosts - Let's see that in action! --- ## Create a few networks and containers .exercise[ - Create two networks, *blue* and *green*: ```bash docker network create --driver overlay blue docker network create --driver overlay green docker network ls ``` - Create containers with names of blue and green things, on their respective networks: ```bash docker run -d --net-alias things --name sky --net blue -m 3G redis docker run -d --net-alias things --name navy --net blue -m 3G redis docker run -d --net-alias things --name grass --net green -m 3G redis docker run -d --net-alias things --name forest --net green -m 3G redis ``` ] --- ## Check connectivity within networks .exercise[ - Check that our containers are on different networks: ```bash docker ps ``` - This will work: ```bash docker run --rm --net blue alpine ping -c 3 navy ``` - This will not: ```bash docker run --rm --net blue alpine ping -c 3 grass ``` ] --- ## Containers connected to multiple networks - Some colors aren't *quite* blue *nor* green .exercise[ - Create a container that we want to be on both networks: ```bash docker run -d --net-alias things --net blue --name turquoise redis ``` - Check connectivity: ```bash docker exec -ti turquoise ping -c 3 navy docker exec -ti turquoise ping -c 3 grass ``` (First works; second doesn't) ] --- ## Dynamically connecting containers - This is achieved with the command: <br/>`docker network connect NETNAME CONTAINER` .exercise[ - Dynamically connect to the green network: ```bash docker network connect green turquoise ``` - Check connectivity: ```bash docker exec -ti turquoise ping -c 3 navy docker exec -ti turquoise ping -c 3 grass ``` (Both commands work now) ] --- ## Network aliases - Each container was created with the network alias `things` - Network aliases are scoped by network .exercise[ - Resolve the `things` alias from both networks: ```bash docker run --rm --net blue alpine sh -c \ "apk add --update drill && drill things" docker run --rm --net green alpine sh -c \ "apk add --update drill && drill things" ``` ] --- ## Under the hood - Each network has an interface in the container - There is also an interface for the default gateway .exercise[ - View interfaces in our `turquoise` container: ```bash docker exec -ti turquoise ip addr ls ``` ] --- ## Dynamically disconnecting containers - There is a mirror command to `docker network connect` .exercise[ - Disconnect the *turquoise* container from *blue* (its original network): ```bash docker network disconnect blue turquoise docker network connect green turquoise ``` - Check connectivity: ```bash docker exec -ti turquoise ping -c 3 navy docker exec -ti turquoise ping -c 3 grass ``` (First command fails, second one works) ] --- ## Cleaning up .exercise[ - Destroy containers: ```bash docker rm -f sky navy grass forest turquoise ``` - Destroy networks: ```bash docker network rm blue docker network rm green ``` ] --- ## Cleaning up after an outage or a crash - You cannot remove a network if it still has containers - There is no `"rm -f"` for network - If a network still has stale endpoints, you can use `"disconnect -f"` --- # Building images with Swarm - This is a relatively new feature (Compose ~1.5) - Builds are scheduled on random nodes - This disrupts caching - Independent builds will be fine, though --- ## Caveats when building with Swarm - Containers are only scheduled where they were built - cause: Swarm will not automatically copy images across nodes - solution: distribute images through a registry (e.g. Docker Hub) - You can end up with inconsistent versions <br/>(i.e. `dockercoins_rng:latest` being different on two nodes) - cause: builds can happen on different nodes - solution: always pin builds to the same node; use explicit version tags - Also, build caching doesn't work all the time --- ## Why can't Swarm do this automatically for us? - Let's step back and think for a minute ... - What should `docker build` do on Swarm? - build on one machine - build everywhere ($$$) - After the build, what should `docker run` do? - run where we built (how do we know where it is?) - run on any machine that has the image - Could Compose+Swarm solve this automatically? --- ## A few words about "sane defaults" - *It would be nice if Swarm could pick a node, and build there!* - but which node should it pick? - what if the build is very expensive? - what if we want to distribute the build across nodes? - what if we want to tag some builder nodes? - ok but what if no node has been tagged? - *It would be nice if Swarm could automatically push images!* - using the Docker Hub is an easy choice <br/>(you just need an account) - but some of us can't/won't use Docker Hub <br/>(for compliance reasons or because no network access) .small[("Sane" defaults are nice only if we agree on the definition of "sane")] --- ## The plan - Build on a single node (`node1`) - Tag images - Upload them to a registry - Update the Compose file to use those images *That's the purpose of the `build-tag-push.py` script!* --- ## Which registry do we want to use? .small[ - **Docker Hub** - hosted by Docker Inc. - requires an account (free, no credit card needed) - images will be public (unless you pay) - located in AWS EC2 us-east-1 - **Docker Trusted Registry** - self-hosted commercial product - requires a subscription (free 30-day trial available) - images can be public or private - located wherever you want - **Docker open source registry** - self-hosted barebones repository hosting - doesn't require anything - doesn't come with anything either - located wherever you want ] --- ## Using Docker Hub - Set the `DOCKER_REGISTRY` environment variable to your Docker Hub user name <br/>(the `build-tag-push.py` script prefixes each image name with that variable) - We will also see how to run the open source registry <br/>(so use whatever option you want!) .exercise[ <!-- ```meta ^{ ``` --> - Set the following environment variable: <br/>`export DOCKER_REGISTRY=jpetazzo` - (Use *your* Docker Hub login, of course!) - Log into the Docker Hub: <br/>`docker login` <!-- ```meta ^} ``` --> ] --- ## Using Docker Trusted Registry If we wanted to use DTR, we would: - make sure we have a Docker Hub account - [activate a Docker Datacenter subscription]( https://hub.docker.com/enterprise/trial/) - install DTR on our machines - set `DOCKER_REGISTRY` to `dtraddress:port/user` *This is out of the scope of this workshop!* --- ## Using open source registry - We need to run a `registry:2` container <br/>(make sure you specify tag `:2` to run the new version!) - It will store images and layers to the local filesystem <br/>(but you can add a config file to use S3, Swift, etc.) - Docker *requires* TLS when communicating with the registry, unless for registries on `localhost` or with the Engine flag `--insecure-registry` - Our strategy: run a reverse proxy on `localhost:5000` on each node --- # Deploying a local registry - There is a Compose file for that .exercise[ - Go to the `registry` directory in the repository: ```bash cd ~/orchestration-workshop/registry ``` ] Let's examine the `docker-compose.yml` file. --- ## Running a local registry with Compose ```yaml version: "2" services: backend: image: registry:2 frontend: image: jpetazzo/hamba command: 5000 backend:5000 ports: - "127.0.0.1:5000:5000" depends_on: - backend ``` - *Backend* is the actual registry. - *Frontend* is the ambassador that we deployed earlier. <br/> It communicates with *backend* using an internal network and network aliases. --- ## Starting a local registry with Compose - We will bring up the registry - Then we will ensure that one *frontend* is running on each node by scaling it to our number of nodes .exercise[ - Start the registry: ```bash docker-compose up -d ``` ] --- ## "Scaling" the local registry - This is a particular kind of scaling - We just want to ensure that one *frontend* is running on every single node of the cluster .exercise[ - Scale the registry: ```bash for N in $(seq 1 5); do docker-compose scale frontend=$N done ``` ] Note: Swarm might do that automatically for us in the future. --- ## Testing our local registry - We can retag a small image, and push it to the registry .exercise[ - Make sure we have the busybox image: ```bash docker pull busybox ``` - Retag the busybox image: ```bash docker tag busybox localhost:5000/busybox ``` - Push it: ```bash docker push localhost:5000/busybox ``` ] --- ## Adapting our Compose file to run on Swarm - We can get rid of all the `ports` section, except for the web UI --- ## Our new Compose file .small[ ```yaml version: '2' services: rng: build: rng hasher: build: hasher webui: build: webui ports: - "8000:80" redis: image: redis worker: build: worker ``` ] Copy-paste this into `docker-compose.yml` <br/>(or you can `cp docker-compose.yml-v2 docker-compose.yml`) --- ## Use images, not builds - We need to replace each `build` with an `image` - We will use the `build-tag-push.py` script for that .exercise[ - Set `DOCKER_REGISTRY` to use our local registry, <br/>then build, tag, and push the application: ```bash export DOCKER_REGISTRY=localhost:5000 ../bin/build-tag-push.py ``` ] --- ## Run the application - At this point, our app is ready to run - We don't need ambassadors or extra containers .exercise[ - Start the application: ```bash export COMPOSE_FILE=docker-compose.yml-`NNN` eval $(docker-machine env node1 --swarm) docker-compose up -d ``` - Observe that it's running on multiple nodes: <br/>(each container name is prefixed with the node it's running on) ```bash docker ps ``` ] --- ## View the performance graph - Load up the graph in the browser .exercise[ - Check the `webui` service address and port: ```bash docker-compose port webui 80 ``` - Open it in your browser ] --- # Scaling workers - Scaling the `worker` service works out of the box (like before) .exercise[ - Scale `worker`: ```bash docker-compose scale worker=10 ``` ] We will hit the bottleneck caused by the `rng` service. How can we scale that service? --- ## Load balancing with overlay networks, take 1 - Replace `rng` with: - multiple copies `rng1`, `rng2`, `rng3`, ... - a load balancer taking over the name `rng`, <br/>and spreading traffic accross all instances - Can we do better? - In a perfect world, we would like to do: ```bash docker-compose scale rng=10 ``` --- ## Load balancing with overlay networks, take 2 - Scale `rng` and rely on DNS round robin records - Problem: the load might end up being totally unbalanced - We need to send `rng` traffic to an actual load balancer - This poses a naming problem --- ## Naming problem - Service is called `rng` - It therefore takes the network name `rng` - Worker code connects to `rng` - So `rng` should point to the load balancer - What do‽ --- ## Naming is *per-network* - Solution: put `rng` on its own network - That way, it doesn't take the network name `rng` <br/>(at least not on the default network) - Have the load balancer sit on both networks - Add the name `rng` to the load balancer --- class: pic Original DockerCoins  --- class: pic Load-balanced DockerCoins  --- ## Declaring networks - Networks (other than the default one) *must* be declared in a top-level `networks` section, placed anywhere in the file .exercise[ - Add the `rng` network to the Compose file: ```yaml version: '2' networks: rng: services: rng: image: ... ... ``` ] --- ## Putting the `rng` service in its network - Services can have a `networks` section - If they don't: they are placed in the default network - If they do: they are placed only in the mentioned networks .exercise[ - Change the `rng` service to put it in its network: ```yaml rng: image: localhost:5000/dockercoins_rng:… networks: rng: ``` ] --- ## Adding the load balancer - The load balancer has to be in both networks: `rng` and `default` - In the `default` network, it must have the `rng` alias - We will use the `jpetazzo/hamba` image .exercise[ - Add the `rng-lb` service to the Compose file: ```yaml rng-lb: image: jpetazzo/hamba command: run networks: rng: default: aliases: [ rng ] ``` ] --- ## Load balancer initial configuration - We specified `run` as the initial command - This tells `hamba` to wait for an initial configuration - The load balancer will not be operational <br/>(until we feed it its configuration) --- ## Start the application .exercise[ - Bring up DockerCoins: ```bash docker-compose up -d ``` - See that `worker` is complaining: ```bash docker-compose logs worker ``` ] --- ## Configure the load balancer - Multiple solutions: - lookup the IP address of the `rng` backend - use the backend's network name - use the backend's container name (easiest!) .exercise[ - Configure the load balancer: ```bash docker run --rm --volumes-from dockercoins_rng-lb_1 \ --net container:dockercoins_rng-lb_1 \ jpetazzo/hamba reconfigure 80 dockercoins_rng_1 80 ``` ] The application should now be working correctly. --- ## Scale the application - Use `docker-compose scale` as planned .exercise[ - Scale `rng`: ```bash docker-compose scale rng=10 ``` ] Of course, the graph doesn't change *yet*. We need to add the new backends to the load balancer configuration first. --- ## Reconfigure the load balancer - The command is similar to the one before - We need to pass the list of all backends .exercise[ - Reconfigure the load balancer: ```bash docker run --rm \ --volumes-from dockercoins_rng-lb_1 \ --net container:dockercoins_rng-lb_1 \ jpetazzo/hamba reconfigure 80 \ $(for N in $(seq 1 10); do echo dockercoins_rng_$N:80 done) ``` ] --- ## Automating the process - Nobody loves artisan YAML handy craft - This can be automated very easily - To make things easier, we can use a label: *each container behind a load balancer will have a `loadbalancer` label giving the name of that loadbalancer* - See script `reconfigure-load-balancers.py` for an example --- ## Use DNS to discover the addresses of all the backends - When multiple containers have the same network alias: - Engine 1.10 returns only one of them (the same one across the whole network) - Engine 1.11 returns all of them (in a random order) - A "smart" client can use all records to implement load balancing - We can compose `jpetazzo/hamba` with a special-purpose container, which will dynamically generate HAProxy's configuration when the DNS records are updated --- ## Introducing `jpetazzo/watchdns` - [100 lines of pure POSIX scriptery]( https://github.com/jpetazzo/watchdns/blob/master/watchdns) - Resolves a given DNS name every second - Each time the result changes, a new HAProxy configuration is generated - When used together with `--volumes-from` and `jpetazzo/hamba`, it updates the configuration of an existing load balancer - Comes with a companion script, `add-load-balancer-v2.py`, to update your Compose files --- ## Using `jpetazzo/watchdns` .exercise[ - First, revert the Compose file to remove the load balancer - Then, run `add-load-balancer-v2.py`: ```bash ../bin/add-load-balancer-v2.py rng ``` - Inspect the resulting Compose file ] --- ## Scaling with `watchdns` .exercise[ - Start the application with the new sidekick containers: ```bash docker-compose up -d ``` - Scale `rng`: ```bash docker-compose scale rng=10 ``` - Check logs: ```bash docker-compose logs rng-wd ``` ] --- ## Cleaning up .exercise[ - Terminate containers and remove them: ```bash docker-compose down ``` ] Note: `docker-compose down` also deletes the networks that had been created for the application. --- ## Comments - This is a very crude implementation of the pattern - A Go version would only be a bit longer, but use much less resources - When there are many backends, reacting quickly to change is less important (i.e. it's not necessary to re-resolve records every second!) ??? # Going further Deploying a new version (difficulty: easy) - Just re-run all the steps! - However, Compose will re-create the containers - You will have to re-create ambassadors <br/>(and configure them) - You will have to cleanup old ambassadors <br/>(left as an exercise for the reader) - You will experience a little bit of downtime ??? ## Going further Zero-downtime deployment (difficulty: medium) - Isolate stateful services (like we did earlier for Redis) - Do blue/green deployment: - deploy and scale version N - point a "top-level" load balancer to the app - deploy and scale version N+1 - put both apps in the "top-level" balancer - slowly switch traffic over to app version N+1 ??? ## Going further Harder projects: - Two-tier or three-tier ambassador deployments - Deploy to Mesos or Kubernetes --- class: title Resiliency <br/> and <br/> high availability --- # Distributing Machine credentials - All the credentials (TLS keys and certs) are on node1 <br/>(the node on which we ran `docker-machine create`) - If we lose node1, we're toast - We need to move (or copy) the credentials somewhere safe - Credentials are regular files, and relatively small - Ah, if only we had a highly available, hierarchic store ... -- - Wait a minute, we have one! -- (That's Consul, if you were wondering) --- ## Storing files in Consul - We will use [Benjamin Wester's consulfs]( https://github.com/bwester/consulfs) - It mounts a Consul key/value store as a local filesystem - Performance will be horrible <br/>(don't run a database on top of that!) - But to store files of a few KB, nobody will notice - We will copy/link/sync... `~/.docker/machine` to Consul --- ## Installing consulfs - Option 1: install Go, git clone, go build ... - Option 2: be lazy and use [jpetazzo/consulfs]( https://hub.docker.com/r/jpetazzo/consulfs/) .exercise[ - Be lazy and use the Docker image: ```bash docker run --rm -v /usr/local/bin:/target jpetazzo/consulfs ``` ] Note: the `jpetazzo/consulfs` image contains the `consulfs` binary. It copies it to `/target` (if `/target` is a volume). --- ## Can't we run consulfs in a container? - Yes we can! - The filesystem will be mounted in the container - It won't be visible outside of the container (from the host) - We can use *shared mounts* to propagate mounts from containers to Docker - But propagating from Docker to the host requires particular systemd flags - ... So we'll run it on the host for now --- ## Running consulfs - The `consulfs` binary takes two arguments: - the Consul server address - a mount point (that has to be created first) .exercise[ - Create a mount point and mount Consul as a local filesystem: ```bash mkdir ~/consul consulfs localhost:8500 ~/consul ``` ] Leave this running in the foreground. --- ## Checking our consulfs mount point - All key/values will be visible: - Swarm discovery - overlay networks - ... anything you put in Consul! .exercise[ - Check that Consul key/values are visible: ```bash ls -l ~/consul/ ``` ] --- ## Copying our credentials to Consul - Use standard UNIX commands - Don't try to preserve permissions, though (`consulfs` doesn't store permissions) .exercise[ - Copy Machine credentials into Consul: ```bash cp -r ~/.docker/machine/. ~/consul/machine/ ``` ] (This command can be re-executed to update the copy.) --- ## Mount Consul on another node - We will repeat the previous steps to mount `~/consul` .exercise[ - Connect to node2: ```bash ssh node2 ``` - Install `consulfs` and mount Consul: ```bash docker run --rm -v /usr/local/bin:/target jpetazzo/consulfs mkdir ~/consul consulfs localhost:8500 ~/consul & ``` ] At this point, `ls -l ~/consul` should show `docker` and `machine` directories. --- ## Access the credentials from the other node - We will create a symlink - We could also copy the credentials .exercise[ - Create the symlink: ```bash mkdir -p ~/.docker/ ln -s ~/consul/machine ~/.docker/ ``` - Check that all nodes are visible: ```bash docker-machine ls ``` ] .warning[Go back to node1 after this.] --- ## A few words on this strategy - Anyone accessing Consul can control your Docker cluster <br/>(to be fair: anyone accessing Consul can wreck serious havoc to your cluster anyway) - ConsulFS doesn't support *all* POSIX operations, so a few things (like `mv`) will not work) - As a consequence, with Machine 0.6, you cannot run `docker-machine create` directly on top of ConsulFS --- ## What if Consul becomes unavailable? - If Consul becomes unavailable (e.g. loses quorum), <br/>you won't be able to access your credentials - If Consul becomes unavailable ... <br/>your cluster will be in a bad state anyway - You can still access each Docker Engine over the local UNIX socket <br/>(and repair Consul that way) --- # Highly available Swarm managers - Until now, the Swarm manager was a SPOF <br/>(Single Point Of Failure) - Swarm has support for replication - When replication is enabled, you deploy multiple (identical) managers - one will be "primary" - the other(s) will be "secondary" - this is determined automatically <br/>(through *leader election*) --- ## Swarm leader election - The leader election mechanism relies on a key/value store <br/>(consul, etcd, zookeeper) - There is no requirement on the number of replicas <br/>(the quorum is achieved through the key/value store) - When the leader (or "primary") is unavailable, <br/>a new election happens automatically - You can issue API requests to any manager: <br/>if you talk to a secondary, it forwards to the primary .warning[There is currently a bug when the Consul cluster itself has a leader election; <br/>see [docker/swarm#1782](https://github.com/docker/swarm/issues/1782).] --- ## Swarm replication in practice - We need to give two extra flags to the Swarm manager: - `--replication` *enables replication (duh!)* - `--advertise ip.ad.dr.ess:port` *address and port where this Swarm manager is reachable* - Do you deploy with Docker Machine? <br/>Then you can use `--swarm-opt` to automatically pass flags to the Swarm manager --- ## Cleaning up our current Swarm containers - We will use Docker Machine to re-provision Swarm - We need to: - remove the nodes from the Machine registry - remove the Swarm containers .exercise[ - Remove the current configuration: ```bash for N in 1 2 3 4 5; do ssh node$N docker rm -f swarm-agent swarm-agent-master docker-machine rm -f node$N done ``` ] --- ## Re-deploy with the new configuration - This time, all nodes can be deployed identically <br/>(instead of 1 manager + 4 non-managers) .exercise[ ```bash grep node[12345] /etc/hosts | grep -v ^127 | while read IPADDR NODENAME; do docker-machine create --driver generic \ --engine-opt cluster-store=consul://localhost:8500 \ --engine-opt cluster-advertise=eth0:2376 \ --swarm --swarm-master \ --swarm-discovery consul://localhost:8500 \ --swarm-opt replication --swarm-opt advertise=$IPADDR:3376 \ --generic-ssh-user docker --generic-ip-address $IPADDR $NODENAME done ``` ] .small[ Note: Consul is still running thanks to the `--restart=always` policy. Other containers are now stopped, because the engines have been reconfigured and restarted. ] --- ## Assess our new cluster health - The output of `docker info` will tell us the status of the node that we are talking to (primary or replica) - If we talk to a replica, it will tell us who is the primary .exercise[ - Talk to a random node, and ask its view of the cluster: ```bash eval $(docker-machine env node3 --swarm) docker info | grep -e ^Name -e ^Role -e ^Primary ``` ] Note: `docker info` is one of the only commands that will work even when there is no elected primary. This helps debugging. --- ## Test Swarm manager failover - The previous command told us which node was the primary manager - if `Role` is `primary`, <br/>then the primary is indicated by `Name` - if `Role` is `replica`, <br/>then the primary is indicated by `Primary` .exercise[ - Kill the primary manager: ```bash ssh node`N` docker kill swarm-agent-master ``` ] Look at the output of `docker info` every few seconds. --- # Highly available containers - Swarm has support for *rescheduling* on node failure - It has to be explicitly enabled on a per-container basis - When the primary manager detects that a node goes down, <br/>those containers are rescheduled elsewhere - If the containers can't be rescheduled (constraints issue), <br/>they are lost (there is no reconciliation loop yet) - In Swarm 1.1, this is an *experimental* feature <br/>(To enable it, you must pass the `--experimental` flag when you start Swarm itself!) - In Swarm 1.2, you don't need the `--experimental` flag anymore --- ## About Swarm generic flags - Some flags like `--experimental` and `--debug` must be *before* the Swarm command <br/>(i.e. `docker run swarm --debug manage ...`) - We cannot use Docker Machine to pass that flag ☹ <br/>(Machine adds flags *after* the Swarm command) - Instead, we can use a custom Swarm image: ```dockerfile FROM swarm ENTRYPOINT ["/swarm", "--debug"] ``` - We can tell Machine to use this with `--swarm-image` --- ## Start a resilient container - By default, containers will not be restarted when their node goes down - You must pass an explicit *rescheduling policy* to make that happen - For now, the only policy is "on-node-failure" .exercise[ - Start a container with a rescheduling policy: ```bash docker run --name highlander -d -e reschedule:on-node-failure nginx ``` ] Check that the container is up and running. --- ## Simulate a node failure - We will reboot the node running this container - Swarm will reschedule it .exercise[ - Check on which node the container is running: </br>`NODE=$(docker inspect --format '{{.Node.Name}}' highlander)` - Reboot that node: <br/>`ssh $NODE sudo reboot` - Check that the container has been recheduled: <br/>`docker ps -a` ] --- ## Reboots - When rebooting a node, Docker is stopped cleanly, and containers are stopped - Our container is rescheduled, but not started - To simulate a "proper" failure, we can use the Chaos Monkey script instead ```bash ~/orchestration-workshop/bin/chaosmonkey $NODE <connect|disconnect|reboot> ``` --- ## Cluster reconciliation - After the cluster rejoins, we can end up with duplicate containers .exercise[ - Once the node is back, remove one of the extraneous containers: ```bash docker rm -f node`N`/highlander ``` ] --- ## .warning[Caveats] - There are some corner cases when the node is also the Swarm leader or the Consul leader; this is being improved right now! - The safest way to address for now this is to run the Consul servers, the Swarm managers, and your containers, on different nodes. - Swarm doesn't handle gracefully the fact that after the reboot, you have *two* containers named `highlander`, and attempts to manipulate the container with its name will not work. This will be improved too. --- # Conclusions - Bad news: we still have work to do to deploy our apps - it's not all unicorns, ponies, and rainbows - *no, Docker will not make your job obsolete* - Good news: a lot of hard things are becoming easier - building, packaging, distributing apps - running distributed systems on clusters --- ## What's next? - November 2015: Compose 1.5 + Engine 1.9 = <br/>first release with multi-host networking - January 2016: Compose 1.6 + Engine 1.10 = <br/>embedded DNS server, experimental high availability - April 2016: Compose 1.7 + Engine 1.11 = <br/>round robin DNS records, huge improvements in HA - Next release: another truckload of features - I will deliver this workshop about twice a month - Check out the GitHub repo for updated content! <br/>(there is a tag for each big round of updates) --- ## High availability - Docker in general, and Swarm in particular, move *fast* - Current high availability features are not Chaos-Monkey proof (yet) - We (well, the Swarm team) is working to change that --- ## Overall complexity - The scripts used here are pretty simple (each is less than 100 LOCs) - You can easily rewrite them in your favorite language, <br/>adapt and customize them, in a few hours of time - FYI: those scripts are smaller and simpler than the scripts (cloud init etc) used to deploy the VMs for this workshop! - Docker Inc. has commercial products to wrap all this: - Docker Cloud <br/>(manage your Docker nodes from a SAAS portal) - Docker Datacenter <br/>(buzzword-compliant management solution: <br/>turnkey, enterprise-class, on-premise, etc.) --- class: title # Thanks! <br/> Questions? ## [@jpetazzo](https://twitter.com/jpetazzo) <br/> [@docker](https://twitter.com/docker)