From 4d2e62ffee6843097fac48d93c481e330e12b8e2 Mon Sep 17 00:00:00 2001 From: Jerome Petazzoni Date: Tue, 1 Mar 2016 05:21:02 -0800 Subject: [PATCH] Adjust for QCON schedule; use local registry --- README.md | 5 + .../delete-ambassadors.sh | 0 fixup-yaml.sh => bin/fixup-yaml.sh | 0 dockercoins/docker-compose.yml-ambassador | 4 +- dockercoins/docker-compose.yml-extra-hosts | 4 +- dockercoins/webui/files/index.html | 2 +- prepare-vms/postprep.rc | 4 +- www/htdocs/index.html | 260 +++++++++++++----- 8 files changed, 199 insertions(+), 80 deletions(-) rename delete-ambassadors.sh => bin/delete-ambassadors.sh (100%) rename fixup-yaml.sh => bin/fixup-yaml.sh (100%) diff --git a/README.md b/README.md index 23515262..ac8bcf03 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,11 @@ at multiple conferences and events like: - KCDC, Kansas City (2015, June) - JDEV, Bordeaux (2015, July) - OSCON, Portland (2015, July) +- StrangeLoop, Saint Louis (2015, September) +- LISA, Washington D.C. (2015, November) +- SCALE, Pasadena (2016, January) +- Zenika, Paris (2016, February) +- Container Solutions, Amsterdam (2016, February) ## Slides diff --git a/delete-ambassadors.sh b/bin/delete-ambassadors.sh similarity index 100% rename from delete-ambassadors.sh rename to bin/delete-ambassadors.sh diff --git a/fixup-yaml.sh b/bin/fixup-yaml.sh similarity index 100% rename from fixup-yaml.sh rename to bin/fixup-yaml.sh diff --git a/dockercoins/docker-compose.yml-ambassador b/dockercoins/docker-compose.yml-ambassador index 9eaf5bd7..2150134f 100644 --- a/dockercoins/docker-compose.yml-ambassador +++ b/dockercoins/docker-compose.yml-ambassador @@ -28,8 +28,8 @@ webui: - redis ports: - "8000:80" - #volumes: - # - "./webui/files/:/files/" + volumes: + - "./webui/files/:/files/" redis: image: jpetazzo/hamba diff --git a/dockercoins/docker-compose.yml-extra-hosts b/dockercoins/docker-compose.yml-extra-hosts index f27d7afb..6ea98cd3 100644 --- a/dockercoins/docker-compose.yml-extra-hosts +++ b/dockercoins/docker-compose.yml-extra-hosts @@ -28,8 +28,8 @@ webui: redis: A.B.C.D ports: - "8000:80" - #volumes: - # - "./webui/files/:/files/" + volumes: + - "./webui/files/:/files/" #redis: # image: redis diff --git a/dockercoins/webui/files/index.html b/dockercoins/webui/files/index.html index b68ef4a3..accd2c23 100644 --- a/dockercoins/webui/files/index.html +++ b/dockercoins/webui/files/index.html @@ -50,7 +50,7 @@ function refresh () { points.push({ x: s2.now, y: speed }); } $("#speed").text("~" + speed.toFixed(1) + " hashes/second"); - var msg = ("I'm attending the @docker Swarm workshop at @ZenikaIT, " + var msg = ("I'm attending the @docker Swarm workshop at @qconlondon, " + "and my #DockerCoins mining rig is crunching " + speed.toFixed(1) + " hashes/second! W00T!"); $("#tweet").attr( diff --git a/prepare-vms/postprep.rc b/prepare-vms/postprep.rc index 15f826dd..e0452dfe 100755 --- a/prepare-vms/postprep.rc +++ b/prepare-vms/postprep.rc @@ -1,8 +1,8 @@ pssh -I tee /tmp/postprep.py < @@ -116,8 +116,15 @@ class: title - All the content is publicly available
(slides, code samples, scripts) + + - Experimental chat support on - [Gitter](https://gitter.im/jpetazzo/workshop-20160219-amsterdam) + [Gitter](https://gitter.im/jpetazzo/workshop-20160311-london) --- @@ -181,17 +188,39 @@ grep '^# ' index.html | grep -v '(on Windows, get [putty](http://www.putty.org/) or [Git BASH](https://msysgit.github.io/)) -- GitHub account (recommended; not mandatory) -- Gitter account (recommended; not mandatory) -- Docker Hub account (only for Swarm hands-on section) + +- [GitHub](https://github.com/join) account +
(if you want to fork the repo; also used to join Gitter) + + +- [Gitter](https://gitter.im/) account +
(to join the conversation during the workshop) + +- [Docker Hub](https://hub.docker.com) account +
(it's one way to distribute images on your Swarm cluster) + - Basic Docker knowledge +
(but that's OK if you're not a Docker expert!) + +--- + +## Hands-on sections + +- The whole workshop is hands-on + +- I will show Docker in action + +- I invite you to reproduce what I do + +- All hands-on sections are clearly identified +
(see below) .exercise[ - This is the stuff you're supposed to do! - Go to [container.training](http://container.training/) to view these slides - Join the chat room on - [Gitter](https://gitter.im/jpetazzo/workshop-20160219-amsterdam) + [Gitter](https://gitter.im/jpetazzo/workshop-20160311-london) ] @@ -251,11 +280,11 @@ If you want to use screen or whatever, you're welcome! ## Brand new versions! -- Engine 1.10.1 +- Engine 1.10.2 -- Compose 1.6.0 +- Compose 1.6.2 -- Swarm 1.1.0 +- Swarm 1.1.3 - Machine 0.6.0 @@ -484,6 +513,8 @@ We have multiple options: - Kill all services with `docker-compose kill`
(rude, but faster!) +- Stop and remove all services with `docker-compose down` + .exercise[ - Use any of those methods to stop `rng` and `hasher` @@ -817,10 +848,10 @@ We will use `httping`. - Scale back the `worker` service to zero:
`docker-compose scale worker=0` -- Open a new SSH connection and check the latency of `rng`: +- Open a new terminal and check the latency of `rng`:
`httping localhost:8001` -- Open a new SSH conection and do the same for `hasher`: +- Open a new terminal and do the same for `hasher`:
`httping localhost:8002` - Keep an eye on both connections! @@ -1344,37 +1375,14 @@ Shortcut: `docker-compose.yml-extra-hosts` redis: A.B.C.D ports: - "8000:80" - #volumes: - # - "./webui/files/:/files/" + volumes: + - "./webui/files/:/files/" ``` ] (Replace `A.B.C.D` with the IP address noted earlier) -.icon[![Warning](warning.png)] Don't forget to comment out the `volumes` section! - ---- - -## Why did we comment out the `volumes` section? - -- Volumes have multiple uses: - - - storing persistent stuff (database files...) - - - sharing files between containers (logs, configuration...) - - - sharing files between host and containers (source...) - -- The `volumes` directive expands to an host path -
.small[(e.g. `/home/docker/orchestration-workshop/dockercoins/webui/files`)] - -- This host path exists on the local machine -
(not on the others) - -- This specific volume is used in development -
(not in production) - --- ## Start the stack on the first machine @@ -1420,13 +1428,20 @@ Shortcut: `docker-compose.yml-extra-hosts` .exercise[ -- Open the Web UI -
(on a node where it's deployed) +- Keep an eye on the web UI -- Deploy one instance of the stack on each node +- Create 20 workers on both nodes: + ``` + for NODE in node1 node2; do + export DOCKER_HOST=tcp://$NODE:55555 + docker-compose scale worker=20 + done + ``` ] +Note: of course, if we wanted, we could run on all five nodes. + --- ## Cleanup @@ -1509,13 +1524,6 @@ Shortcut: `docker-compose.yml-extra-hosts` command: 6379 AA.BB.CC.DD EEEEE ``` -- Make sure that the `volumes` section is commented in `webui`: - - ``` - #volumes: - # - "./webui/files/:/files/" - ``` - ] Shortcut: `docker-compose.yml-ambassador` @@ -1638,6 +1646,30 @@ Let's celebrate our success! --- +## Stop the app (but leave Redis running) + +- Let's use `docker-compose down` + +- It will stop and remove the DockerCoins app +
(but leave other containers running) + +.exercise[ + +- We can do another simple shell loop: + ``` + for N in $(seq 1 5); do + export DOCKER_HOST=tcp://node$N:55555 + docker-compose down & + done + ``` + +] + +(We need to keep the `myredis` container for +our next section, which will be about backups!) + +--- + # Various considerations about ambassadors - "But, ambassadors are adding an extra hop!" @@ -2235,9 +2267,10 @@ of this workshop. - scale ElasticSearch - scale Logstash - - (probably) switch away from GELF+UDP + - move away from UDP *or* put one Logstash per node + - interpose a Redis or Kafka queue -- Configure your Engines to send all logs to the stack by default +- Configure your Engines to send all logs to ELK by default - Start the logging containers with a different logging system
(to avoid a logging loop) @@ -2263,7 +2296,6 @@ of this workshop. ## Upgrading the Docker daemon - Stop all containers cleanly -
(`docker ps -q | xargs docker stop`) - Stop the Docker daemon @@ -2278,6 +2310,23 @@ of this workshop. --- +## In practice + +- Keep track of running containers before stopping the Engine: + ``` + docker ps --no-trunc -q | + tee /tmp/running | + xargs -n1 -P10 docker stop + ``` + +- Restart those containers after the Engine is running again: + ``` + xargs docker start < /tmp/running + ``` +
(Run this multiple times if you have linked containers!) + +--- + ## Upgrading container images - When a vulnerability is announced: @@ -2294,7 +2343,11 @@ of this workshop. - restart containers -(The procedure is simple and plain, just follow it!) +Compose makes this particularly easy: +``` +docker-compose build --pull --no-cache +docker-compose up -d +``` --- @@ -2611,7 +2664,7 @@ in the discovery service hosted by Docker Inc. .exercise[ -- Create your token, saving it preciusly to disk as well: +- Create your token, saving it preciously to disk as well: ``` TOKEN=$(docker run swarm create | tee token) @@ -2991,8 +3044,7 @@ Before trying to build our app, we will remove previous images. - Tag images -- Upload them to the hub -
(Note: this part requires a Docker Hub account!) +- Upload them to a registry - Update the Compose file to use those images @@ -3000,26 +3052,90 @@ Before trying to build our app, we will remove previous images. --- -## Docker Hub account +## Which registry do we want to use? -- You need a Docker Hub account for that part +.small[ -- If you don't have one, create it +- **Docker Hub** + + - hosted by Docker Inc. + - requires an account (free, no credit card needed) + - images will be public (unless you pay) + - located in AWS EC2 us-east-1 + +- **Docker Trusted Registry** + + - self-hosted commercial product + - requires a subscription (free 30-day trial available) + - images can be public or private + - located wherever you want + +- **Docker open source registry** + + - self-hosted barebones repository hosting + - doesn't require anything + - doesn't come with anything either + - located wherever you want +] + +--- + +## Using Docker Hub + +- To tell `build-tag-push.py` to use Docker Hub, +
set the `DOCKER_REGISTRY` environment variable +
to your Docker Hub user name + +- We will also see how to run the open source registry +
(so use whatever option you want!) .exercise[ - Set the following environment variable: - - ``` - export DOCKERHUB_USER=jpetazzo - ``` +
`export DOCKER_REGISTRY=jpetazzo` - (Use *your* Docker Hub login, of course!) - Log into the Docker Hub: +
`docker login` +] + +--- + +## Using Docker Trusted Registry + +If we wanted to use DTR, we would: + +- make sure we have a Docker Hub account +- [activate a Docker Datacenter subscription]( + https://hub.docker.com/enterprise/trial/) +- install DTR on our machines +- set `DOCKER_REGISTRY` to `dtraddress:port/user` + +*This is out of the scope of this workshop!* + +--- + +## Using open source registry + +- All we need to do is to run a `registry:2` container +
(make sure you specify tag `:2` to run the new version!) + +- It will store images and layers to the local filesystem +
(but you can add a config file to use S3, Swift, etc.) + +.exercise[ + +- Start your registry on your Swarm cluster: ``` - docker login + eval $(docker-machine env node1 --swarm) + docker run -dP --name registry registry:2 + ``` + +- Set the `DOCKER_REGISTRY` variable: + ``` + export DOCKER_REGISTRY=$(docker port registry 5000) ``` ] @@ -3030,17 +3146,15 @@ Before trying to build our app, we will remove previous images. Let's inspect the source code of `build-tag-push.py` and run it. -.icon[![Warning](warning.png)] It is better to run it against a single node! - -(There are some race conditions within Swarm when building+pushing too fast.) +.icon[![Warning](warning.png)] Make sure to run it against a single node! .exercise[ - Point to a single node:
`eval $(docker-machine env node1)` -- Run the script (from the `dockercoins` directory): -
`../build-tag-push.py` +- Run the script (from the `bin` directory): +
`../bin/build-tag-push.py` - Inspect the `docker-compose.yml-XXX` file that it created