Add slides about rockstars

slides: code block comment formatting on snap install

README.md

@@ -1,8 +1,253 @@
-# Orchestration at scale(s)
+# Docker Orchestration Workshop
 
-This is the material for the "Docker orchestration workshop"
-written and delivered by Jérôme Petazzoni (and possibly others)
-at multiple conferences and events like:
+This is the material (slides, scripts, demo app, and other
+code samples) for the "Docker orchestration workshop"
+written and delivered by Jérôme Petazzoni (and lots of others)
+non-stop since June 2015.
+
+
+## Content
+
+- Chapter 1: Getting Started: running apps with docker-compose
+- Chapter 2: Scaling out with Swarm Mode
+- Chapter 3: Operating the Swarm (networks, updates, logging, metrics)
+- Chapter 4: Deeper in Swarm (stateful services, scripting, DAB's)
+
+
+## Quick start (or, "I want to try it!")
+
+This workshop is designed to be *hands on*, i.e. to give you a step-by-step
+guide where you will build your own Docker cluster, and use it to deploy
+a sample application.
+
+The easiest way to follow the workshop is to attend it when it is delivered
+by an instructor. In that case, the instructor will generally give you
+credentials (IP addresses, login, password) to connect to your own cluster
+of virtual machines; and the [slides](http://jpetazzo.github.io/orchestration-workshop)
+assume that you have your own cluster indeed.
+
+If you want to follow the workshop on your own, and want to have your
+own cluster, we have multiple solutions for you!
+
+
+### Using [play-with-docker](http://play-with-docker.com/)
+
+This method is very easy to get started (you don't need any extra account
+or resources!) but will require a bit of adaptation from the workshop slides.
+
+To get started, go to [play-with-docker](http://play-with-docker.com/), and
+click on _ADD NEW INSTANCE_ five times. You will get five "docker-in-docker"
+containers, all on a private network. These are your five nodes for the workshop!
+
+When the instructions in the slides tell you to "SSH on node X", just go to
+the tab corresponding to that node.
+
+The nodes are not directly reachable from outside; so when the slides tell
+you to "connect to the IP address of your node on port XYZ" you will have
+to use a different method.
+
+We suggest to use "supergrok", a container offering a NGINX+ngrok combo to
+expose your services. To use it, just start (on any of your nodes) the
+`jpetazzo/supergrok` image. The image will output further instructions:
+
+```
+docker run --name supergrok -d jpetazzo/supergrok
+docker logs --follow supergrok
+```
+
+The logs of the container will give you a tunnel address and explain you
+how to connected to exposed services. That's all you need to do!
+
+We are also working on a native proxy, embedded to Play-With-Docker.
+Stay tuned!
+
+<!--
+
+- You can use a proxy provided by Play-With-Docker. When the slides
+  instruct you to connect to nodeX on port ABC, instead, you will connect
+  to http://play-with-docker.com/XXX.XXX.XXX.XXX:ABC, where XXX.XXX.XXX.XXX
+  is the IP address of nodeX.
+
+-->
+
+Note that the instances provided by Play-With-Docker have a short lifespan
+(a few hours only), so if you want to do the workshop over multiple sessions,
+you will have to start over each time ... Or create your own cluster with
+one of the methods described below.
+
+
+### Using Docker Machine to create your own cluster
+
+This method requires a bit more work to get started, but you get a permanent
+cluster, with less limitations.
+
+You will need Docker Machine (if you have Docker Mac, Docker Windows, or
+the Docker Toolbox, you're all set already). You will also need:
+
+- credentials for a cloud provider (e.g. API keys or tokens),
+- or a local install of VirtualBox or VMware (or anything supported
+  by Docker Machine).
+
+Full instructions are in the [prepare-machine](prepare-machine) subdirectory.
+
+
+### Using our scripts to mass-create a bunch of clusters
+
+Since we often deliver the workshop during conferences or similar events,
+we have scripts to automate the creation of a bunch of clusters using
+AWS EC2. If you want to create multiple clusters and have EC2 credits,
+check the [prepare-vms](prepare-vms) directory for more information.
+
+
+## How This Repo is Organized
+
+- **dockercoins**
+  - Sample App: compose files and source code for the dockercoins sample apps
+  used throughout the workshop
+- **docs**
+  - Slide Deck: presentation slide deck, works out-of-box with GitHub Pages,
+  uses https://remarkjs.com
+- **prepare-local**
+  - untested scripts for automating the creation of local virtualbox VM's
+  (could use your help validating)
+- **prepare-machine**
+  - instructions explaining how to use Docker Machine to create VMs
+- **prepare-vms**
+  - scripts for automating the creation of AWS instances for students
+
+
+## Slide Deck
+
+- The slides are in the `docs` directory.
+- To view them locally open `docs/index.html` in your browser. It works
+  offline too.
+- To view them online open https://jpetazzo.github.io/orchestration-workshop/
+  in your browser.
+- When you fork this repo, be sure GitHub Pages is enabled in repo Settings
+  for "master branch /docs folder" and you'll have your own website for them.
+- They use https://remarkjs.com to allow simple markdown in a html file that
+  remark will transform into a presentation in the browser.
+
+
+## Sample App: Dockercoins!
+
+The sample app is in the `dockercoins` directory. It's used during all chapters
+for explaining different concepts of orchestration.
+
+To see it in action:
+
+- `cd dockercoins && docker-compose up -d`
+- this will build and start all the services
+- the web UI will be available on port 8000
+
+
+*If you just want to run the workshop for yourself, you can stop reading
+here. If you want to deliver the workshop for others (i.e. if you
+want to become an instructor), keep reading!*
+
+
+## Running the Workshop
+
+
+### General timeline of planning a workshop
+
+- Fork repo and run through slides, doing the hands-on to be sure you
+  understand the different `dockercoins` repo's and the steps we go through to
+  get to a full Swarm Mode cluster of many containers. You'll update the first
+  few slides and last slide at a minimum, with your info.
+- Your docs directory can use GitHub Pages.
+- This workshop expects 5 servers per student. You can get away with as little
+  as 2 servers per student, but you'll need to change the slide deck to
+  accommodate. More servers = more fun.
+- If you have more then ~20 students, try to get an assistant (TA) to help
+  people with issues, so you don't have to stop the workshop to help someone
+  with ssh etc.
+- AWS is our most tested process for generating student machines. In
+  `prepare-vms` you'll find scripts to create EC2 instances, install docker,
+  pre-pull images, and even print "cards" to place at each students seat with
+  IP's and username/password.
+- Test AWS Scripts: Be sure to test creating *all* your needed servers a week
+  before workshop (just for a few minutes). You'll likely hit AWS limits in the
+  region closest to your class, and it sometimes takes days to get AWS to raise
+  those limits with a support ticket.
+- Create a https://gitter.im chat room for your workshop and update slides
+  with url. Also useful for TA to monitor this during workshop. You can use it
+  before/after to answer questions, and generally works as a better answer then
+  "email me that question".
+- If you can send an email to students ahead of time, mention how they should
+  get SSH, and test that SSH works. If they can `ssh github.com` and get
+  `permission denied (publickey)` then they know it worked, and SSH is properly
+  installed and they don't have anything blocking it. SSH and a browser are all
+  they need for class.
+- Typically you create the servers the day before or morning of workshop, and
+  leave them up the rest of day after workshop. If creating hundreds of servers,
+  you'll likely want to run all these `trainer` commands from a dedicated
+  instance you have in same region as instances you want to create. Much faster
+  this way if you're on poor internet. Also, create 2 sets of servers for
+  yourself, and use one during workshop and the 2nd is a backup.
+- Remember you'll need to print the "cards" for students, so you'll need to
+  create instances while you have a way to print them.
+
+### Things That Could Go Wrong
+
+- Creating AWS instances ahead of time, and you hit its limits in region and
+  didn't plan enough time to wait on support to increase your limits. :(
+- Students have technical issues during workshop. Can't get ssh working,
+  locked-down computer, host firewall, etc.
+- Horrible wifi, or ssh port TCP/22 not open on network! If wifi sucks you
+  can try using MOSH https://mosh.org which handles SSH over UDP. TMUX can also
+  prevent you from loosing your place if you get disconnected from servers.
+  https://tmux.github.io
+- Forget to print "cards" and cut them up for handing out IP's.
+- Forget to have fun and focus on your students!
+
+### Creating the VMs
+
+`prepare-vms/trainer` is the script that gets you most of what you need for
+setting up instances. See
+[prepare-vms/README.md](prepare-vms)
+for all the info on tools and scripts.
+
+### Content for Different Workshop Durations
+
+With all the slides, this workshop is a full day long. If you need to deliver
+it in shorter timelines, here's some recommendations on what to cut out. You
+can replace `---` with `???` which will hide slides. Or leave them there and
+add something like `(EXTRA CREDIT)` to title so students can still view the
+content but you also know to skip during presentation.
+
+#### 3 Hour Version
+
+- Limit time on debug tools, maybe skip a few. *"Chapter 1:
+  Identifying bottlenecks"*
+- Limit time on Compose, try to have them building the Swarm Mode by 30
+  minutes in
+- Skip most of Chapter 3, Centralized Logging and ELK
+- Skip most of Chapter 4, but keep stateful services and DAB's if possible
+- Mention what DAB's are, but make this part optional in case you run out
+  of time
+
+#### 2 Hour Version
+
+- Skip all the above, and:
+- Skip the story arc of debugging dockercoins all together, skipping the
+  troubleshooting tools. Just focus on getting them from single-host to
+  multi-host and multi-container.
+- Goal is first 30min on intro and Docker Compose and what dockercoins is,
+  and getting it up on one node in docker-compose.
+- Next 60-75 minutes is getting dockercoins in Swarm Mode services across
+  servers. Big Win.
+- Last 15-30 minutes is for stateful services, DAB files, and questions.
+
+
+## Past events
+
+Since its inception, this workshop has been delivered dozens of times,
+to thousands of people, and has continuously evolved. This is a short
+history of the first times it was delivered. Look also in the "tags"
+of this repository: they all correspond to successive iterations of
+this workshop. If you attended a past version of the workshop, you
+can use these tags to see what has changed since then.
 
 - QCON, New York City (2015, June)
 - KCDC, Kansas City (2015, June)
@@ -13,80 +258,7 @@ at multiple conferences and events like:
 - SCALE, Pasadena (2016, January)
 - Zenika, Paris (2016, February)
 - Container Solutions, Amsterdam (2016, February)
-
-
-## Slides
-
-The slides are in the `www/htdocs` directory.
-
-The recommended way to view them is to:
-
-- have a Docker host
-- clone this repository to your Docker host
-- `cd www && docker-compose up -d`
-- this will start a web server on port 80
-- point your browser at your Docker host and enjoy
-
-
-## Sample code
-
-The sample app is in the `dockercoins` directory.
-
-To see it in action:
-
-- `cd dockercoins && docker-compose up -d`
-- this will build and start all the services
-- the web UI will be available on port 8000
-
-
-## Running the workshop
-
-WARNING: those instructions are incomplete. Consider
-them as notes quickly drafted on a napkin rather than
-proper documentation!
-
-
-### Creating the VMs
-
-I use the `trainctl` script from the `docker-fundamentals`
-repository. Sorry if you don't have that!
-
-After starting the VMs, use the `trainctl ips` command
-to dump the list of IP addresses into a file named `ips.txt`.
-
-
-### Generating the printed cards
-
-- Put `ips.txt` file in `prepare-vms` directory.
-- Generate HTML file.
-- Open it in Chrome.
-- Transform to PDF.
-- Print it.
-
-
-### Deploying your SSH key to all the machines
-
-- Make sure that you have SSH keys loaded (`ssh-add -l`).
-- Source `rc`.
-- Run `pcopykey`.
-
-
-### Installing extra packages
-
-- Source `postprep.rc`.
-  (This will install a few extra packages, add entries to
-  /etc/hosts, generate SSH keys, and deploy them on all hosts.)
-
-
-### Final touches
-
-- Set two groups of machines for instructor's use.
-- You will use the first group during the workshop.
-- The second group will run a web server with the slides.
-- Log into the first machine of the second group.
-- Git clone this repo.
-- Put up the web server as instructed above.
-- Use cli53 to add an A record for e.g. `view.dckr.info`.
+- ... and much more!
 
 
 # Problems? Bugs? Questions?
@@ -108,3 +280,4 @@ conference or for your company: contact me (jerome
 at docker dot com).
 
 Thank you!
+

bin/build-tag-push.py

@@ -16,9 +16,11 @@ if not registry:
 # Get the name of the current directory.
 project_name = os.path.basename(os.path.realpath("."))
 
-# Generate a Docker image tag, using the UNIX timestamp.
-# (i.e. number of seconds since January 1st, 1970)
-version = str(int(time.time()))
+# Version used to tag the generated Docker image, using the UNIX timestamp or the given version.
+if "VERSION" not in os.environ:
+    version = str(int(time.time()))
+else:
+    version = os.environ["VERSION"]
 
 # Execute "docker-compose build" and abort if it fails.
 subprocess.check_call(["docker-compose", "-f", "docker-compose.yml", "build"])
@@ -33,7 +35,7 @@ push_operations = dict()
 for service_name, service in compose_file.services.items():
     if "build" in service:
         compose_image = "{}_{}".format(project_name, service_name)
-        registry_image = "{}/{}_{}:{}".format(registry, project_name, service_name, version)
+        registry_image = "{}/{}:{}".format(registry, compose_image, version)
         # Re-tag the image so that it can be uploaded to the registry.
         subprocess.check_call(["docker", "tag", compose_image, registry_image])
         # Spawn "docker push" to upload the image.

cadvisor/docker-compose.yml

@@ -1,10 +1,12 @@
-cadvisor:
-  image: google/cadvisor
-  ports:
-    - "8080:8080"
-  volumes:
-    - "/:/rootfs:ro"
-    - "/var/run:/var/run:rw"
-    - "/sys:/sys:ro"
-    - "/var/lib/docker/:/var/lib/docker:ro"
+version: "2"
 
+services:
+    cadvisor:
+      image: google/cadvisor
+      ports:
+        - "8080:8080"
+      volumes:
+        - "/:/rootfs:ro"
+        - "/var/run:/var/run:rw"
+        - "/sys:/sys:ro"
+        - "/var/lib/docker/:/var/lib/docker:ro"

dockercoins/docker-compose.yml-ambassador

@@ -1,27 +0,0 @@
-version: "2"
-
-services:
-  rng:
-    build: rng
-    ports:
-    - "8001:80"
-
-  hasher:
-    build: hasher
-    ports:
-    - "8002:80"
-
-  webui:
-    build: webui
-    ports:
-    - "8000:80"
-    volumes:
-    - "./webui/files/:/files/"
-
-  redis:
-    image: jpetazzo/hamba
-    command: 6379 AA.BB.CC.DD:EEEEE
-
-  worker:
-    build: worker
-

dockercoins/docker-compose.yml-images

@@ -1,30 +1,21 @@
 version: "2"
 
 services:
-  rng1:
-    build: rng
-  rng2:
-    build: rng
-  rng3:
-    build: rng
-
   rng:
-    image: jpetazzo/hamba
-    command: 80 rng1:80 rng2:80 rng3:80
-    depends_on:
-    - rng1
-    - rng2
-    - rng3
+    build: rng
+    image: ${REGISTRY_SLASH}rng${COLON_TAG}
     ports:
     - "8001:80"
 
   hasher:
     build: hasher
+    image: ${REGISTRY_SLASH}hasher${COLON_TAG}
     ports:
     - "8002:80"
 
   webui:
     build: webui
+    image: ${REGISTRY_SLASH}webui${COLON_TAG}
     ports:
     - "8000:80"
     volumes:
@@ -35,4 +26,5 @@ services:
 
   worker:
     build: worker
+    image: ${REGISTRY_SLASH}worker${COLON_TAG}
 

dockercoins/docker-compose.yml-portmap

@@ -1,26 +0,0 @@
-version: "2"
-
-services:
-  rng:
-    build: rng
-    ports:
-    - "80"
-
-  hasher:
-    build: hasher
-    ports:
-    - "8002:80"
-
-  webui:
-    build: webui
-    ports:
-    - "8000:80"
-    volumes:
-    - "./webui/files/:/files/"
-
-  redis:
-    image: redis
-
-  worker:
-    build: worker
-

dockercoins/docker-compose.yml-v2

@@ -1,20 +0,0 @@
-version: '2'
-
-services:
-  rng:
-    build: rng
-
-  hasher:
-    build: hasher
-
-  webui:
-    build: webui
-    ports:
-    - "8000:80"
-
-  redis:
-    image: redis
-
-  worker:
-    build: worker
-

dockercoins/ports.yml

@@ -1,5 +0,0 @@
-hasher: 80
-redis: 6379
-rng: 80
-webui: 80
-

dockercoins/webui/files/index.html

@@ -50,7 +50,7 @@ function refresh () {
             points.push({ x: s2.now, y: speed });
         }
         $("#speed").text("~" + speed.toFixed(1) + " hashes/second");
-        var msg = ("I'm attending the @docker workshop at @scaleconf, "
+        var msg = ("I'm attending the @docker workshop at #LinuxCon, "
                    + "and my #DockerCoins mining rig is crunching "
                    + speed.toFixed(1) + " hashes/second! W00T!");
         $("#tweet").attr(

docs/chat/index.html

@@ -0,0 +1,9 @@
+<html>
+<!-- Generated with index.html.sh -->
+<head>
+<meta http-equiv="refresh" content="0; URL='https://gitter.im/jpetazzo/workshop-20170322-sanjose'" />
+</head>
+<body>
+<a href="https://gitter.im/jpetazzo/workshop-20170322-sanjose">https://gitter.im/jpetazzo/workshop-20170322-sanjose</a>
+</body>
+</html>

docs/chat/index.html.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+LINK=https://gitter.im/jpetazzo/workshop-20170322-sanjose
+#LINK=https://dockercommunity.slack.com/messages/captains
+#LINK=https://usenix-lisa.slack.com/messages/docker
+sed "s,@@LINK@@,$LINK,g" >index.html <<EOF
+<html>
+<!-- Generated with index.html.sh -->
+<head>
+<meta http-equiv="refresh" content="0; URL='$LINK'" />
+</head>
+<body>
+<a href="$LINK">$LINK</a>
+</body>
+</html>
+EOF
+

docs/extract-section-titles.py

@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+"""
+Extract and print level 1 and 2 titles from workshop slides.
+"""
+
+separators = [
+    "---",
+    "--"
+]
+
+slide_count = 1
+for line in open("index.html"):
+    line = line.strip()
+    if line in separators:
+        slide_count += 1
+    if line.startswith('#  '):
+        print slide_count, '# #', line
+    elif line.startswith('# '):
+        print slide_count, line

efk/README.md

@@ -0,0 +1,36 @@
+# Elasticsearch + Fluentd + Kibana
+
+This is a variation on the classic "ELK" stack.
+
+The [fluentd](fluentd/) subdirectory contains a Dockerfile to build
+a fluentd image embarking a simple configuration file, accepting log
+entries on port 24224 and storing them in Elasticsearch in a format
+that Kibana can use.
+
+You can also use a pre-built image, `jpetazzo/fluentd:v0.1`
+(e.g. if you want to deploy on a cluster and don't want to deploy
+your own registry).
+
+Once this fluentd container is running, and assuming you expose
+its port 24224/tcp somehow, you can send container logs to fluentd
+by using Docker's fluentd logging driver.
+
+You can bring up the whole stack with the associated Compoes file.
+With Swarm mode, you can bring up the whole stack like this:
+
+```bash
+docker network create efk --driver overlay
+docker service create --network efk \
+       --name elasticsearch elasticsearch:2
+docker service create --network efk --publish 5601:5601 \
+       --name kibana kibana
+docker service create --network efk --publish 24224:24224 \
+       --name fluentd jpetazzo/fluentd:v0.1
+```
+
+And then, from any node on your cluster, you can send logs to fluentd like this:
+
+```bash
+docker run --log-driver fluentd --log-opt fluentd-address=localhost:24224 \
+  alpine echo ohai there
+```

efk/docker-compose.yml

@@ -0,0 +1,24 @@
+version: "2"
+
+services:
+  elasticsearch:
+    image: elasticsearch
+    # If you need to access ES directly, just uncomment those lines.
+    #ports:
+    #  - "9200:9200"
+    #  - "9300:9300"
+
+  fluentd:
+    #build: fluentd
+    image: jpetazzo/fluentd:v0.1
+    ports:
+      - "127.0.0.1:24224:24224"
+    depends_on:
+      - elasticsearch
+
+  kibana:
+    image: kibana
+    ports:
+      - "5601:5601"
+    environment:
+      ELASTICSEARCH_URL: http://elasticsearch:9200

efk/fluentd/Dockerfile

@@ -0,0 +1,5 @@
+FROM ruby
+RUN gem install fluentd
+RUN gem install fluent-plugin-elasticsearch
+COPY fluentd.conf /fluentd.conf
+CMD ["fluentd", "-c", "/fluentd.conf"]

efk/fluentd/fluentd.conf

@@ -0,0 +1,12 @@
+<source>
+  @type forward
+  port 24224
+  bind 0.0.0.0
+</source>
+
+<match **>
+  @type elasticsearch
+  host elasticsearch
+  logstash_format true
+  flush_interval 1
+</match>

elk/logstash.conf

@@ -0,0 +1,34 @@
+input {
+  # Listens on 514/udp and 514/tcp by default; change that to non-privileged port
+  syslog { port => 51415 }
+  # Default port is 12201/udp
+  gelf { }
+  # This generates one test event per minute.
+  # It is great for debugging, but you might
+  # want to remove it in production.
+  heartbeat { }
+}
+# The following filter is a hack!
+# The "de_dot" filter would be better, but it
+# is not pre-installed with logstash by default.
+filter {
+  ruby {
+    code => "
+      event.to_hash.keys.each { |k| event[ k.gsub('.','_') ] = event.remove(k) if k.include?'.' }
+    "
+  }
+}
+output {
+  elasticsearch {
+    hosts => ["elasticsearch:9200"]
+  }
+  # This will output every message on stdout.
+  # It is great when testing your setup, but in
+  # production, it will probably cause problems;
+  # either by filling up your disks, or worse,
+  # by creating logging loops! BEWARE!
+  stdout {
+    codec => rubydebug
+  }
+}
+

prepare-local/README.md

@@ -13,11 +13,12 @@ Virtualbox, Vagrant and Ansible
 - Virtualbox: https://www.virtualbox.org/wiki/Downloads
 
 - Vagrant: https://www.vagrantup.com/downloads.html
+  - install vagrant-vbguest plugin (https://github.com/dotless-de/vagrant-vbguest)
 
 - Ansible:
   - install Ansible's prerequisites:
 
-          $ sudo pip install paramiko PyYAML Jinja2 httplib2 six
+          $ sudo pip install paramiko PyYAML Jinja2 httplib2 six pycrypto
 
   - clone the Ansible repository and checkout to a stable version
     (don't forget the `--recursive` argument when cloning!):

prepare-local/provisioning.yml

@@ -93,7 +93,7 @@
         line: "{{ item.private_ip }} {{ item.hostname }}"
         regexp: "^{{ item.private_ip }} {{ item.hostname }}$"
         state: present
-      with_items: instances
+      with_items: "{{ instances }}"
 
     - name: copying the ssh key to the nodes
       copy:

prepare-machine/README.md

@@ -0,0 +1,242 @@
+# Setting up your own cluster 
+If you want to go through this orchestration workshop on your own,
+you will need a cluster of Docker nodes.
+
+These instructions will walk you through the required steps,
+using [Docker Machine](https://docs.docker.com/machine/) to
+create the nodes.
+
+
+## Requirements
+
+You need Docker Machine. To check if it is installed, try to
+run the following command:
+
+```bash
+$ docker-machine -v
+docker-machine version 0.8.2, build e18a919
+```
+
+If you see a Docker Machine version number, perfect! Otherwise,
+you need to install it; either as part of the Docker Toolbox,
+or as a stand-alone tool. See [Docker Machine installation docs](
+https://docs.docker.com/machine/install-machine/) for details.
+
+You also need either credentials for a cloud provider, or a
+local VirtualBox or VMware installation (or anything supported
+by Docker Machine, really).
+
+
+## Discrepancies with official environment
+
+The resulting environment will be slightly different from the
+one that we provision for people attending the workshop at
+conferences and similar events, and you will have to adapt a
+few things.
+
+We try to list all the differences here.
+
+
+### User name
+
+The official environment uses user `docker`. If you use
+Docker Machine, the user name will probably be different.
+
+### Node aliases
+
+In the official environment, aliases are seeded in
+`/etc/hosts`, allowing you to resolve node IP addresses
+with the aliases `node1`, `node2`, etc.; if you use
+Docker Machine, you will have to lookup the IP addresses
+with the `docker-machine ip nodeX` command instead.
+
+### SSH keys
+
+In the official environment, you can log from one node
+to another with SSH, without having to provide a password,
+thanks to pre-generated (and pre-copied) SSH keys.
+If you use Docker Machine, you will have to use
+`docker-machine ssh` from your machine instead.
+
+### Machine and Compose
+
+In the official environment, Docker Machine and Docker
+Compose are installed on your nodes. If you use Docker
+Machine you will have to install at least Docker Compose.
+
+The easiest way to install Compose (verified to work
+with the EC2 and VirtualBox drivers, and probably others
+as well) is do use `docker-machine ssh` to connect
+to your node, then run the following command:
+
+```bash
+sudo curl -L \
+  https://github.com/docker/compose/releases/download/1.9.0/docker-compose-`uname -s`-`uname -m` \
+  -o /usr/local/bin/docker-compose
+sudo chmod +x /usr/local/bin/docker-compose
+```
+
+Note that it is not necessary (or even useful) to
+install Docker Machine on your nodes, since if you're
+following that guide, you already have Machine on
+your local computer. ☺
+
+
+### IP addresses
+
+In some environments, your nodes will have multiple
+IP addresses. This is the case with VirtualBox, for
+instance. At any point in the workshop, if you need
+a node's IP address, you should use the address
+given by the `docker-machine ip` command.
+
+
+## Creating your nodes with Docker Machine
+
+Here are some instructions for various Machine Drivers.
+
+
+### AWS EC2
+
+You have to retrieve your AWS access key and secret access key,
+and set the following environment variables:
+
+```bash
+export MACHINE_DRIVER=amazonec2
+export AWS_ACCESS_KEY_ID=AKI...
+export AWS_SECRET_ACCESS_KEY=...
+```
+
+Optionally, you can also set `AWS_DEFAULT_REGION` to the region
+closest to you. See [AWS documentation](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions)
+for the list of available regions and their codes.
+
+For instance, if you are on the US West Coast, I recommend
+that you set `AWS_DEFAULT_REGION` to `us-west-2`; if you are
+in Europe, to `eu-central-1` (except in UK and Ireland where
+you probably want `eu-west-1`), etc.
+
+If you don't specify anything, your nodes will be in `us-east-1`.
+
+You can also set `AWS_INSTANCE_TYPE` if you want bigger or smaller
+instances than `t2.micro`. For the official workshops, we use
+`m3.large`, but remember: the bigger the instance, the more
+expensive it gets, obviously!
+
+After setting these variables, run the following command:
+
+```bash
+for N in $(seq 1 5); do
+  docker-machine create node$N
+  docker-machine ssh node$N usermod -aG docker ubuntu
+done
+```
+
+And after a few minutes, your five nodes will be ready. To log
+into a node, use `docker-machine ssh nodeX`.
+
+By default, Docker Machine places the created nodes in a
+security group aptly named `docker-machine`. By default, this
+group is pretty restrictive, and will only let you connect
+to the Docker API and SSH. For the purpose of the workshop,
+you will need to open that security group to normal traffic.
+You can do that through the AWS EC2 console, or with the
+following CLI command:
+
+```bash
+aws ec2 authorize-security-group-ingress --group-name docker-machine --protocol -1 --cidr 0.0.0.0/0
+```
+
+If Docker Machine fails, complaining that it cannot find
+the default VPC or subnet, this could be because you have
+an "old" EC2 account (created before the introduction of EC2
+VPC) and your account has no default VPC. In that case,
+you will have to create a VPC, a subnet in that VPC,
+and use the corresponding Machine flags (`--amazonec2-vpc-id`
+and `--amazonec2-subnet-id`) or environment variables
+(`AWS_VPC_ID` and `AWS_SUBNET_ID`) to tell Machine what to use.
+
+You will get similar error messages if you *have* set these
+flags (or environment variables) but the VPC (or subnets)
+indicated do not exist. This can happen if you frequently
+switch between different EC2 accounts, and forget that you
+have set the `AWS_VPC_ID` or `AWS_SUBNET_ID`.
+
+
+### Microsoft Azure
+
+You have to retrieve your subscription ID, and set the following environment
+variables:
+
+```bash
+export MACHINE_DRIVER=azure
+export AZURE_SUBSCRIPTION_ID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
+```
+
+Additionally, you can set `AZURE_LOCATION` to an Azure datacenter
+close to you. By default, it will pick "West US". You can see
+the available regions [on Azure's website](
+https://azure.microsoft.com/en-us/regions/services/).
+For instance, if you want to deploy on the US East Coast,
+set `AZURE_LOCATION` to `East US` or `eastus` (capitalization
+and spacing shouldn't matter; just use the names shown on the
+map or table on Azure's website).
+
+Then run the following command:
+
+```bash
+for N in $(seq 1 5); do
+  docker-machine create node$N
+  docker-machine ssh node$N usermod -aG docker docker-user
+done
+```
+
+The CLI will give you instructions to authenticate on the Azure portal,
+and once you've done that, it will create your VMs.
+
+You will log into your nodes with `docker-machine ssh nodeX`.
+
+By default, the firewall only allows access to the Docker API
+and SSH ports. To open access to other ports, you can use the
+following command:
+
+```bash
+for N in $(seq 1 5); do
+az network nsg rule create -g docker-machine --name AllowAny --nsg-name node$N-firewall \
+   --access allow --direction inbound --protocol '*' \
+   --source-address-prefix '*' --source-port-range '*' \
+   --destination-address-prefix '*' --destination-port-range '*'
+done
+```
+
+(The command takes a while. Be patient.)
+
+
+### Local VirtualBox or VMware Fusion
+
+If you want to run with local VMs, set the environment variable
+`MACHINE_DRIVER` to `virtualbox` or `vmwarefusion` and create your nodes:
+
+```bash
+export MACHINE_DRIVER=virtualbox
+for N in $(seq 1 5); do
+  docker-machine create node$N
+done
+```
+
+
+### Terminating instances
+
+When you're done, if you started your instance on a public
+cloud (or anywhere where it costs you money!) you will want to
+terminate (destroy) them. This can be done with the following
+command:
+
+```bash
+for N in $(seq 1 5); do
+  docker-machine rm -f node$N
+done
+```
+
+
+

prepare-vms/Dockerfile

@@ -24,7 +24,7 @@ RUN pip install \
     PyYAML \
     termcolor
 
-WORKDIR $HOME
+WORKDIR $$HOME
 RUN echo "alias ll='ls -lahF'" >> /root/.bashrc
 ENTRYPOINT ["/root/prepare-vms/scripts/trainer-cli"]
 

prepare-vms/README.md

@@ -1,67 +1,113 @@
-# Trainer tools to prepare VMs for Docker workshops
+# Trainer tools to create and prepare VMs for Docker workshops on AWS
 
-## 1. Prerequisites
+## Prerequisites
 
-* [Docker](https://docs.docker.com/engine/installation/)
-* [Docker Compose](https://docs.docker.com/compose/install/)
+- [Docker](https://docs.docker.com/engine/installation/)
+- [Docker Compose](https://docs.docker.com/compose/install/)
 
-## 2. Clone the repo
+## General Workflow
+
+- fork/clone repo
+- set required environment variables for AWS
+- create your own setting file from `settings/example.yaml`
+- run `./trainer` commands to create instances, install docker, setup each users environment in node1, other management tasks
+- run `./trainer cards` command to generate PDF for printing handouts of each users host IP's and login info
+
+## Clone/Fork the Repo, and Build the Tools Image
+
+The Docker Compose file here is used to build a image with all the dependencies to run the `./trainer` commands and optional tools. Each run of the script will check if you have those dependencies locally on your host, and will only use the container if you're [missing a dependency](trainer#L5).
 
     $ git clone https://github.com/jpetazzo/orchestration-workshop.git
     $ cd orchestration-workshop/prepare-vms
     $ docker-compose build
-    $ ./trainer <commands>  # See "Summary of commands" section below
 
-## 3. Preparing the environment
+## Preparing to Run `./trainer`
 
-Required environment variables:
+### Required AWS Permissions/Info
 
-* `AWS_ACCESS_KEY_ID`
-* `AWS_SECRET_ACCESS_KEY`
-* `AWS_DEFAULT_REGION`
+- Initial assumptions are you're using a root account. If you'd like to use a IAM user, it will need  `AmazonEC2FullAccess` and `IAMReadOnlyAccess`.
+- Using a non-default VPC or Security Group isn't supported out of box yet, but until then you can [customize the `trainer-cli` script](scripts/trainer-cli#L396-L401).
+- These instances will assign the default VPC Security Group, which does not open any ports from Internet by default. So you'll need to add Inbound rules for `SSH | TCP | 22 | 0.0.0.0/0` and `Custom TCP Rule | TCP | 8000 - 8002 | 0.0.0.0/0`, or run `./trainer opensg` which opens up all ports.
 
-### 4. Update settings.yaml
+### Required Environment Variables
 
-Then pass `settings/YOUR_WORKSHOP_NAME-settings.yaml` as an argument to `deploy`, `cards`, etc.
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+- `AWS_DEFAULT_REGION`
 
-## Usage
+### Update/copy `settings/example.yaml`
 
-### Summary of commands
+Then pass `settings/YOUR_WORKSHOP_NAME-settings.yaml` as an argument to `trainer deploy`, `trainer cards`, etc.
 
-The `trainer` script can be executed directly.
+./trainer cards 2016-09-28-00-33-bret settings/orchestration.yaml
 
-Summary of steps to launch a batch of instances for a workshop:
+## `./trainer` Usage
 
-* Export the environment variables needed by the AWS CLI (see **2. Preparing the environment** above)
-* `./trainer start N` (where `N` is the number of AWS instances to create)
-* `./trainer list` to view the list of tags
-* `./trainer list TAG` to view the instances with a given `TAG`
-* `./trainer deploy TAG settings/somefile.yaml` to run `scripts/postprep.rc` via parallel-ssh
-* `./trainer pull-images TAG` to pre-pull a bunch of Docker images to the instances
-* `./trainer test TAG`
-* `./trainer cards TAG settings/somefile.yaml` to generate a PDF and an HTML file you can print and cut to hand out cards with connection information to attendees
+```
+./trainer <command> [n-instances|tag] [settings/file.yaml]
 
-`./trainer` will run locally if all its dependencies are fulfilled; otherwise it will run in a Docker container.
+Core commands:
+  start        n              Start n instances
+  list         [TAG]          If a tag is provided, list its VMs. Otherwise, list tags.
+  deploy       TAG            Deploy all instances with a given tag
+  pull-images  TAG            Pre-pull docker images. Run only after deploying.
+  stop         TAG            Stop and delete instances tagged TAG
 
-It will check for the necessary environment variables. Then, if all its dependencies are installed
-locally, it will execute `trainer-cli`. If not, it will look for a local Docker image
-tagged `preparevms_prepare-vms` (created automatically when you run `docker-compose build`).
-If found, it will run in a container. If not found, the user will be prompted to
-either install the missing dependencies or run `docker-compose build`.
+Extras:
+  ips          TAG            List all IPs of instances with a given tag (updates ips.txt)
+  ids          TAG/TOKEN      List all instance IDs with a given tag
+  shell                       Get a shell in the trainer container
+  status       TAG            Print information about this tag and its VMs
+  tags                        List all tags (per-region)
+  retag        TAG/TOKEN TAG  Retag instances with a new tag
 
-## Detailed usage
+Beta:
+  ami                         Look up Amazon Machine Images
+  cards        FILE           Generate cards
+  opensg                      Modify AWS security groups
+```
 
-### Start some VMs
+### Summary of What `./trainer` Does For You
 
-    $ ./trainer start 10
+- Used to manage bulk AWS instances for you without needing to use AWS cli or gui.
+- Can manage multiple "tags" or groups of instances, which are tracked in `prepare-vms/tags/`
+- Can also create PDF/HTML for printing student info for instance IP's and login.
+- The `./trainer` script can be executed directly.
+- It will run locally if all its dependencies are fulfilled; otherwise it will run in the Docker container you created with `docker-compose build` (preparevms_prepare-vms).
+- During `start` it will add your default local SSH key to all instances under the `ubuntu` user.
+- During `deploy` it will create the `docker` user with password `training`, which is printing on the cards for students. For now, this is hard coded.
 
-A few things will happen:
+### Example Steps to Launch a Batch of Instances for a Workshop
 
-* Your local SSH key will be synced
-* AWS instances will be created and tagged
-* A directory will be created
+- Export the environment variables needed by the AWS CLI (see **Required Environment Variables** above)
+- Run `./trainer start N` Creates `N` EC2 instances
+  - Your local SSH key will be synced to instances under `ubuntu` user
+  - AWS instances will be created and tagged based on date, and IP's stored in `prepare-vms/tags/`
+- Run `./trainer deploy TAG settings/somefile.yaml` to run `scripts/postprep.rc` via parallel-ssh
+  - If it errors or times out, you should be able to rerun
+  - Requires good connection to run all the parallel SSH connections, up to 100 parallel (ProTip: create dedicated management instance in same AWS region where you run all these utils from)
+- Run `./trainer pull-images TAG` to pre-pull a bunch of Docker images to the instances
+- Run `./trainer cards TAG settings/somefile.yaml` generates PDF/HTML files to print and cut and hand out to students
+- *Have a great workshop*
+- Run `./trainer stop TAG` to terminate instances.
 
-Details below.
+## Other Tools
+
+### Deploying your SSH key to all the machines
+
+- Make sure that you have SSH keys loaded (`ssh-add -l`).
+- Source `rc`.
+- Run `pcopykey`.
+
+
+### Installing extra packages
+
+- Source `postprep.rc`.
+  (This will install a few extra packages, add entries to
+  /etc/hosts, generate SSH keys, and deploy them on all hosts.)
+
+
+## Even More Details
 
 #### Sync of SSH keys
 
@@ -83,7 +129,7 @@ This ips.txt file will be created in the $TAG/ directory and a symlink will be p
 
 If you create new VMs, the symlinked file will be overwritten.
 
-## Deployment
+#### Deployment
 
 Instances can be deployed manually using the `deploy` command:
 
@@ -91,29 +137,29 @@ Instances can be deployed manually using the `deploy` command:
 
 The `postprep.rc` file will be copied via parallel-ssh to all of the VMs and executed.
 
-### Pre-pull images
+#### Pre-pull images
 
     $ ./trainer pull-images TAG
 
-### Generate cards
+#### Generate cards
 
     $ ./trainer cards TAG settings/somefile.yaml
 
-### List tags
+#### List tags
 
     $ ./trainer list
 
-### List VMs
+#### List VMs
 
     $ ./trainer list TAG
 
 This will print a human-friendly list containing some information about each instance.
 
-### Stop and destroy VMs
+#### Stop and destroy VMs
 
     $ ./trainer stop TAG
 
 ## ToDo
 
-  * Don't write to bash history in system() in postprep 
-  * compose, etc version inconsistent (int vs str)
+  - Don't write to bash history in system() in postprep
+  - compose, etc version inconsistent (int vs str)

prepare-vms/docker-compose.yml

@@ -1,23 +1,26 @@
-prepare-vms:
-    build: .
-    container_name: prepare-vms
-    working_dir: /root/prepare-vms
-    volumes:
-        - $HOME/.aws/:/root/.aws/
-        - /etc/localtime:/etc/localtime:ro
-        - /tmp/.X11-unix:/tmp/.X11-unix
-        - $SSH_AUTH_DIRNAME:$SSH_AUTH_DIRNAME
-        - $PWD/:/root/prepare-vms/
-    environment:
-        SCRIPT_DIR: /root/prepare-vms
-        DISPLAY: ${DISPLAY}
-        SSH_AUTH_SOCK: ${SSH_AUTH_SOCK}
-        SSH_AGENT_PID: ${SSH_AGENT_PID}
-        AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
-        AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
-        AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
-        AWS_DEFAULT_OUTPUT: json
-        AWS_INSTANCE_TYPE: ${AWS_INSTANCE_TYPE}
-        AWS_VPC_ID: ${AWS_VPC_ID}
-        USER: ${USER}
-    entrypoint: /root/prepare-vms/scripts/trainer-cli
+version: "2"
+
+services:
+    prepare-vms:
+        build: .
+        container_name: prepare-vms
+        working_dir: /root/prepare-vms
+        volumes:
+            - $HOME/.aws/:/root/.aws/
+            - /etc/localtime:/etc/localtime:ro
+            - /tmp/.X11-unix:/tmp/.X11-unix
+            - $SSH_AUTH_DIRNAME:$SSH_AUTH_DIRNAME
+            - $PWD/:/root/prepare-vms/
+        environment:
+            SCRIPT_DIR: /root/prepare-vms
+            DISPLAY: ${DISPLAY}
+            SSH_AUTH_SOCK: ${SSH_AUTH_SOCK}
+            SSH_AGENT_PID: ${SSH_AGENT_PID}
+            AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
+            AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
+            AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
+            AWS_DEFAULT_OUTPUT: json
+            AWS_INSTANCE_TYPE: ${AWS_INSTANCE_TYPE}
+            AWS_VPC_ID: ${AWS_VPC_ID}
+            USER: ${USER}
+        entrypoint: /root/prepare-vms/scripts/trainer-cli

prepare-vms/scripts/postprep.rc

@@ -1,5 +1,7 @@
 pssh -I tee /tmp/settings.yaml < $SETTINGS
 
+pssh sudo apt-get update
+pssh sudo apt-get install -y python-setuptools
 pssh sudo easy_install pyyaml
 
 pssh -I tee /tmp/postprep.py <<EOF
@@ -123,18 +125,6 @@ system("echo 1000000 | sudo tee /proc/sys/net/nf_conntrack_max")
 # This will install the latest Docker.
 system("curl --silent https://{}/ | grep -v '( set -x; sleep 20 )' | sudo sh".format(ENGINE_VERSION))
 
-# Make sure that the daemon listens on 55555 (for orchestration workshop).
-# To test, run: export DOCKER_HOST=tcp://localhost:55555 ; docker ps
-# or, run "curl localhost:55555" (it should return 404 not found). If it tells you connection refused, that's a bad sign
-system("sudo sed -i 's,-H fd://$,-H fd:// -H tcp://0.0.0.0:55555,' /lib/systemd/system/docker.service")
-system("sudo systemctl daemon-reload")
-
-# There seems to be a bug in the systemd scripts; so work around it.
-# See https://github.com/docker/docker/issues/18444
-# If docker is already running, need to do a restart
-system("curl --silent localhost:55555 || sudo systemctl restart docker ")  # does this work? if not, next line should cover it
-system("sudo systemctl start docker || true")
-
 ### Install docker-compose
 #system("sudo pip install -U docker-compose=={}".format(COMPOSE_VERSION))
 system("sudo curl -sSL -o /usr/local/bin/docker-compose https://github.com/docker/compose/releases/download/{}/docker-compose-{}-{}".format(COMPOSE_VERSION, platform.system(), platform.machine()))
@@ -155,7 +145,6 @@ system("while ! sudo -u docker docker version ; do sleep 2; done")
 system("docker pull swarm:{}".format(SWARM_VERSION))
 system("docker tag -f swarm:{} swarm".format(SWARM_VERSION))
 
-
 ### BEGIN CLUSTERING ###
 
 addresses = list(l.strip() for l in sys.stdin)
@@ -207,3 +196,6 @@ pssh "grep docker@ /home/docker/.ssh/authorized_keys \
     || cat /home/docker/.ssh/id_rsa.pub \
         | sudo -u docker tee -a /home/docker/.ssh/authorized_keys"
 
+# On node1, create and deploy TLS certs using Docker Machine
+pssh "if grep -q node1 /tmp/node; then grep ' node' /etc/hosts | xargs -n2 sudo -H -u docker docker-machine create -d generic --generic-ssh-user docker --generic-ip-address; fi"
+

prepare-vms/scripts/rc

@@ -11,8 +11,9 @@ pssh () {
     }
 
     echo "[parallel-ssh] $@"
+    export PSSH=$(which pssh || which parallel-ssh)
 
-    parallel-ssh -h $HOSTFILE -l ubuntu \
+    $PSSH -h $HOSTFILE -l ubuntu \
     --par 100 \
     -O LogLevel=ERROR \
     -O UserKnownHostsFile=/dev/null \

prepare-vms/scripts/trainer-cli

@@ -208,19 +208,22 @@ get_ami() {
     #AMI=$(suggest_amis | grep -v ^REGION | head -1 | awk '{print $7}')
     case $AWS_DEFAULT_REGION in
     eu-central-1)
-      AMI=ami-4d2bc622
+      AMI=ami-82cf0aed
       ;;
     eu-west-1)
-      AMI=ami-bdb13bce
+      AMI=ami-07174474
       ;;
     us-east-1)
-      AMI=ami-3ffc1052
+      AMI=ami-2808313f
+      ;;
+    us-east-2)
+      AMI=ami-1b772d7e
       ;;
     us-west-1)
-      AMI=ami-bdb13bce
+      AMI=ami-dab5e0ba
       ;;
     us-west-2)
-      AMI=ami-b33dc0d3
+      AMI=ami-9ee24ffe
       ;;
     esac
     echo $AMI
@@ -396,7 +399,7 @@ run_cli() {
         result=$(aws ec2 run-instances \
             --key-name $AWS_KEY_NAME \
             --count $2 \
-            --instance-type c3.large \
+            --instance-type t2.medium \
             --client-token $TOKEN \
             --image-id $AMI)
         reservation_id=$(echo "$result" | head -1 | awk '{print $2}' )

prepare-vms/settings/example.yaml

@@ -0,0 +1,36 @@
+# This file is passed by trainer-cli to scripts/ips-txt-to-html.py
+
+workshop_name: Docker Orchestration
+workshop_short_name: orchestration
+repo: https://github.com/jpetazzo/orchestration-workshop
+url: http://container.training/ # moreinfo link printed on cards
+
+#engine_version: experimental.docker.com #extra features that may change/runaway
+#engine_version: test.docker.com
+engine_version: get.docker.com #prod release
+compose_version: 1.8.1
+machine_version: 0.8.2
+swarm_version: 1.2.5
+
+# for now these are hard coded in script, and only used for printing cards
+instance_login: docker
+instance_password: training
+
+# 12 per page works well, but is quite small text
+clustersize: 5  # Number of VMs per cluster
+pagesize: 12    # Number of cards to print per page
+
+background_image: https://raw.githubusercontent.com/jpetazzo/orchestration-workshop/master/prepare-vms/media/swarm.png
+
+# To be printed on the cards:
+blurb: >
+    Here is the connection information to your very own
+       {cluster_or_machine} for this {workshop_name} workshop. You can connect
+       to {this_or_each} VM with any SSH client.
+
+       Your {machine_is_or_machines_are}:
+
+# {url} will be replaced by the script
+footer: >
+    <p>For slides, chat and other useful links, see: </p>
+    <center>{url}</center>

prepare-vms/settings/fundamentals.yaml

@@ -10,9 +10,7 @@ instance_password: training
 clustersize: 1  # Number of VMs per cluster
 pagesize: 15    # Number of cards to print per page
 
-#background_image: https://myapps.developer.ubuntu.com/site_media/appmedia/2014/12/swarm.png
-background_image: http://www.yellosoft.us/public/images/docker.png
-#background_image: ../media/swarm.png
+background_image: https://www.docker.com/sites/default/files/Engine.png
 
 # To be printed on the cards:
 blurb: >
@@ -30,6 +28,6 @@ footer: >
 url: http://container.training/
 
 engine_version: get.docker.com
-compose_version: 1.7.1
-machine_version: 0.6.0
-swarm_version: 1.2.2
+compose_version: 1.8.1
+machine_version: 0.8.2
+swarm_version: latest

prepare-vms/settings/orchestration.yaml

@@ -10,8 +10,7 @@ instance_password: training
 clustersize: 5  # Number of VMs per cluster
 pagesize: 12    # Number of cards to print per page
 
-#background_image: https://myapps.developer.ubuntu.com/site_media/appmedia/2014/12/swarm.png
-background_image: http://www.yellosoft.us/public/images/docker.png
+background_image: https://www.docker.com/sites/default/files/Engine.png
 #background_image: ../media/swarm.png
 
 # To be printed on the cards:
@@ -30,6 +29,6 @@ footer: >
 url: http://container.training/
 
 engine_version: get.docker.com
-compose_version: 1.7.1
-machine_version: 0.6.0
-swarm_version: 1.2.2
+compose_version: 1.8.1
+machine_version: 0.8.2
+swarm_version: latest

prepare-vms/settings/rc.yaml

@@ -30,6 +30,6 @@ footer: >
 url: http://container.training/
 
 engine_version: test.docker.com
-compose_version: 1.7.0-rc2
-machine_version: 0.7.0-rc1
-swarm_version: 1.2.0-rc2
+compose_version: 1.9.0
+machine_version: 0.9.0-rc1
+swarm_version: latest

prom/Dockerfile

@@ -0,0 +1,3 @@
+FROM prom/prometheus:v1.4.1
+COPY prometheus.yml /etc/prometheus/prometheus.yml
+

prom/prometheus.yml

@@ -0,0 +1,17 @@
+global:
+  scrape_interval: 10s
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['localhost:9090']
+  - job_name: 'node'
+    dns_sd_configs:
+      - names: ['tasks.node']
+        type: 'A'
+        port: 9100
+  - job_name: 'cadvisor'
+    dns_sd_configs:
+      - names: ['tasks.cadvisor']
+        type: 'A'
+        port: 8080
+

snap/docker-influxdb.json

@@ -0,0 +1,29 @@
+{
+    "version": 1,
+    "schedule": {
+        "type": "simple",
+        "interval": "10s"
+    },
+    "max-failures": 10,
+    "workflow": {
+        "collect": {
+            "metrics": {
+                "/intel/docker/*/stats/cgroups/cpu_stats/cpu_usage/total_usage": {},
+                "/intel/docker/*/stats/cgroups/memory_stats/usage/usage": {}
+            },
+            "process": null,
+            "publish": [
+                {
+                    "plugin_name": "influx",                            
+                    "config": {
+                        "host": "127.0.0.1",
+                        "port": 8086,
+                        "database": "snap",
+                        "user": "admin",
+                        "password": "admin"
+                    }
+                }
+            ]                                            
+        }
+    }
+}

snap/psutil-file.yml

@@ -0,0 +1,21 @@
+---
+  version: 1
+  schedule:
+    type: "simple"
+    interval: "1s"
+  max-failures: 10
+  workflow:
+    collect:
+      metrics:
+        /intel/psutil/load/load1: {}
+        /intel/psutil/load/load15: {}
+        /intel/psutil/load/load5: {}
+        /intel/psutil/vm/available: {}
+        /intel/psutil/vm/free: {}
+        /intel/psutil/vm/used: {}
+      config:
+      publish:
+        -
+            plugin_name: "mock-file"
+            config:
+                file: "/tmp/snap-psutil-file.log"

stacks/dockercoins

@@ -0,0 +1 @@
+../dockercoins

stacks/dockercoins+gelf.yml

@@ -0,0 +1,48 @@
+version: "3"
+
+services:
+  rng:
+    build: dockercoins/rng
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}rng${COLON_TAG-:latest}
+    logging:
+      driver: gelf
+      options:
+        gelf-address: udp://127.0.0.1:12201
+    deploy:
+      mode: global
+
+  hasher:
+    build: dockercoins/hasher
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}hasher${COLON_TAG-:latest}
+    logging:
+      driver: gelf
+      options:
+        gelf-address: udp://127.0.0.1:12201
+
+  webui:
+    build: dockercoins/webui
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}webui${COLON_TAG-:latest}
+    logging:
+      driver: gelf
+      options:
+        gelf-address: udp://127.0.0.1:12201
+    ports:
+    - "8000:80"
+
+  redis:
+    image: redis
+    logging:
+      driver: gelf
+      options:
+        gelf-address: udp://127.0.0.1:12201
+
+  worker:
+    build: dockercoins/worker
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}worker${COLON_TAG-:latest}
+    logging:
+      driver: gelf
+      options:
+        gelf-address: udp://127.0.0.1:12201
+    deploy:
+      replicas: 10
+

stacks/dockercoins.yml

@@ -0,0 +1,28 @@
+version: "3"
+
+services:
+  rng:
+    build: dockercoins/rng
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}rng${COLON_TAG-:latest}
+    deploy:
+      mode: global
+
+  hasher:
+    build: dockercoins/hasher
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}hasher${COLON_TAG-:latest}
+
+  webui:
+    build: dockercoins/webui
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}webui${COLON_TAG-:latest}
+    ports:
+    - "8000:80"
+
+  redis:
+    image: redis
+
+  worker:
+    build: dockercoins/worker
+    image: ${REGISTRY_SLASH-127.0.0.1:5000/}worker${COLON_TAG-:latest}
+    deploy:
+      replicas: 10
+

stacks/elk.yml

@@ -0,0 +1,40 @@
+version: "3"
+
+services:
+  elasticsearch:
+    image: elasticsearch:2
+
+  logstash:
+    image: logstash
+    command: |
+      -e '
+      input {
+        gelf { }
+        heartbeat { }
+      }
+      filter {
+        ruby {
+          code => "
+            event.to_hash.keys.each { |k| event[ k.gsub('"'.'"','"'_'"') ] = event.remove(k) if k.include?'"'.'"' }
+          "
+        }
+      }
+      output {
+        elasticsearch {
+          hosts => ["elasticsearch:9200"]
+        }
+        stdout {
+          codec => rubydebug
+        }
+      }'
+
+    ports:
+      - "12201:12201/udp"
+
+  kibana:
+    image: kibana:4
+    ports:
+      - "5601:5601"
+    environment:
+      ELASTICSEARCH_URL: http://elasticsearch:9200
+

stacks/prometheus.yml

@@ -0,0 +1,30 @@
+version: "3"
+
+services:
+
+  prometheus:
+    build: ../prom
+    image: 127.0.0.1:5000/prom
+    ports:
+      - "9090:9090"
+
+  node:
+    image: prom/node-exporter
+    command: -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^(sys|proc|dev|host|etc)($$|/)"
+    deploy:
+      mode: global
+    volumes:
+      - "/proc:/host/proc"
+      - "/sys:/host/sys"
+      - "/:/rootfs"
+
+  cadvisor:
+    image: google/cadvisor
+    deploy:
+      mode: global
+    volumes:
+      - "/:/rootfs"
+      - "/var/run:/var/run"
+      - "/sys:/sys"
+      - "/var/lib/docker:/var/lib/docker"
+

stacks/registry.yml

@@ -0,0 +1,8 @@
+version: "3"
+
+services:
+  registry:
+    image: registry:2
+    ports:
+      - "5000:5000"
+

www/docker-compose.yml

@@ -1,6 +0,0 @@
-www:
-  image: nginx
-  ports:
-    - "8080:80"
-  volumes:
-    - "./htdocs:/usr/share/nginx/html"

www/extract-section-titles.py

@@ -1,40 +0,0 @@
-#!/usr/bin/env python
-"""
-Extract and print level 1 and 2 titles from workshop slides.
-"""
-
-with open("htdocs/index.html", "r") as f:
-    data = f.read()
-
-# @jpetazzo abuses "class: title" to make a point sometimes
-skip = [
-    "Why?",
-    "---",
-    "But ...",
-    "WHY?!?",
-]
-
-# Ditch linebreaks from main section titles
-replace = [
-    "<br/>",
-]
-
-# remove blank lines
-sections = [x for x in data.split('\n') if x] # and x not in skip]
-sections = "\n".join(sections)
-sections = sections.split('class: title')
-del(sections[0]) # delete the CSS frontmatter
-
-for section in sections:
-    lines = [x for x in section.split("\n") if x]
-
-    if lines[0] not in skip:
-        title = lines[0]
-        title = title.replace("<br/> ", "")
-        title = title.replace("# ", "")
-        del(lines[0])
-        print("{}".format(title))
-
-    titles = [x[2:] for x in lines if x.startswith("# ")]
-    for title in titles:
-        print("\t{}".format(title))