diff --git a/docs/index.html b/docs/index.html index 6be9f3ef..b874aab2 100644 --- a/docs/index.html +++ b/docs/index.html @@ -196,7 +196,7 @@ grep '^# ' index.html | grep -v '.dab` +- Can be directly used by a Swarm cluster through `docker stack ...` commands -- It's JSON because you're not supposed to edit it manually +- Introduces a `deploy` section to pass Swarm-specific parameters -- It can be generated by Compose, and consumed by Docker (experimental branch) +- Resource limits are moved to this `deploy` section -- In addition to image names, it contains their exact SHA256 +- See [here](https://github.com/aanand/docker.github.io/blob/8524552f99e5b58452fcb1403e1c273385988b71/compose/compose-file.md#upgrading) for the complete list of changes + +- Supersedes *Distributed Application Bundles* + + (JSON payload describing an application; could be generated from a Compose file) --- -## Generating a DAB +## Removing everything -- This is done with the Compose `bundle` command +- Before deploying using "stacks," let's get a clean slate .exercise[ -- Create the DAB for the DockerCoins application: +- Remove *all* the services: ```bash - docker-compose bundle - ``` - -- Inspect the resulting file: - ```bash - cat dockercoins.dab + docker service ls -q | xargs docker service rm ``` ] --- -## Using a DAB +## Our first stack -- This is done with `docker stack deploy ` +We need a registry to move images around. + +Before, we deployed it with the following command: + +```bash +docker service create --publish 5000:5000 registry:2 +``` + +Now, we are going to deploy it with the following stack file: + +```yaml +version: "3" + +services: + registry: + image: registry:2 + ports: + - "5000:5000" +``` + +--- + +## Checking our stack files + +- All the stack files that we will use are in the `stacks` directory .exercise[ -- Try to deploy the DAB: +- Go to the `stacks` directory: ```bash - docker stack deploy dockercoins + cd ~/orchestration-workshop/stacks + ``` + +- Check `registry.yml`: + ```bash + cat registry.yml + ``` + +] + +--- + +## Deploying our first stack + +- All stack manipulation commands start with `docker stack` + +- Under the hood, they map to `docker service` commands + +- Stacks have a *name* (which also serves as a namespace) + +- Stacks are specified with the aforementioned Compose file format version 3 + +.exercise[ + +- Deploy our local registry: + ```bash + docker stack deploy registry --compose-file registry.yml + ``` + +] + +--- + +## Inspecting stacks + +- `docker stack ps` shows the detailed state of all services of a stack + +.exercise[ + +- Check that our registry is running correctly: + ```bash + docker stack ps registry + ``` + +- Confirm that we get the same output with the following command: + ```bash + docker service ps registry_registry + ``` + +] + +--- + +## Specifics of stack deployment + +Our registry is not *exactly* identical to the one deployed with `docker service create`! + +- Each stack gets its own overlay network + +- Services of the task are connected to this network +
(unless specified differently in the Compose file) + +- Services get network aliases matching their name in the Compose file +
(just like when Compose brings up an app specified in a v2 file) + +- Services are explicitly named `_` + +- Services and tasks also get an internal label indicating which stack they belong to + +--- + +## Building and pushing stack services + +- We are going to use the `build` + `image` trick that we showed earlier: + + ```bash + docker-compose -f my_stack_file.yml build + docker-compose -f my_stack_file.yml push + docker stack deploy my_stack --compose-file my_stack_file.yml + ``` + +.exercise[ + +- Try it: + ```bash + docker-compose -f dockercoins.yml build ``` ] -- -Oh, right, we need the *experimental* build of Docker! +It doesn't work!?! --- -## Installing Docker experimental CLI +## Upgrading Compose -- We don't need to upgrade our Docker Engines; we just need an upgraded CLI +- Compose file format version 3 is not supported in Compose 1.9 -- We will download and extract it in a separate directory (to keep the original intact) +- We have to use 1.10 (which is not released yet) .exercise[ -- Download and unpack the latest experimental build of Docker: +- Upgrade Compose, using the `master` branch: ```bash - curl -sSL \ - https://experimental.docker.com/builds/$(uname -s)/$(uname -m)/docker-latest.tgz \ - | tar -C ~ -zxf- + pip install git+git://github.com/docker/compose ``` ] --- -## Using Docker experimental CLI +## Trying again -- Just invoke `~/docker/docker` instead of `docker` +- We can now build and push our container images .exercise[ -- Deploy our app using the DAB file: +- Build our application: ```bash - ~/docker/docker stack deploy dockercoins + docker-compose -f dockercoins.yml build ``` -- Check the stack deployment: +- Push the images to our registry: ```bash - ~/docker/docker stack ps dockercoins + docker-compose -f dockercoins.yml push ``` ] +Let's have a look at the `dockercoins.yml` file while this is building and pushing. + +--- + +```yaml +version: "3" + +services: + rng: + build: dockercoins/rng + image: ${REGISTRY_SLASH-localhost:5000/}rng${COLON_TAG-:latest} + logging: + driver: gelf + options: + gelf-address: udp://localhost:12201 + deploy: + mode: global + ... + redis: + image: redis + ... + worker: + build: dockercoins/worker + image: ${REGISTRY_SLASH-localhost:5000/}worker${COLON_TAG-:latest} + ... + deploy: + replicas: 10 +``` + +--- + +## Deploying the application + +- Now that the images are on the registry, we can deploy our application stack + +.exercise[ + +- Create the application stack: + ```bash + docker stack deploy dockercoins --compose-file dockercoins.yml + ``` + +] + +We can now connect to any of our nodes on port 8000, and we will see the familiar hashing speed graph. + +--- + +## Deploying the metrics stack + +- Remember these super long obscure `docker service create` commands we did earlier? + +-- + +- Neither do I! + +-- + +.exercise[ + +- Deploy Prometheus, cAdvisor, and the node exporter, just like we deployed DockerCoins: + ```bash + docker-compose -f prometheus.yml build + docker-compose -f prometheus.yml push + docker stack deploy prometheus --compose-file prometheus.yml + ``` + +] + +Look at `prometheus.yml` while it's building and pushing. + +--- + +```yaml +version: "3" + +services: + + prometheus: + build: ../prom + image: localhost:5000/prom + ports: + - "9090:9090" + + node: + ... + + cadvisor: + image: google/cadvisor + deploy: + mode: global + volumes: + - "/:/rootfs" + - "/var/run:/var/run" + - "/sys:/sys" + - "/var/lib/docker:/var/lib/docker" +``` + +--- + +## Accessing our new metrics stack + +.exercise[ + +- Go to any node, port 9090 + +- Check that data scraping works (click on "status", then "targets") + +- Select a metric from the "insert metric at cursor" dropdown + +- Execute! + +] + --- -## Look at the newly deployed stack +## Caveats -.exercise[ +- Compose file format v3 is very new, and not written in the stone yet -- Let's find out which port was allocated for `webui`: - ```bash - docker service inspect dockercoins_webui \ - --format '{{ (index .Endpoint.Ports 0).PublishedPort }}' +- It exists because nobody wants to maintain redundant files if they can avoid it + +- Some features are not fully supported yet: + + - logging, see [#29116](https://github.com/docker/docker/issues/29116) + + - secrets + +- You can re-run `docker stack deploy` to update a stack + +- ... But unsupported features will be wiped each time you redeploy (!) + + (This will likely be fixed/improved soon) + +--- + +## Maintaining multiple environments + +There are many ways to handle variations between environments. + +- Compose loads `docker-compose.yml` and (if it exists) `docker-compose.override.yml` + +- Compose can load alternate file(s) by setting the `-f` flag or the `COMPOSE_FILE` environment variable + +- Compose files can *extend* other Compose files, selectively including services: + + ```yaml + web: + extends: + file: common-services.yml + service: webapp ``` -- Point your navigator to any node on that port - -] - -Note: we can use the "normal" CLI for everything else. - -We only need it for `docker stack`. - ---- - -## Clean up - -- Unsurprisingly, there is a `docker stack rm` command - -.exercise[ - -- Clean up the stack we just deployed: - ```bash - ~/docker/docker stack rm dockercoins - ``` - -] - ---- - -## Scoping - -- All resources (service names, network names...) are prefixed with the stack name - -- This allows us to stage up multiple instances side by side - - (Just like before with Compose's project name parameter) - ---- - -## Some features are not fully supported yet - -- Global scheduling - -- Scaling - -- Fixed port numbers - -- Logging options - -- ... and much more - -You can specify *most* of them in the DAB itself, but Compose can't generate it (yet). +See [this documentation page](https://docs.docker.com/compose/extends/) for more details about these techniques. --- diff --git a/stacks/dockercoins b/stacks/dockercoins new file mode 120000 index 00000000..8adbb1b5 --- /dev/null +++ b/stacks/dockercoins @@ -0,0 +1 @@ +../dockercoins \ No newline at end of file diff --git a/stacks/dockercoins.yml b/stacks/dockercoins.yml new file mode 100644 index 00000000..2739e4ee --- /dev/null +++ b/stacks/dockercoins.yml @@ -0,0 +1,48 @@ +version: "3" + +services: + rng: + build: dockercoins/rng + image: ${REGISTRY_SLASH-localhost:5000/}rng${COLON_TAG-:latest} + logging: + driver: gelf + options: + gelf-address: udp://localhost:12201 + deploy: + mode: global + + hasher: + build: dockercoins/hasher + image: ${REGISTRY_SLASH-localhost:5000/}hasher${COLON_TAG-:latest} + logging: + driver: gelf + options: + gelf-address: udp://localhost:12201 + + webui: + build: dockercoins/webui + image: ${REGISTRY_SLASH-localhost:5000/}webui${COLON_TAG-:latest} + logging: + driver: gelf + options: + gelf-address: udp://localhost:12201 + ports: + - "8000:80" + + redis: + image: redis + logging: + driver: gelf + options: + gelf-address: udp://localhost:12201 + + worker: + build: dockercoins/worker + image: ${REGISTRY_SLASH-localhost:5000/}worker${COLON_TAG-:latest} + logging: + driver: gelf + options: + gelf-address: udp://localhost:12201 + deploy: + replicas: 10 + diff --git a/stacks/elk.yml b/stacks/elk.yml new file mode 100644 index 00000000..d2460b39 --- /dev/null +++ b/stacks/elk.yml @@ -0,0 +1,40 @@ +version: "3" + +services: + elasticsearch: + image: elasticsearch:2 + + logstash: + image: logstash + command: | + -e ' + input { + gelf { } + heartbeat { } + } + filter { + ruby { + code => " + event.to_hash.keys.each { |k| event[ k.gsub('"'.'"','"'_'"') ] = event.remove(k) if k.include?'"'.'"' } + " + } + } + output { + elasticsearch { + hosts => ["elasticsearch:9200"] + } + stdout { + codec => rubydebug + } + }' + + ports: + - "12201:12201/udp" + + kibana: + image: kibana:4 + ports: + - "5601:5601" + environment: + ELASTICSEARCH_URL: http://elasticsearch:9200 + diff --git a/stacks/prometheus.yml b/stacks/prometheus.yml new file mode 100644 index 00000000..da442814 --- /dev/null +++ b/stacks/prometheus.yml @@ -0,0 +1,30 @@ +version: "3" + +services: + + prometheus: + build: ../prom + image: localhost:5000/prom + ports: + - "9090:9090" + + node: + image: prom/node-exporter + command: -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^(sys|proc|dev|host|etc)($$|/)" + deploy: + mode: global + volumes: + - "/proc:/host/proc" + - "/sys:/host/sys" + - "/:/rootfs" + + cadvisor: + image: google/cadvisor + deploy: + mode: global + volumes: + - "/:/rootfs" + - "/var/run:/var/run" + - "/sys:/sys" + - "/var/lib/docker:/var/lib/docker" + diff --git a/stacks/registry.yml b/stacks/registry.yml new file mode 100644 index 00000000..4fedb07a --- /dev/null +++ b/stacks/registry.yml @@ -0,0 +1,8 @@ +version: "3" + +services: + registry: + image: registry:2 + ports: + - "5000:5000" +