github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-02-14 17:49:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

DAB -> Compose v3

Browse Source
This commit is contained in:
Jerome Petazzoni
2016-12-05 08:53:31 -08:00
parent 6854698fe1
commit 9078cfb57d
6 changed files with 417 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

380
docs/index.html
View File

@@ -196,7 +196,7 @@ grep '^# ' index.html | grep -v '<br' | tr '#' '-'
- Dealing with stateful services
- Scripting image building and pushing
- Distributed Application Bundles
- Integration with Compose
- Controlling Docker from a container
- Node management
@@ -5325,7 +5325,7 @@ You can also directly use the file `docker-compose.yml-images`.
---
# Distributed Application Bundles
# Integration with Compose
- The previous section showed us how to streamline image build and push
@@ -5337,158 +5337,358 @@ You can also directly use the file `docker-compose.yml-images`.
---
## What is a Distributed Application Bundle?
## Compose file version 3
- Conceptually similar to a Compose file, but for Swarm clusters
(New in Docker Engine 1.13)
- A Distributed Application Bundle is a JSON payload describing the services
- Almost identical to version 2
- It's typically stored as `<stackname>.dab`
- Can be directly used by a Swarm cluster through `docker stack ...` commands
- It's JSON because you're not supposed to edit it manually
- Introduces a `deploy` section to pass Swarm-specific parameters
- It can be generated by Compose, and consumed by Docker (experimental branch)
- Resource limits are moved to this `deploy` section
- In addition to image names, it contains their exact SHA256
- See [here](https://github.com/aanand/docker.github.io/blob/8524552f99e5b58452fcb1403e1c273385988b71/compose/compose-file.md#upgrading) for the complete list of changes
- Supersedes *Distributed Application Bundles*
(JSON payload describing an application; could be generated from a Compose file)
---
## Generating a DAB
## Removing everything
- This is done with the Compose `bundle` command
- Before deploying using "stacks," let's get a clean slate
.exercise[
- Create the DAB for the DockerCoins application:
- Remove *all* the services:
```bash
docker-compose bundle
```
- Inspect the resulting file:
```bash
cat dockercoins.dab
docker service ls -q | xargs docker service rm
```
]
---
## Using a DAB
## Our first stack
- This is done with `docker stack deploy <stackname>`
We need a registry to move images around.
Before, we deployed it with the following command:
```bash
docker service create --publish 5000:5000 registry:2
```
Now, we are going to deploy it with the following stack file:
```yaml
version: "3"
services:
registry:
image: registry:2
ports:
- "5000:5000"
```
---
## Checking our stack files
- All the stack files that we will use are in the `stacks` directory
.exercise[
- Try to deploy the DAB:
- Go to the `stacks` directory:
```bash
docker stack deploy dockercoins
cd ~/orchestration-workshop/stacks
```
- Check `registry.yml`:
```bash
cat registry.yml
```
]
---
## Deploying our first stack
- All stack manipulation commands start with `docker stack`
- Under the hood, they map to `docker service` commands
- Stacks have a *name* (which also serves as a namespace)
- Stacks are specified with the aforementioned Compose file format version 3
.exercise[
- Deploy our local registry:
```bash
docker stack deploy registry --compose-file registry.yml
```
]
---
## Inspecting stacks
- `docker stack ps` shows the detailed state of all services of a stack
.exercise[
- Check that our registry is running correctly:
```bash
docker stack ps registry
```
- Confirm that we get the same output with the following command:
```bash
docker service ps registry_registry
```
]
---
## Specifics of stack deployment
Our registry is not *exactly* identical to the one deployed with `docker service create`!
- Each stack gets its own overlay network
- Services of the task are connected to this network
<br/>(unless specified differently in the Compose file)
- Services get network aliases matching their name in the Compose file
<br/>(just like when Compose brings up an app specified in a v2 file)
- Services are explicitly named `<stack_name>_<service_name>`
- Services and tasks also get an internal label indicating which stack they belong to
---
## Building and pushing stack services
- We are going to use the `build` + `image` trick that we showed earlier:
```bash
docker-compose -f my_stack_file.yml build
docker-compose -f my_stack_file.yml push
docker stack deploy my_stack --compose-file my_stack_file.yml
```
.exercise[
- Try it:
```bash
docker-compose -f dockercoins.yml build
```
]
--
Oh, right, we need the *experimental* build of Docker!
It doesn't work!?!
---
## Installing Docker experimental CLI
## Upgrading Compose
- We don't need to upgrade our Docker Engines; we just need an upgraded CLI
- Compose file format version 3 is not supported in Compose 1.9
- We will download and extract it in a separate directory (to keep the original intact)
- We have to use 1.10 (which is not released yet)
.exercise[
- Download and unpack the latest experimental build of Docker:
- Upgrade Compose, using the `master` branch:
```bash
curl -sSL \
https://experimental.docker.com/builds/$(uname -s)/$(uname -m)/docker-latest.tgz \
| tar -C ~ -zxf-
pip install git+git://github.com/docker/compose
```
]
---
## Using Docker experimental CLI
## Trying again
- Just invoke `~/docker/docker` instead of `docker`
- We can now build and push our container images
.exercise[
- Deploy our app using the DAB file:
- Build our application:
```bash
~/docker/docker stack deploy dockercoins
docker-compose -f dockercoins.yml build
```
- Check the stack deployment:
- Push the images to our registry:
```bash
~/docker/docker stack ps dockercoins
docker-compose -f dockercoins.yml push
```
]
Let's have a look at the `dockercoins.yml` file while this is building and pushing.
---
```yaml
version: "3"
services:
rng:
build: dockercoins/rng
image: ${REGISTRY_SLASH-localhost:5000/}rng${COLON_TAG-:latest}
logging:
driver: gelf
options:
gelf-address: udp://localhost:12201
deploy:
mode: global
...
redis:
image: redis
...
worker:
build: dockercoins/worker
image: ${REGISTRY_SLASH-localhost:5000/}worker${COLON_TAG-:latest}
...
deploy:
replicas: 10
```
---
## Deploying the application
- Now that the images are on the registry, we can deploy our application stack
.exercise[
- Create the application stack:
```bash
docker stack deploy dockercoins --compose-file dockercoins.yml
```
]
We can now connect to any of our nodes on port 8000, and we will see the familiar hashing speed graph.
---
## Deploying the metrics stack
- Remember these super long obscure `docker service create` commands we did earlier?
--
- Neither do I!
--
.exercise[
- Deploy Prometheus, cAdvisor, and the node exporter, just like we deployed DockerCoins:
```bash
docker-compose -f prometheus.yml build
docker-compose -f prometheus.yml push
docker stack deploy prometheus --compose-file prometheus.yml
```
]
Look at `prometheus.yml` while it's building and pushing.
---
```yaml
version: "3"
services:
prometheus:
build: ../prom
image: localhost:5000/prom
ports:
- "9090:9090"
node:
...
cadvisor:
image: google/cadvisor
deploy:
mode: global
volumes:
- "/:/rootfs"
- "/var/run:/var/run"
- "/sys:/sys"
- "/var/lib/docker:/var/lib/docker"
```
---
## Accessing our new metrics stack
.exercise[
- Go to any node, port 9090
- Check that data scraping works (click on "status", then "targets")
- Select a metric from the "insert metric at cursor" dropdown
- Execute!
]
---
## Look at the newly deployed stack
## Caveats
.exercise[
- Compose file format v3 is very new, and not written in the stone yet
- Let's find out which port was allocated for `webui`:
```bash
docker service inspect dockercoins_webui \
--format '{{ (index .Endpoint.Ports 0).PublishedPort }}'
- It exists because nobody wants to maintain redundant files if they can avoid it
- Some features are not fully supported yet:
- logging, see [#29116](https://github.com/docker/docker/issues/29116)
- secrets
- You can re-run `docker stack deploy` to update a stack
- ... But unsupported features will be wiped each time you redeploy (!)
(This will likely be fixed/improved soon)
---
## Maintaining multiple environments
There are many ways to handle variations between environments.
- Compose loads `docker-compose.yml` and (if it exists) `docker-compose.override.yml`
- Compose can load alternate file(s) by setting the `-f` flag or the `COMPOSE_FILE` environment variable
- Compose files can *extend* other Compose files, selectively including services:
```yaml
web:
extends:
file: common-services.yml
service: webapp
```
- Point your navigator to any node on that port
]
Note: we can use the "normal" CLI for everything else.
We only need it for `docker stack`.
---
## Clean up
- Unsurprisingly, there is a `docker stack rm` command
.exercise[
- Clean up the stack we just deployed:
```bash
~/docker/docker stack rm dockercoins
```
]
---
## Scoping
- All resources (service names, network names...) are prefixed with the stack name
- This allows us to stage up multiple instances side by side
(Just like before with Compose's project name parameter)
---
## Some features are not fully supported yet
- Global scheduling
- Scaling
- Fixed port numbers
- Logging options
- ... and much more
You can specify *most* of them in the DAB itself, but Compose can't generate it (yet).
See [this documentation page](https://docs.docker.com/compose/extends/) for more details about these techniques.
---

1
stacks/dockercoins Symbolic link
View File

@@ -0,0 +1 @@
../dockercoins

48
stacks/dockercoins.yml Normal file
View File

@@ -0,0 +1,48 @@
version: "3"
services:
rng:
build: dockercoins/rng
image: ${REGISTRY_SLASH-localhost:5000/}rng${COLON_TAG-:latest}
logging:
driver: gelf
options:
gelf-address: udp://localhost:12201
deploy:
mode: global
hasher:
build: dockercoins/hasher
image: ${REGISTRY_SLASH-localhost:5000/}hasher${COLON_TAG-:latest}
logging:
driver: gelf
options:
gelf-address: udp://localhost:12201
webui:
build: dockercoins/webui
image: ${REGISTRY_SLASH-localhost:5000/}webui${COLON_TAG-:latest}
logging:
driver: gelf
options:
gelf-address: udp://localhost:12201
ports:
- "8000:80"
redis:
image: redis
logging:
driver: gelf
options:
gelf-address: udp://localhost:12201
worker:
build: dockercoins/worker
image: ${REGISTRY_SLASH-localhost:5000/}worker${COLON_TAG-:latest}
logging:
driver: gelf
options:
gelf-address: udp://localhost:12201
deploy:
replicas: 10

40
stacks/elk.yml Normal file
View File

@@ -0,0 +1,40 @@
version: "3"
services:
elasticsearch:
image: elasticsearch:2
logstash:
image: logstash
command: |
-e '
input {
gelf { }
heartbeat { }
}
filter {
ruby {
code => "
event.to_hash.keys.each { |k| event[ k.gsub('"'.'"','"'_'"') ] = event.remove(k) if k.include?'"'.'"' }
"
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
}
stdout {
codec => rubydebug
}
}'
ports:
- "12201:12201/udp"
kibana:
image: kibana:4
ports:
- "5601:5601"
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200

30
stacks/prometheus.yml Normal file
View File

@@ -0,0 +1,30 @@
version: "3"
services:
prometheus:
build: ../prom
image: localhost:5000/prom
ports:
- "9090:9090"
node:
image: prom/node-exporter
command: -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^(sys|proc|dev|host|etc)($$|/)"
deploy:
mode: global
volumes:
- "/proc:/host/proc"
- "/sys:/host/sys"
- "/:/rootfs"
cadvisor:
image: google/cadvisor
deploy:
mode: global
volumes:
- "/:/rootfs"
- "/var/run:/var/run"
- "/sys:/sys"
- "/var/lib/docker:/var/lib/docker"

8
stacks/registry.yml Normal file
View File

@@ -0,0 +1,8 @@
version: "3"
services:
registry:
image: registry:2
ports:
- "5000:5000"