container.training/docs/swarmtools.md

SwarmKit debugging tools

• The SwarmKit repository comes with debugging tools

• They are low level tools; not for general use

• We are going to see two of these tools:

  • swarmctl, to communicate directly with the SwarmKit API

  • swarm-rafttool, to inspect the content of the Raft log

---

Building the SwarmKit tools

• We are going to install a Go compiler, then download SwarmKit source and build it

.exercise[

• Download, compile, and install SwarmKit with this one-liner:

  docker run -v /usr/local/bin:/go/bin golang \
         go get `-v` github.com/docker/swarmkit/...
  

]

Remove -v if you don't like verbose things.

Shameless promo: for more Go and Docker love, check this blog post!

Note: in the unfortunate event of SwarmKit master branch being broken, the build might fail. In that case, just skip the Swarm tools section.

---

Getting cluster-wide task information

• The Docker API doesn't expose this directly (yet)

• But the SwarmKit API does

• We are going to query it with swarmctl

• swarmctl is an example program showing how to interact with the SwarmKit API

---

Using swarmctl

• The Docker Engine places the SwarmKit control socket in a special path

• You need root privileges to access it

.exercise[

• If you are using Play-With-Docker, set the following alias:

    alias swarmctl='/lib/ld-musl-x86_64.so.1 /usr/local/bin/swarmctl \
                    --socket /var/run/docker/swarm/control.sock'
  

• Otherwise, set the following alias:

    alias swarmctl='sudo swarmctl \
                    --socket /var/run/docker/swarm/control.sock'
  

]

---

swarmctl in action

• Let's review a few useful swarmctl commands

.exercise[

• List cluster nodes (that's equivalent to docker node ls):

  swarmctl node ls
  

• View all tasks across all services:

  swarmctl task ls
  

]

---

swarmctl notes

• SwarmKit is vendored into the Docker Engine

• If you want to use swarmctl, you need the exact version of SwarmKit that was used in your Docker Engine

• Otherwise, you might get some errors like:

  Error: grpc: failed to unmarshal the received message proto: wrong wireType = 0
  

• With Docker 1.12, the control socket was in /var/lib/docker/swarm/control.sock

---

swarm-rafttool

• SwarmKit stores all its important data in a distributed log using the Raft protocol

  (This log is also simply called the "Raft log")

• You can decode that log with swarm-rafttool

• This is a great tool to understand how SwarmKit works

• It can also be used in forensics or troubleshooting

  (But consider it as a very low level tool!)

---

The powers of swarm-rafttool

With swarm-rafttool, you can:

• view the latest snapshot of the cluster state;

• view the Raft log (i.e. changes to the cluster state);

• view specific objects from the log or snapshot;

• decrypt the Raft data (to analyze it with other tools).

It cannot work on live files, so you must stop Docker or make a copy first.

---

Using swarm-rafttool

• First, let's make a copy of the current Swarm data

.exercise[

• If you are using Play-With-Docker, the Docker data directory is /graph:

  cp -r /graph/swarm /swarmdata
  

• Otherwise, it is in the default /var/lib/docker:

  sudo cp -r /var/lib/docker/swarm /swarmdata
  

]

---

Dumping the Raft log

• We have to indicate the path holding the Swarm data

  (Otherwise swarm-rafttool will try to use the live data, and complain that it's locked!)

.exercise[

• If you are using Play-With-Docker, you must use the musl linker:

  /lib/ld-musl-x86_64.so.1 /usr/local/bin/swarm-rafttool -d /swarmdata/ dump-wal
  

• Otherwise, you don't need the musl linker but you need to get root:

  sudo swarm-rafttool -d /swarmdata/ dump-wal
  

]

Reminder: this is a very low-level tool, requiring a knowledge of SwarmKit's internals!