Big autopilot update

'keys' does not handle special keys (like ^J) anymore.
Instead, we should use `key`, which will pass its entire
argument to tmux, without any processing. It is therefore
possible to do something like:

```key ^C```

Or

```key Escape```

Most (if not all) calls to special keys have been
converted to use 'key' instead of 'keys'.

Action ```copypaste``` has been deprecated in favor
of three separate actions:

```copy REGEX``` (searches the regex in the active pane,
and if found, places it in an internal clipboard)

```paste``` (inserts the content of the clipboard as
keystrokes)

```check``` (forces a status check)

Also, a 'tmux' command has been added. It allows to
do stuff like:

```tmux split-pane -v```
This commit is contained in:
Jerome Petazzoni
2020-01-18 09:49:18 -06:00
parent da9921d68a
commit 7d6ab6974d
30 changed files with 323 additions and 82 deletions

View File

@@ -3,6 +3,7 @@ apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
@@ -36,6 +37,7 @@ apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: default
labels:
app: fluentd
spec:
@@ -95,6 +97,7 @@ metadata:
labels:
app: elasticsearch
name: elasticsearch
namespace: default
spec:
selector:
matchLabels:
@@ -122,6 +125,7 @@ metadata:
labels:
app: elasticsearch
name: elasticsearch
namespace: default
spec:
ports:
- port: 9200
@@ -137,6 +141,7 @@ metadata:
labels:
app: kibana
name: kibana
namespace: default
spec:
selector:
matchLabels:
@@ -160,6 +165,7 @@ metadata:
labels:
app: kibana
name: kibana
namespace: default
spec:
ports:
- port: 5601

View File

@@ -26,6 +26,7 @@ IPADDR = None
class State(object):
def __init__(self):
self.clipboard = ""
self.interactive = True
self.verify_status = False
self.simulate_type = True
@@ -38,6 +39,7 @@ class State(object):
def load(self):
data = yaml.load(open("state.yaml"))
self.clipboard = str(data["clipboard"])
self.interactive = bool(data["interactive"])
self.verify_status = bool(data["verify_status"])
self.simulate_type = bool(data["simulate_type"])
@@ -51,6 +53,7 @@ class State(object):
def save(self):
with open("state.yaml", "w") as f:
yaml.dump(dict(
clipboard=self.clipboard,
interactive=self.interactive,
verify_status=self.verify_status,
simulate_type=self.simulate_type,
@@ -85,9 +88,11 @@ class Snippet(object):
# On single-line snippets, the data follows the method immediately
if '\n' in content:
self.method, self.data = content.split('\n', 1)
else:
self.data = self.data.strip()
elif ' ' in content:
self.method, self.data = content.split(' ', 1)
self.data = self.data.strip()
else:
self.method, self.data = content, None
self.next = None
def __str__(self):
@@ -186,7 +191,7 @@ def wait_for_prompt():
if last_line == "$":
# This is a perfect opportunity to grab the node's IP address
global IPADDR
IPADDR = re.findall("^\[(.*)\]", output, re.MULTILINE)[-1]
IPADDR = re.findall("\[(.*)\]", output, re.MULTILINE)[-1]
return
# When we are in an alpine container, the prompt will be "/ #"
if last_line == "/ #":
@@ -264,19 +269,31 @@ for slide in re.split("\n---?\n", content):
slides.append(Slide(slide))
# Send a single key.
# Useful for special keys, e.g. tmux interprets these strings:
# ^C (and all other sequences starting with a caret)
# Space
# ... and many others (check tmux manpage for details).
def send_key(data):
subprocess.check_call(["tmux", "send-keys", data])
# Send multiple keys.
# If keystroke simulation is off, all keys are sent at once.
# If keystroke simulation is on, keys are sent one by one, with a delay between them.
def send_keys(data):
if state.simulate_type and data[0] != '^':
if not state.simulate_type:
subprocess.check_call(["tmux", "send-keys", data])
else:
for key in data:
if key == ";":
key = "\\;"
if key == "\n":
if interruptible_sleep(1): return
subprocess.check_call(["tmux", "send-keys", key])
send_key(key)
if interruptible_sleep(0.15*random.random()): return
if key == "\n":
if interruptible_sleep(1): return
else:
subprocess.check_call(["tmux", "send-keys", data])
def capture_pane():
@@ -323,7 +340,10 @@ def check_bounds():
while True:
state.save()
slide = slides[state.slide]
snippet = slide.snippets[state.snippet-1] if state.snippet else None
if state.snippet and state.snippet <= len(slide.snippets):
snippet = slide.snippets[state.snippet-1]
else:
snippet = None
click.clear()
print("[Slide {}/{}] [Snippet {}/{}] [simulate_type:{}] [verify_status:{}] "
"[switch_desktop:{}] [sync_slides:{}] [open_links:{}] [run_hidden:{}]"
@@ -398,7 +418,9 @@ while True:
continue
method, data = snippet.method, snippet.data
logging.info("Running with method {}: {}".format(method, data))
if method == "keys":
if method == "key":
send_key(data)
elif method == "keys":
send_keys(data)
elif method == "bash" or (method == "hide" and state.run_hidden):
# Make sure that we're ready
@@ -421,7 +443,7 @@ while True:
wait_for_prompt()
# Verify return code
check_exit_status()
elif method == "copypaste":
elif method == "copy":
screen = capture_pane()
matches = re.findall(data, screen, flags=re.DOTALL)
if len(matches) == 0:
@@ -430,8 +452,12 @@ while True:
match = matches[-1]
# Remove line breaks (like a screen copy paste would do)
match = match.replace('\n', '')
send_keys(match + '\n')
# FIXME: we should factor out the "bash" method
logging.info("Copied {} to clipboard.".format(match))
state.clipboard = match
elif method == "paste":
logging.info("Pasting {} from clipboard.".format(state.clipboard))
send_keys(state.clipboard)
elif method == "check":
wait_for_prompt()
check_exit_status()
elif method == "open":
@@ -445,6 +471,8 @@ while True:
if state.interactive:
print("Press any key to continue to next step...")
click.getchar()
elif method == "tmux":
subprocess.check_call(["tmux"] + data.split())
else:
logging.warning("Unknown method {}: {!r}".format(method, data))
move_forward()

View File

@@ -1 +0,0 @@
../swarm/links.md

View File

@@ -0,0 +1,12 @@
# Links and resources
- [Docker Community Slack](https://community.docker.com/registrations/groups/4316)
- [Docker Community Forums](https://forums.docker.com/)
- [Docker Hub](https://hub.docker.com)
- [Docker Blog](https://blog.docker.com/)
- [Docker documentation](https://docs.docker.com/)
- [Docker on StackOverflow](https://stackoverflow.com/questions/tagged/docker)
- [Docker on Twitter](https://twitter.com/docker)
- [Play With Docker Hands-On Labs](https://training.play-with-docker.com/)
.footnote[These slides (and future updates) are on → https://container.training/]

View File

@@ -118,9 +118,9 @@ installed and set up `kubectl` to communicate with your cluster.
<!--
```wait Connected to localhost```
```keys INFO server```
```keys ^J```
```key ^J```
```keys QUIT```
```keys ^J```
```key ^J```
-->
- Terminate the port forwarder:

View File

@@ -547,7 +547,7 @@ It's important to note a couple of details in these flags...
- Exit the container with `exit` or `^D`
<!-- ```keys ^D``` -->
<!-- ```key ^D``` -->
]

View File

@@ -109,7 +109,7 @@ spec:
<!--
```longwait latest: digest: sha256:```
```keys ^C```
```key ^C```
-->
]

View File

@@ -174,7 +174,7 @@ spec:
<!--
```longwait registry:5000/rng-kaniko:latest:```
```keys ^C```
```key ^C```
-->
]

View File

@@ -110,20 +110,22 @@
```bash vim rng.yml```
```wait kind: Deployment```
```keys /Deployment```
```keys ^J```
```key ^J```
```keys cwDaemonSet```
```keys ^[``` ]
```key ^[``` ]
```keys :wq```
```keys ^J```
```key ^J```
-->
- Save, quit
- Try to create our new resource:
```
```bash
kubectl apply -f rng.yml
```
<!-- ```wait error:``` -->
]
--
@@ -501,11 +503,11 @@ be any interruption.*
<!--
```wait Please edit the object below```
```keys /app: rng```
```keys ^J```
```key ^J```
```keys noenabled: yes```
```keys ^[``` ]
```key ^[``` ]
```keys :wq```
```keys ^J```
```key ^J```
-->
]
@@ -538,19 +540,18 @@ be any interruption.*
.exercise[
- Update the service to add `enabled: "yes"` to its selector:
```bash
kubectl edit service rng
```
- Update the YAML manifest of the service
- Add `enabled: "yes"` to its selector
<!--
```wait Please edit the object below```
```keys /app: rng```
```keys ^J```
```keys noenabled: "yes"```
```keys ^[``` ]
```keys /yes```
```key ^J```
```keys cw"yes"```
```key ^[``` ]
```keys :wq```
```keys ^J```
```key ^J```
-->
]
@@ -589,16 +590,25 @@ If we did everything correctly, the web UI shouldn't show any change.
```bash
POD=$(kubectl get pod -l app=rng,pod-template-hash -o name)
kubectl logs --tail 1 --follow $POD
```
(We should see a steady stream of HTTP logs)
<!--
```wait HTTP/1.1```
```tmux split-pane -v```
-->
- In another window, remove the label from the pod:
```bash
kubectl label pod -l app=rng,pod-template-hash enabled-
```
(The stream of HTTP logs should stop immediately)
<!--
```key ^D```
```key ^C```
-->
]
There might be a slight change in the web UI (since we removed a bit

View File

@@ -162,6 +162,8 @@ Instead, it has the fields expected in a DaemonSet.
kubectl diff -f web.yaml
```
<!-- ```wait status:``` -->
]
Note: we don't need to specify `--validate=false` here.

View File

@@ -105,19 +105,36 @@
- Monitor pod CPU usage:
```bash
watch kubectl top pods
watch kubectl top pods -l app=busyhttp
```
<!--
```wait NAME```
```tmux split-pane -v```
```bash CLUSTERIP=$(kubectl get svc busyhttp -o jsonpath={.spec.clusterIP})```
-->
- Monitor service latency:
```bash
httping http://`ClusterIP`/
httping http://`$CLUSTERIP`/
```
<!--
```wait connected to```
```tmux split-pane -v```
-->
- Monitor cluster events:
```bash
kubectl get events -w
```
<!--
```wait Normal```
```tmux split-pane -v```
```bash CLUSTERIP=$(kubectl get svc busyhttp -o jsonpath={.spec.clusterIP})```
-->
]
---
@@ -130,9 +147,15 @@
- Send a lot of requests to the service, with a concurrency level of 3:
```bash
ab -c 3 -n 100000 http://`ClusterIP`/
ab -c 3 -n 100000 http://`$CLUSTERIP`/
```
<!--
```wait be patient```
```tmux split-pane -v```
```tmux selectl even-vertical```
-->
]
The latency (reported by `httping`) should increase above 3s.
@@ -193,6 +216,20 @@ This can also be set with `--cpu-percent=`.
kubectl edit deployment busyhttp
```
<!--
```wait Please edit```
```keys /resources```
```key ^J```
```keys $xxxo requests:```
```key ^J```
```key Space```
```key Space```
```keys cpu: "1"```
```key Escape```
```keys :wq```
```key ^J```
-->
- In the `containers` list, add the following block:
```yaml
resources:
@@ -243,3 +280,29 @@ This can also be set with `--cpu-percent=`.
- The metrics provided by metrics server are standard; everything else is custom
- For more details, see [this great blog post](https://medium.com/uptime-99/kubernetes-hpa-autoscaling-with-custom-and-external-metrics-da7f41ff7846) or [this talk](https://www.youtube.com/watch?v=gSiGFH4ZnS8)
---
## Cleanup
- Since `busyhttp` uses CPU cycles, let's stop it before moving on
.exercise[
- Delete the `busyhttp` Deployment:
```bash
kubectl delete deployment busyhttp
```
<!--
```key ^D```
```key ^C```
```key ^D```
```key ^C```
```key ^D```
```key ^C```
```key ^D```
```key ^C```
-->
]

View File

@@ -124,7 +124,10 @@ kubectl create service externalname k8s --external-name kubernetes.io
kubectl get pods -w
```
<!-- ```keys ^C``` -->
<!--
```wait NAME```
```tmux split-pane -h```
-->
- Create a deployment for this very lightweight HTTP server:
```bash
@@ -191,6 +194,8 @@ kubectl create service externalname k8s --external-name kubernetes.io
<!--
```hide kubectl wait deploy httpenv --for condition=available```
```key ^D```
```key ^C```
-->
- Send a few requests:

View File

@@ -101,7 +101,7 @@ If we wanted to talk to the API, we would need to:
<!--
```wait /version```
```keys ^J```
```key ^J```
-->
- Terminate the proxy:

View File

@@ -154,6 +154,11 @@ pod/pingpong-7c8bbcd9bc-6c9qz 1/1 Running 0 10m
- Leave that command running, so that we can keep an eye on these logs
<!--
```wait seq=3```
```tmux split-pane -h```
-->
]
---
@@ -206,11 +211,21 @@ We could! But the *deployment* would notice it right away, and scale back to the
- Interrupt `kubectl logs` (with Ctrl-C)
<!--
```tmux last-pane```
```key ^C```
-->
- Restart it:
```bash
kubectl logs deploy/pingpong --tail 1 --follow
```
<!--
```wait using pod/pingpong-```
```tmux last-pane```
-->
]
`kubectl logs` will warn us that multiple pods were found, and that it's showing us only one of them.
@@ -235,10 +250,30 @@ Let's leave `kubectl logs` running while we keep exploring.
watch kubectl get pods
```
<!--
```wait Every 2.0s```
```tmux split-pane -v```
-->
- Destroy the pod currently shown by `kubectl logs`:
```
kubectl delete pod pingpong-xxxxxxxxxx-yyyyy
```
<!--
```tmux select-pane -t 0```
```copy pingpong-[^-]*-.....```
```tmux last-pane```
```keys kubectl delete pod ```
```paste```
```key ^J```
```check```
```key ^D```
```tmux select-pane -t 1```
```key ^C```
```key ^D```
-->
]
---
@@ -307,7 +342,7 @@ Let's leave `kubectl logs` running while we keep exploring.
- Create the Cron Job:
```bash
kubectl run --schedule="*/3 * * * *" --restart=OnFailure --image=alpine sleep 10
kubectl run every3mins --schedule="*/3 * * * *" --restart=OnFailure --image=alpine sleep 10
```
- Check the resource that was created:
@@ -418,7 +453,7 @@ Let's leave `kubectl logs` running while we keep exploring.
<!--
```wait seq=```
```keys ^C```
```key ^C```
-->
]
@@ -447,6 +482,8 @@ class: extra-details
kubectl logs -l run=pingpong --tail 1 -f
```
<!-- ```wait error:``` -->
]
We see a message like the following one:

View File

@@ -12,9 +12,9 @@
<!--
```wait RESTARTS```
```keys ^C```
```key ^C```
```wait AVAILABLE```
```keys ^C```
```key ^C```
-->
- Now, create more `worker` replicas:

View File

@@ -97,6 +97,8 @@
ship init https://github.com/jpetazzo/kubercoins
```
<!-- ```wait Open browser``` -->
]
---
@@ -189,6 +191,11 @@
kubectl logs deploy/worker --tail=10 --follow --namespace=kustomcoins
```
<!--
```wait units of work done```
```key ^C```
-->
]
Note: it might take a minute or two for the worker to start.

View File

@@ -84,14 +84,14 @@ Exactly what we need!
.exercise[
- View the logs for all the rng containers:
- View the logs for all the pingpong containers:
```bash
stern rng
stern pingpong
```
<!--
```wait HTTP/1.1```
```keys ^C```
```wait seq=```
```key ^C```
-->
]
@@ -117,7 +117,7 @@ Exactly what we need!
<!--
```wait weave-npc```
```keys ^C```
```key ^C```
-->
]
@@ -138,14 +138,14 @@ Exactly what we need!
.exercise[
- View the logs for all the things started with `kubectl create deployment`:
- View the logs for all the things started with `kubectl run`:
```bash
stern -l app
stern -l run
```
<!--
```wait units of work```
```keys ^C```
```wait seq=```
```key ^C```
-->
]

View File

@@ -120,6 +120,12 @@ This is our game plan:
kubectl create deployment testweb --image=nginx
```
<!--
```bash
kubectl wait deployment testweb --for condition=available
```
-->
- Find out the IP address of the pod with one of these two commands:
```bash
kubectl get pods -o wide -l app=testweb
@@ -154,6 +160,11 @@ The `curl` command should show us the "Welcome to nginx!" page.
curl $IP
```
<!--
```wait curl```
```key ^C```
-->
]
The `curl` command should now time out.

View File

@@ -108,7 +108,7 @@ kubectl wait deploy/worker --for condition=available
<!--
```wait units of work done, updating hash counter```
```keys ^C```
```key ^C```
-->
]

View File

@@ -220,6 +220,8 @@
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
```
<!-- ```wait apiVersion``` -->
]
---
@@ -240,6 +242,16 @@
- Save, quit
<!--
```keys /--enable-admission-plugins=```
```key ^J```
```key $```
```keys a,PodSecurityPolicy```
```key Escape```
```keys :wq```
```key ^J```
-->
]
---
@@ -271,6 +283,8 @@
kubectl run testpsp1 --image=nginx --restart=Never
```
<!-- ```wait forbidden: no providers available``` -->
- Try to create a Deployment:
```bash
kubectl run testpsp2 --image=nginx
@@ -498,3 +512,22 @@ class: extra-details
- bind `psp:restricted` to the group `system:authenticated`
- bind `psp:privileged` to the ServiceAccount `kube-system:default`
---
## Fixing the cluster
- Let's disable the PSP admission plugin
.exercise[
- Edit the Kubernetes API server static pod manifest
- Remove the PSP admission plugin
- This can be done with this one-liner:
```bash
sudo sed -i s/,PodSecurityPolicy// /etc/kubernetes/manifests/kube-apiserver.yaml
```
]

View File

@@ -197,7 +197,7 @@ If you want to use an external key/value store, add one of the following:
<!--
```longwait PX node status reports portworx service is healthy```
```keys ^C```
```key ^C```
-->
]
@@ -374,7 +374,7 @@ spec:
autopilot prompt detection expects $ or # at the beginning of the line.
```wait postgres@postgres```
```keys PS1="\u@\h:\w\n\$ "```
```keys ^J```
```key ^J```
-->
- Check that default databases have been created correctly:
@@ -428,7 +428,7 @@ autopilot prompt detection expects $ or # at the beginning of the line.
psql demo -c "select count(*) from pgbench_accounts"
```
<!-- ```keys ^D``` -->
<!-- ```key ^D``` -->
]
@@ -491,7 +491,7 @@ By "disrupt" we mean: "disconnect it from the network".
- Logout to go back on `node1`
<!-- ```keys ^D``` -->
<!-- ```key ^D``` -->
- Watch the events unfolding with `kubectl get events -w` and `kubectl get pods -w`
@@ -519,7 +519,7 @@ By "disrupt" we mean: "disconnect it from the network".
<!--
```wait postgres@postgres```
```keys PS1="\u@\h:\w\n\$ "```
```keys ^J```
```key ^J```
-->
- Check the number of rows in the `pgbench_accounts` table:
@@ -527,7 +527,7 @@ By "disrupt" we mean: "disconnect it from the network".
psql demo -c "select count(*) from pgbench_accounts"
```
<!-- ```keys ^D``` -->
<!-- ```key ^D``` -->
]

View File

@@ -94,7 +94,7 @@
<!--
```wait NAME```
```keys ^C```
```key ^C```
-->
- Update `worker` either with `kubectl edit`, or by running:
@@ -150,7 +150,7 @@ That rollout should be pretty quick. What shows in the web UI?
<!--
```wait Waiting for deployment```
```keys ^C```
```key ^C```
-->
]
@@ -229,11 +229,7 @@ If you didn't deploy the Kubernetes dashboard earlier, just skip this slide.
.exercise[
<!--
```keys
^C
```
-->
<!-- ```key ^C``` -->
- Cancel the deployment and wait for the dust to settle:
```bash
@@ -336,7 +332,7 @@ We might see something like 1, 4, 5.
- Check the annotations for our replica sets:
```bash
kubectl describe replicasets -l app=worker | grep -A3
kubectl describe replicasets -l app=worker | grep -A3 ^Annotations
```
]

View File

@@ -19,17 +19,14 @@
.exercise[
- Open two new terminals to check what's going on with pods and deployments:
- Open a new terminal to keep an eye on our pods:
```bash
kubectl get pods -w
kubectl get deployments -w
```
<!--
```wait RESTARTS```
```keys ^C```
```wait AVAILABLE```
```keys ^C```
```tmux split-pane -h```
-->
- Now, create more `worker` replicas:
@@ -73,6 +70,11 @@ The graph in the web UI should go up again.
kubectl scale deployment worker --replicas=10
```
<!--
```key ^D```
```key ^C```
-->
]
--

View File

@@ -105,6 +105,11 @@
docker run ctr.run/github.com/jpetazzo/container.training/dockercoins/hasher
```
<!--
```longwait Sinatra```
```key ^C```
-->
]
There might be a long pause before the first layer is pulled,

View File

@@ -427,7 +427,7 @@ nodes and encryption of gossip traffic) were removed for simplicity.
<!--
```wait Synced node info```
```keys ^C```
```key ^C```
-->
- Check the health of the cluster:

View File

@@ -110,6 +110,8 @@ It runs a single NGINX container.
kubectl create -f ~/container.training/k8s/nginx-1-without-volume.yaml
```
<!-- ```bash kubectl wait pod/nginx-without-volume --for condition=ready ``` -->
- Get its IP address:
```bash
IPADDR=$(kubectl get pod nginx-without-volume -o jsonpath={.status.podIP})
@@ -175,6 +177,8 @@ spec:
kubectl create -f ~/container.training/k8s/nginx-2-with-volume.yaml
```
<!-- ```bash kubectl wait pod/nginx-with-volume --for condition=ready ``` -->
- Get its IP address:
```bash
IPADDR=$(kubectl get pod nginx-with-volume -o jsonpath={.status.podIP})
@@ -269,6 +273,11 @@ spec:
kubectl get pods -o wide --watch
```
<!--
```wait NAME```
```tmux split-pane -v```
-->
]
---
@@ -282,11 +291,18 @@ spec:
kubectl create -f ~/container.training/k8s/nginx-3-with-git.yaml
```
<!--
```bash kubectl wait pod/nginx-with-git --for condition=initialized```
```bash IP=$(kubectl get pod nginx-with-git -o jsonpath={.status.podIP})```
-->
- As soon as we see its IP address, access it:
```bash
curl $IP
```
<!-- ```bash /bin/sleep 5``` -->
- A few seconds later, the state of the pod will change; access it again:
```bash
curl $IP
@@ -399,10 +415,19 @@ spec:
## Trying the init container
.exercise[
- Repeat the same operation as earlier
(try to send HTTP requests as soon as the pod comes up)
<!--
```key ^D```
```key ^C```
-->
]
- This time, instead of "403 Forbidden" we get a "connection refused"
- NGINX doesn't start until the git container has done its job

View File

@@ -40,7 +40,7 @@ class: extra-details
<!--
```wait units of work done```
```keys ^C```
```key ^C```
-->
]
@@ -75,7 +75,7 @@ Tip: use `^S` and `^Q` to pause/resume log output.
```bash top```
```wait Tasks```
```keys ^C```
```key ^C```
-->
- run `vmstat 1` to see I/O usage (si/so/bi/bo)
@@ -85,7 +85,7 @@ Tip: use `^S` and `^Q` to pause/resume log output.
```bash vmstat 1```
```wait memory```
```keys ^C```
```key ^C```
-->
]

View File

@@ -343,7 +343,7 @@ class: extra-details
- Stop the application by hitting `^C`
<!--
```keys ^C```
```key ^C```
-->
]

View File

@@ -267,7 +267,7 @@ class: extra-details
- Switch back to `node1` (with `exit`, `Ctrl-D` ...)
<!-- ```keys ^D``` -->
<!-- ```key ^D``` -->
- View the cluster from `node1`, which is a manager:
```bash

View File

@@ -72,7 +72,7 @@
```
<!-- ```wait User-Agent``` -->
<!-- ```keys ^C``` -->
<!-- ```key ^C``` -->
]

View File

@@ -158,7 +158,7 @@ class: elk-manual
```
<!-- ```wait "message" => "ok"``` -->
<!-- ```keys ^C``` -->
<!-- ```key ^C``` -->
]
@@ -266,7 +266,7 @@ The test message should show up in the logstash container logs.
```
<!-- ```wait Detected task failure``` -->
<!-- ```keys ^C``` -->
<!-- ```key ^C``` -->
]