Gamified chaos engineering and analysis tool for Kubernetes. It is like Space Invaders but the aliens are pods or worker nodes.
Table of Contents
- Description
- New Version (game part in pure JS)
- Special Input Keys and features
- Prometheus metrics and Grafana
- Installation
- Notes for large clusters
- Configuration
Description
Through KubeInvaders you can stress a Kubernetes cluster in a fun way and check how it is resilient.
New Version
KubeInvaders is going to be full open-source. Meanwhile it is possibile choose between 2 different versions.
- Legacy version: Made using Defold as game engine.
- New version: It has been written in pure Javascript and it is totally open-source.
The new version of KubeInvaders has fewer features than legacy but is optimized for chaos engineering because pods and nodes are rendered together and there is a shuffle functionality for better random experiments.
Special Input Keys and features
| Input | Action | Version (New or Legacy) |
|---|---|---|
| n | Change namespace (you should define namespaces list. Ex: TARGET_NAMESPACE=foo1,foo2,foo3). | New, Legacy |
| a | Switch to automatic mode. | Legacy |
| m | Switch to manual mode. | Legacy |
| h | Show special keys. | New, Legacy |
| q | Hide help for special keys. | New, Legacy |
| i | Show pod's name. Move the ship towards an alien. | Legacy |
| r | Refresh log of a pod when spaceship is over the alien. | Legacy |
| s | Activate or deactivate shuffle | New |
| k | (NEW) Perform kube-linter analysis for a pod. | Legacy |
| w | (NEW) Chaos engineering against Kubernetes nodes. | New, Legacy |
Known problems
- It seems that KubeInvaders does not work with EKS because of problems with ServiceAccount. Work in progress!
Hands-on Tutorial
To experience KubeInvaders in action, try it out in this free O'Reilly Katacoda scenario, KubeInvaders.
Metrics
KubeInvaders exposes metrics for Prometheus through the standard endpoint /metrics
This is an example of Prometheus configuration
scrape_configs:
- job_name: kubeinvaders
static_configs:
- targets:
- kubeinvaders.kubeinvaders.svc.cluster.local:8080
Example of metrics
| Metric | Description |
|---|---|
| chaos_jobs_node_count{node=workernode01} | Total number of chaos jobs executed per node |
| chaos_node_jobs_total | Total number of chaos jobs executed against all worker nodes |
| deleted_pods_total 16 | Total number of deleted pods |
| deleted_namespace_pods_count{namespace=myawesomenamespace} | Total number of deleted pods per namespace |
Installation
Install to Kubernetes with Helm (v3+)
helm repo add kubeinvaders https://lucky-sideburn.github.io/helm-charts/
kubectl create namespace kubeinvaders
helm install kubeinvaders --set-string target_namespace="namespace1\,namespace2" \
-n kubeinvaders kubeinvaders/kubeinvaders --set ingress.hostName=kubeinvaders.io
Install legacy version
helm install kubeinvaders --set-string target_namespace="namespace1\,namespace2" \
-n kubeinvaders kubeinvaders/kubeinvaders --set ingress.hostName=kubeinvaders.io --set image.tag=legacy
Security Notes
In order to restrict the access to the Kubeinvaders endpoint add this annotation into the ingress.
nginx.ingress.kubernetes.io/whitelist-source-range: <your_ip>/32
Install KubeInvaders on OpenShift
To Install KubeInvaders on your OpenShift Cluster clone this repo and launch the following commands:
oc create clusterrole kubeinvaders-role --verb=watch,get,delete,list --resource=pods,pods/log,jobs
## You can define multiple namespaces ex: TARGET_NAMESPACE=foobar,foobar2
TARGET_NAMESPACE=foobar,awesome-namespace
# Choose route host for your kubeinvaders instance.
ROUTE_HOST=kubeinvaders.org
# Please add your source ip IP_WHITELIST. This will add haproxy.router.openshift.io/ip_whitelist in KubeInvaders route
# https://docs.openshift.com/container-platform/3.9/architecture/networking/routes.html#whitelist
IP_WHITELIST="93.44.96.4"
oc new-project kubeinvaders --display-name='KubeInvaders'
oc create sa kubeinvaders -n kubeinvaders
oc adm policy add-cluster-role-to-user kubeinvaders-role -z kubeinvaders -n kubeinvaders
KUBEINVADERS_SECRET=$(oc get secret -n kubeinvaders --field-selector=type==kubernetes.io/service-account-token | grep 'kubeinvaders-token' | awk '{ print $1}' | head -n 1)
oc process -f openshift/KubeInvaders.yaml -p ROUTE_HOST=$ROUTE_HOST -p TARGET_NAMESPACE=$TARGET_NAMESPACE -p KUBEINVADERS_SECRET=$KUBEINVADERS_SECRET | oc create -f -
Notes for large clusters
For clusters with many workers-nodes, KubeInvaders (legacy version) selects a subset of random items.
| Item | Max Number |
|---|---|
| Nodes | 15 |
Configuration
(Legacy Version) Environment Variables - Make the game more difficult to win!
Set the following variables in Kubernetes Deployment or OpenShift DeploymentConfig:
| ENV Var | Description |
|---|---|
| ALIENPROXIMITY (default 15) | Reduce the value to increase distance between aliens. |
| HITSLIMIT (default 0) | Seconds of CPU time to wait before shooting. |
| UPDATETIME (default 1) | Seconds to wait before update PODs status (you can set also 0.x Es: 0.5). |