mirror of
https://github.com/lucky-sideburn/kubeinvaders.git
synced 2026-04-07 02:46:52 +00:00
155 lines
7.7 KiB
Markdown
155 lines
7.7 KiB
Markdown
|
|
|
|
*Gamified chaos engineering and analysis tool for Kubernetes. It is like Space Invaders but the aliens are pods or worker nodes.*
|
|
|
|
|
|
|
|
# Table of Contents
|
|
|
|
1. [Description](#Description)
|
|
2. [New Version (game part in pure JS)](#new-version)
|
|
3. [Special Input Keys and features](#Special-Input-Keys-and-features)
|
|
4. [Prometheus metrics and Grafana](#Metrics)
|
|
5. [Installation](#Installation)
|
|
6. [Notes for large clusters](#Notes-for-large-clusters)
|
|
7. [Configuration](#Configuration)
|
|
|
|
|
|
## Description
|
|
|
|
Through KubeInvaders you can stress a Kubernetes cluster in a fun way and check how it is resilient.
|
|
|
|
|
|
## New Version
|
|
|
|
KubeInvaders is going to be full open-source. Meanwhile it is possibile choose between 2 different versions.
|
|
|
|
* Legacy version: Made using Defold as game engine.
|
|
* New version: It has been written in pure Javascript and it is totally open-source.
|
|
|
|
The new version of KubeInvaders has fewer features than legacy but is optimized for chaos engineering because pods and nodes are rendered together and there is a shuffle functionality for better random experiments.
|
|
|
|
## Special Input Keys and features
|
|
|
|
| Input | Action | Version (New or Legacy)|
|
|
|-----------------|--------------------------------------------------------------------------------------------|------------------------|
|
|
| n | Change namespace (you should define namespaces list. Ex: TARGET_NAMESPACE=foo1,foo2,foo3). | New, Legacy |
|
|
| a | Switch to automatic mode. | Legacy |
|
|
| m | Switch to manual mode. | Legacy |
|
|
| h | Show special keys. | New, Legacy |
|
|
| q | Hide help for special keys. | New, Legacy |
|
|
| i | Show pod's name. Move the ship towards an alien. | Legacy |
|
|
| r | Refresh log of a pod when spaceship is over the alien. | Legacy |
|
|
| s | Activate or deactivate shuffle | New |
|
|
| k | *(NEW)* Perform [kube-linter](https://github.com/stackrox/kube-linter) analysis for a pod. | Legacy |
|
|
| w | *(NEW)* Chaos engineering against Kubernetes nodes. | New, Legacy |
|
|
|
|
### Known problems
|
|
|
|
* It seems that KubeInvaders does not work with EKS because of problems with ServiceAccount. Work in progress!
|
|
|
|
## Hands-on Tutorial
|
|
|
|
To experience KubeInvaders in action, try it out in this free O'Reilly Katacoda scenario, [KubeInvaders](https://www.katacoda.com/kuber-ru/courses/kubernetes-chaos).
|
|
|
|
|
|
## Metrics
|
|
|
|
KubeInvaders exposes metrics for Prometheus through the standard endpoint /metrics
|
|
|
|
This is an example of Prometheus configuration
|
|
|
|
```bash
|
|
scrape_configs:
|
|
- job_name: kubeinvaders
|
|
static_configs:
|
|
- targets:
|
|
- kubeinvaders.kubeinvaders.svc.cluster.local:8080
|
|
```
|
|
Example of metrics
|
|
|
|
| Metric | Description |
|
|
|------------------|--------------------------------------------------------------------------------------------------------------------------------------|
|
|
| chaos_jobs_node_count{node=workernode01} | Total number of chaos jobs executed per node |
|
|
| chaos_node_jobs_total | Total number of chaos jobs executed against all worker nodes |
|
|
| deleted_pods_total 16 | Total number of deleted pods |
|
|
| deleted_namespace_pods_count{namespace=myawesomenamespace} |Total number of deleted pods per namespace |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Installation
|
|
### Install to Kubernetes with Helm (v3+)
|
|
|
|
```bash
|
|
helm repo add kubeinvaders https://lucky-sideburn.github.io/helm-charts/
|
|
|
|
kubectl create namespace kubeinvaders
|
|
|
|
helm install kubeinvaders --set-string target_namespace="namespace1\,namespace2" \
|
|
-n kubeinvaders kubeinvaders/kubeinvaders --set ingress.hostName=kubeinvaders.io
|
|
```
|
|
### Install legacy version
|
|
|
|
```bash
|
|
helm install kubeinvaders --set-string target_namespace="namespace1\,namespace2" \
|
|
-n kubeinvaders kubeinvaders/kubeinvaders --set ingress.hostName=kubeinvaders.io --set image.tag=legacy
|
|
```
|
|
### Security Notes
|
|
|
|
In order to restrict the access to the Kubeinvaders endpoint add this annotation into the ingress.
|
|
|
|
```yaml
|
|
nginx.ingress.kubernetes.io/whitelist-source-range: <your_ip>/32
|
|
```
|
|
### Install KubeInvaders on OpenShift
|
|
|
|
To Install KubeInvaders on your OpenShift Cluster clone this repo and launch the following commands:
|
|
|
|
```bash
|
|
|
|
oc create clusterrole kubeinvaders-role --verb=watch,get,delete,list --resource=pods,pods/log,jobs
|
|
|
|
TARGET_NAMESPACE=foobar,awesome-namespace
|
|
## You can define multiple namespaces ex: TARGET_NAMESPACE=foobar,foobar2
|
|
|
|
# Choose route host for your kubeinvaders instance.
|
|
ROUTE_HOST=kubeinvaders.org
|
|
|
|
# Please add your source ip IP_WHITELIST. This will add haproxy.router.openshift.io/ip_whitelist in KubeInvaders route
|
|
# https://docs.openshift.com/container-platform/3.9/architecture/networking/routes.html#whitelist
|
|
IP_WHITELIST="93.44.96.4"
|
|
|
|
oc new-project kubeinvaders --display-name='KubeInvaders'
|
|
oc create sa kubeinvaders -n kubeinvaders
|
|
oc adm policy add-cluster-role-to-user kubeinvaders-role -z kubeinvaders -n kubeinvaders
|
|
|
|
KUBEINVADERS_SECRET=$(oc get secret -n kubeinvaders --field-selector=type==kubernetes.io/service-account-token | grep 'kubeinvaders-token' | awk '{ print $1}' | head -n 1)
|
|
|
|
oc process -f openshift/KubeInvaders.yaml -p ROUTE_HOST=$ROUTE_HOST -p TARGET_NAMESPACE=$TARGET_NAMESPACE -p KUBEINVADERS_SECRET=$KUBEINVADERS_SECRET | oc create -f -
|
|
```
|
|
|
|
## Notes for large clusters
|
|
|
|
For clusters with many workers-nodes, KubeInvaders (legacy version) selects a subset of random items.
|
|
|
|
| Item | Max Number |
|
|
|-----------|--------------|
|
|
| Nodes | 15 |
|
|
|
|
|
|
## Configuration
|
|
|
|
### (Legacy Version) Environment Variables - Make the game more difficult to win!
|
|
|
|
Set the following variables in Kubernetes Deployment or OpenShift DeploymentConfig:
|
|
|
|
| ENV Var | Description |
|
|
|-----------------------------|-------------------------------------------------------------------------------|
|
|
| ALIENPROXIMITY (default 15) | Reduce the value to increase distance between aliens. |
|
|
| HITSLIMIT (default 0) | Seconds of CPU time to wait before shooting. |
|
|
| UPDATETIME (default 1) | Seconds to wait before update PODs status (you can set also 0.x Es: 0.5). |
|