Commit Graph

52 Commits

Author SHA1 Message Date
Alejandro Gullón
e19f98fd70 changing input params checks (#190)
* changing input params checks

* small fix

* small fix
2021-12-01 12:40:46 -05:00
Pravin Dsilva
38302e7d95 Add timeout for Openstack node scenarios
Signed-off-by: Pravin Dsilva <pdsilva@redhat.com>
2021-11-25 20:56:59 -05:00
Paige Rubendall
f17ad062cf Ci tests (#184)
* Adding in working ci tests

* spacing in readme
2021-11-24 15:12:47 -05:00
Alejandro Gullón
baa812b7f0 Added new scenario to fill up a given volumen (#182)
* Added new scenario to fill up a given volumen

* fixing small issues and style

* adding PVC as input param instead of pod name

* small fix

* get container name and volumen name
replace oc with kubectl commands

* adding yaml file to create a pv, pvc and pod to run pvc_scenario

* adding support to match both string for describe command when looking for pod_name

* added support to find the pvc from a given pod

* small fix

* small fix
2021-11-24 12:18:49 -05:00
prubenda
8e0f4e63af Adding container for pod_exec in time scenarios 2021-11-23 11:33:49 -05:00
Paige Rubendall
2bb4686585 adding check for if namesapce exists 2021-11-16 09:45:04 -05:00
Paige Rubendall
0023d679f7 fixing killed container no reference issue 2021-11-03 10:06:13 -04:00
Naga Ravi Chaitanya Elluri
f3bbc85dd5 Fix issue with matching labels
This commit fixes the issue with application outages scenario where
the pod-selector is not being mapped properly.
2021-10-29 10:45:46 -04:00
Naga Ravi Chaitanya Elluri
43b1e5b727 Remove item only when the list is not empty
This commit fixes the case where the fault injected containers take
longer time to recover as the current checks are bombing out if the
list is empty.
2021-10-28 12:12:45 -04:00
Paige Rubendall
87aa9eef4d Adding multiple node names and instance count for label selectors 2021-10-26 13:44:28 -04:00
Naga Ravi Chaitanya Elluri
2674e09407 Ignore validation for network policy creation
This commit helps the cases where targeting application pods in a
namespace using pod-selector to create an outage fails because of
not being able to validate the selector.

Error message for reference:
error validating data: ValidationError(NetworkPolicy.spec.podSelector):
unknown field "app=dittybopper" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector
2021-10-14 19:31:38 -04:00
Paige Rubendall
10e9b09819 Adding fix for openstack node name issue 2021-10-14 14:56:46 -04:00
Naga Ravi Chaitanya Elluri
cdf3bc03d2 Add support to block traffic to an application
This commit enables users to simulate a downtime of an application
by blocking the traffic for the specified duration to see how
it/other components communicating with it behave in case of downtime.
2021-10-01 10:13:40 -04:00
Paige Rubendall
22df024312 adding validation that namespace becomes active 2021-09-28 09:58:55 -04:00
Naga Ravi Chaitanya Elluri
f36da323e7 Prioritize filtering on namespace to improve performance
This will avoid querying all namespaces for pods matching the label_selector
if defined as shown in the sample scenario config. This commit also prints a
pointer to the report generated at the end of the run.
2021-09-22 15:03:39 -04:00
Paige Rubendall
16b5214fdd Adding specific tag version of powerfulseal 2021-09-21 12:37:45 -04:00
Naga Ravi Chaitanya Elluri
036e51a6b1 Delete litmus crd's during the cleanup
This commit will ensure that the litmus resources installed on the
cluster get cleaned up and also creates the chaosengine in the
specified namespace.
2021-09-16 16:30:21 -04:00
Paige Rubendall
a9056ddf43 adding litmus logging 2021-09-08 17:11:49 -04:00
Naga Ravi Chaitanya Elluri
5da0b259c5 Run all the litmus resources in a single namespace
- This eases the usage and debuggability by running the fault injection pods in
  the same namespace as other resources of litmus. This will also ease the
  deletion process and ensure that there are no leftover objects on the cluster.

- This commit also enables users to use the same rbac template for all the litmus
  scenarios without having to pull in a specic one for each of the scenarios.
2021-09-08 16:37:07 -04:00
Naga Ravi Chaitanya Elluri
adb465cab0 Add support for multi-zone disruption
This will enable users to disrupt multiple zones in the cluster simultaneously
to be able to understand the behaviour of various components.
2021-08-26 08:23:24 -04:00
Paige Rubendall
22fcab57f5 container checking in pod 2021-08-25 09:28:03 -04:00
prubenda
9b0bcdbf0e Adding node memory hog scenario 2021-08-20 14:02:00 -04:00
Naga Ravi Chaitanya Elluri
6456eec76a Add zone outage scenarios
This commit adds support to create zone outage in AWS by denying both
ingress and egress traffic to the instances belonging to a particular
subnet belonging to the zone by tweaking the network acl. This creates
an outage of all the nodes in the zone - both master and workers.
2021-08-17 11:43:13 -04:00
Naga Ravi Chaitanya Elluri
716057eab6 Monitor user application availability during chaos
Current Kraken integration with Cerberus monitors the cluster as well as the
application health post chaos and pass/fails if they are not healthy after chaos.
This commit adds ability to monitor the user application health during the chaos
and fails the run in case of downtime as it's potentially a downtime in case of
customers environment as well. It is especially useful in case of control plane
failure scenarios including API server, Etcd, Ingress etc.
2021-07-27 13:15:57 -04:00
Naga Ravi Chaitanya Elluri
590edff63b Avoid namespace context switch
There are cases where the kubeconfig can be read only like when running
Kraken as a kubernetes deployment. This commit fixes the instances to
use -n flag instead of a namespace context switch.
2021-07-27 11:31:32 -04:00
koflerm
304f606b2b Use jsonpath to retrieve pod nodename (#129) 2021-07-23 20:08:06 -04:00
Naga Ravi Chaitanya Elluri
c0b9cb46da Improve error handling
This commit:
- Adds timeout to avoid operations hanging for long durations.
- Improves exception handling and exits wherever needed.
- Sets KUBECONFIG env var globoally to access the cluster.
2021-07-21 12:48:06 -04:00
Paige Rubendall
f051c1c30f Merge pull request #120 from paigerube14/container_kill
Container kill
2021-07-15 15:07:58 -04:00
prubenda
76efac8f9b Adding delete of namespaces 2021-07-13 13:31:45 -04:00
prubenda
46a1823291 Adding killing of specific containers in pods 2021-07-08 17:10:48 -04:00
Jared O'Connell
9b83dbcf04 Baremetal Node Support (#74)
* Support for baremtal node scenarious

* Finished baremetal support

* Added documentation for baremetal

* Clarify limitations of implementation in documentation

* Add baremetal support to new run.py file

* Allow use on newer machines

Some older machines require lanplus instead of lan

* Setup to allow per-device user, pass, and bmc address

Also set min version for a dependency

* Fix linting issues

* More linting issue fixes

* More linter issues

* Account for linter standard non-conformity

* Added baremetal warning

Co-authored-by: jaredoconnell <jocnnel@redhat.com>
2021-07-02 17:31:40 -04:00
prubenda
41bf815f98 Adding shut down scenario for gcp, az, aws, openstack 2021-06-23 09:00:58 -04:00
Naga Ravi Chaitanya Elluri
e30a4243f6 Add support to alerting on metrics evaluation
This commit enables alerting in Kraken based on the Prometheus queries defined
by the user and modifies the return code of the run to determine pass/fail for
the run.
2021-06-22 15:22:37 -04:00
Naga Ravi Chaitanya Elluri
7e8f0450d6 Add support to scrape and index metrics
This commit:
- Enables Kraken to leverage kube-burner to scrape metrics from
  Prometheus and index them into Elasticsearch. This way we can
  take a look at the metrics in Grafana long term even after the
  cluster is terminated.
- Enables separation of operations based on distribution with
  OpenShift as the default option. One of the use cases is to
  capture Prometheus instance details as it's installed by default
  while it's optional for Kubernetes.
2021-06-21 14:55:50 -04:00
Naga Ravi Chaitanya Elluri
871eb3d74e Avoid circular dependencies
This commit deletes unneeded imports and fixes the circular dependency
issues.
2021-06-17 11:18:34 -04:00
Naga Ravi Chaitanya Elluri
5c2453b07e Refactor code base
This commit:
- Refactors the code base to be more modular by moving functions
  into respective modules to make it lean and reusable.
- Uses black to reformat the code to follow PEP 8 practices.
2021-06-14 17:41:10 -04:00
Paige Rubendall
190cf5d462 Blank node name error message (#97)
* adding contribute doc

* Fixing blank node name param printing off incorrect data
2021-05-06 10:13:17 -04:00
Amit Sagtani
d00d6ec69e Install pre-commit and use GitHub Actions (#94)
* added pre-commit and code-cleaning

* removed tox and TravisCI
2021-05-05 09:53:45 -04:00
prubenda
c7bb32f633 Adding azure to node scenarios 2021-03-17 17:41:07 -04:00
Pravin Dsilva
918b5fb6d3 Add node level chaos scenarios for bastion node
Signed-off-by: Pravin Dsilva <pravin.d-silva@ibm.com>
2021-02-16 09:04:55 -08:00
Naga Ravi Chaitanya Elluri
a7e28ca490 Add support to deploy performance dashboards
This commit enables performance monitoring on the cluster when
running Kraken to be able to observe how cluster reacts to failures
as it's important to make sure the cluster is healthy in terms of
both recovery as well as performance.
2021-02-10 16:06:55 -05:00
arcprabh
8dd18af161 Enable support for Openstack cloud.
Signed-off-by: arcprabh <arcprabh@in.ibm.com>

Incorporated first round of review comments

Signed-off-by: arcprabh <arcprabh@in.ibm.com>

Resolve multiple node name issue for single ip

Signed-off-by: arcprabh <arcprabh@in.ibm.com>
2021-02-02 20:47:30 +05:30
prubenda
1fc9683c8c Adding litmus scenario options 2020-12-03 12:45:35 -05:00
prubenda
d3e01db574 adding start to fix for all other cloud types 2020-11-24 16:32:43 -05:00
prubenda
5d86077f63 Adding in pod exec using k8s python client 2020-11-17 13:19:57 -05:00
prubenda
72fe662e05 Adding GCP node scenarios support 2020-11-17 09:57:39 -05:00
prubenda
6f31519e5f adding time scenario 2020-10-27 08:37:54 -04:00
Yashashree Suresh
31f06b861a Added node scenarios to stop and terminate instance
This commit:
- Adds a node scenario to stop and start an instance
- Adds a node scenario to terminate an instance
- Adds a node scenario to reboot an instance
- Adds a node scenario to stop the kubelet
- Adds a node scenario to crash the node
2020-08-27 16:50:42 -04:00
prubenda
0fc82090f2 Adding watch to see if components recovered 2020-08-18 16:26:04 -04:00
prubenda
58b51ebffc Adding kraken containerization with readme instructions 2020-07-07 11:59:34 -04:00