Commit Graph
53 Commits
Author SHA1 Message Date
Janos Bonic d8c4c27a7d Fixes #265: Replace powerfulseal 2022-06-07 17:11:53 +02:00
Naga Ravi Chaitanya Elluri 9208f39e06 Add support to run on Kubernetes
This commit:
- Leverages distribution flag in the config set by the user to skip
  things not supported on OpenShift to be able to run scenarios on
  Kubernetes.
- Adds sample config and scenario files that work on Kubernetes.
2022-06-01 07:27:06 -05:00
Naga Ravi Chaitanya Elluri 90e1f20d50 Add support for setting performance-dashboards on Kubernetes 2022-05-25 09:45:39 -04:00
Sanja Bonic f52f16ade8 Starts fixing small parts of issue #185 2022-05-18 20:01:59 +02:00
Adolfo Aguirrezabal 3adf5847b2 Add option to avoid litmus uninstall before chaos run (#242)
* Adds option to avoid litmus uninstall before chaos run

* Add new option to the config files
2022-05-05 09:02:25 -04:00
Paige Rubendall 7f60701444 adding alibaba node scenario start 2022-04-01 16:46:29 -04:00
yogananth-subramanian 50dd9873c1 Node egress traffic shaping
Patch adds a scenario to create variations in egress traffic of a Node's interface using the tc and Netem.
2021-12-16 12:54:53 -05:00
Alejandro Gullón baa812b7f0 Added new scenario to fill up a given volumen (#182)
* Added new scenario to fill up a given volumen

* fixing small issues and style

* adding PVC as input param instead of pod name

* small fix

* get container name and volumen name
replace oc with kubectl commands

* adding yaml file to create a pv, pvc and pod to run pvc_scenario

* adding support to match both string for describe command when looking for pod_name

* added support to find the pvc from a given pod

* small fix

* small fix
2021-11-24 12:18:49 -05:00
Naga Ravi Chaitanya Elluri 674eb74a75 Expose setting the signal in the config
This commit enables users to start Kraken to act as listener by setting
the signal to PAUSE in the config to get the cluster to a desired test or
run any setup before injecting chaos by setting the signal to RUN. This
helps in cases where we have test cases that need to coordinate the chaos
at a desired time depending on the state of the cluster/test run.
2021-10-26 09:05:25 -04:00
Paige Rubendall 6b865fc573 Adding server set up for kraken 2021-10-25 08:58:46 -04:00
Naga Ravi Chaitanya Elluri cdf3bc03d2 Add support to block traffic to an application
This commit enables users to simulate a downtime of an application
by blocking the traffic for the specified duration to see how
it/other components communicating with it behave in case of downtime.
2021-10-01 10:13:40 -04:00
Paige Rubendall 22df024312 adding validation that namespace becomes active 2021-09-28 09:58:55 -04:00
Naga Ravi Chaitanya Elluri f36da323e7 Prioritize filtering on namespace to improve performance
This will avoid querying all namespaces for pods matching the label_selector
if defined as shown in the sample scenario config. This commit also prints a
pointer to the report generated at the end of the run.
2021-09-22 15:03:39 -04:00
Naga Ravi Chaitanya Elluri 036e51a6b1 Delete litmus crd's during the cleanup
This commit will ensure that the litmus resources installed on the
cluster get cleaned up and also creates the chaosengine in the
specified namespace.
2021-09-16 16:30:21 -04:00
Paige Rubendall a9056ddf43 adding litmus logging 2021-09-08 17:11:49 -04:00
Naga Ravi Chaitanya Elluri 5da0b259c5 Run all the litmus resources in a single namespace
- This eases the usage and debuggability by running the fault injection pods in
  the same namespace as other resources of litmus. This will also ease the
  deletion process and ensure that there are no leftover objects on the cluster.

- This commit also enables users to use the same rbac template for all the litmus
  scenarios without having to pull in a specic one for each of the scenarios.
2021-09-08 16:37:07 -04:00
Naga Ravi Chaitanya Elluri 6456eec76a Add zone outage scenarios
This commit adds support to create zone outage in AWS by denying both
ingress and egress traffic to the instances belonging to a particular
subnet belonging to the zone by tweaking the network acl. This creates
an outage of all the nodes in the zone - both master and workers.
2021-08-17 11:43:13 -04:00
Naga Ravi Chaitanya Elluri 716057eab6 Monitor user application availability during chaos
Current Kraken integration with Cerberus monitors the cluster as well as the
application health post chaos and pass/fails if they are not healthy after chaos.
This commit adds ability to monitor the user application health during the chaos
and fails the run in case of downtime as it's potentially a downtime in case of
customers environment as well. It is especially useful in case of control plane
failure scenarios including API server, Etcd, Ingress etc.
2021-07-27 13:15:57 -04:00
Naga Ravi Chaitanya Elluri c0b9cb46da Improve error handling
This commit:
- Adds timeout to avoid operations hanging for long durations.
- Improves exception handling and exits wherever needed.
- Sets KUBECONFIG env var globoally to access the cluster.
2021-07-21 12:48:06 -04:00
Paige Rubendall f051c1c30f Merge pull request #120 from paigerube14/container_kill
Container kill
2021-07-15 15:07:58 -04:00
prubenda 76efac8f9b Adding delete of namespaces 2021-07-13 13:31:45 -04:00
prubenda 46a1823291 Adding killing of specific containers in pods 2021-07-08 17:10:48 -04:00
Naga Ravi Chaitanya Elluri e195922504 Document pip version and add more logging 2021-07-07 09:49:52 -04:00
prubenda 5456fce924 Adding getting started docs 2021-06-23 13:58:43 -04:00
prubenda 41bf815f98 Adding shut down scenario for gcp, az, aws, openstack 2021-06-23 09:00:58 -04:00
Naga Ravi Chaitanya Elluri e30a4243f6 Add support to alerting on metrics evaluation
This commit enables alerting in Kraken based on the Prometheus queries defined
by the user and modifies the return code of the run to determine pass/fail for
the run.
2021-06-22 15:22:37 -04:00
Naga Ravi Chaitanya Elluri 7e8f0450d6 Add support to scrape and index metrics
This commit:
- Enables Kraken to leverage kube-burner to scrape metrics from
  Prometheus and index them into Elasticsearch. This way we can
  take a look at the metrics in Grafana long term even after the
  cluster is terminated.
- Enables separation of operations based on distribution with
  OpenShift as the default option. One of the use cases is to
  capture Prometheus instance details as it's installed by default
  while it's optional for Kubernetes.
2021-06-21 14:55:50 -04:00
Naga Ravi Chaitanya Elluri 871eb3d74e Avoid circular dependencies
This commit deletes unneeded imports and fixes the circular dependency
issues.
2021-06-17 11:18:34 -04:00
Naga Ravi Chaitanya Elluri 5c2453b07e Refactor code base
This commit:
- Refactors the code base to be more modular by moving functions
  into respective modules to make it lean and reusable.
- Uses black to reformat the code to follow PEP 8 practices.
2021-06-14 17:41:10 -04:00
Amit Sagtani d00d6ec69e Install pre-commit and use GitHub Actions (#94)
* added pre-commit and code-cleaning

* removed tox and TravisCI
2021-05-05 09:53:45 -04:00
prubenda a14f4c2561 Adding wait in litmus scenario before cerberus integration 2021-04-20 14:45:31 -04:00
Naga Ravi Chaitanya Elluri db42f054ba Add pointer to the new blog
This commit:
- Adds a pointer to a new blog which emphasizes the importance
  of making chaos part of Perf/Scale test runs.
- Bumps up the allowed max-line-length for the linters.
2021-03-22 19:37:20 -04:00
prubenda c7bb32f633 Adding azure to node scenarios 2021-03-17 17:41:07 -04:00
Pravin Dsilva 918b5fb6d3 Add node level chaos scenarios for bastion node
Signed-off-by: Pravin Dsilva <pravin.d-silva@ibm.com>
2021-02-16 09:04:55 -08:00
Naga Ravi Chaitanya Elluri a7e28ca490 Add support to deploy performance dashboards
This commit enables performance monitoring on the cluster when
running Kraken to be able to observe how cluster reacts to failures
as it's important to make sure the cluster is healthy in terms of
both recovery as well as performance.
2021-02-10 16:06:55 -05:00
arcprabh 8dd18af161 Enable support for Openstack cloud.
Signed-off-by: arcprabh <arcprabh@in.ibm.com>

Incorporated first round of review comments

Signed-off-by: arcprabh <arcprabh@in.ibm.com>

Resolve multiple node name issue for single ip

Signed-off-by: arcprabh <arcprabh@in.ibm.com>
2021-02-02 20:47:30 +05:30
prubenda 1fc9683c8c Adding litmus scenario options 2020-12-03 12:45:35 -05:00
Mike Fiedler 4d4ffdccbc Reorganize scenario selection/injection 2020-11-25 11:30:25 -05:00
prubenda d3e01db574 adding start to fix for all other cloud types 2020-11-24 16:32:43 -05:00
prubenda 5d86077f63 Adding in pod exec using k8s python client 2020-11-17 13:19:57 -05:00
prubenda 72fe662e05 Adding GCP node scenarios support 2020-11-17 09:57:39 -05:00
Naga Ravi Chaitanya Elluri c5d8d9e88a Display the pod scenarios logs
The current implementation just reports the pass/fail when
running the pod scenarios. This commit prints the pod scenarios logs
to stdout to give users the ability to understand the actions run by
powerfulseal under the hood - number of pods matching the filter,
pods picked for killing, post checks etc. This also helps in
debugging when the scenarios fail.
2020-10-30 10:53:37 -04:00
Yashashree1997 47847d86cd Adds the ability to run a specific type of scenario multiple times
With the current implementation, all the scenarios of specific type
(for example, pod scenario) has to be executed together. All
pod_scenarios are followed by node_scenarios and so on.
(pod_scenarios -> node_scenarios -> pod_scenarios is not possible)
This commit enables the user to run a specific type of scenario
multiple times. For example, few pod_scenarios followed by
node_scenarios followed by few_scenarios.
2020-10-30 10:40:42 -04:00
prubenda 6f31519e5f adding time scenario 2020-10-27 08:37:54 -04:00
prubenda 8f5b688fba working on powerfulseal retry logic 2020-09-11 17:08:31 -04:00
Yashashree Suresh 31f06b861a Added node scenarios to stop and terminate instance
This commit:
- Adds a node scenario to stop and start an instance
- Adds a node scenario to terminate an instance
- Adds a node scenario to reboot an instance
- Adds a node scenario to stop the kubelet
- Adds a node scenario to crash the node
2020-08-27 16:50:42 -04:00
prubenda 0fc82090f2 Adding watch to see if components recovered 2020-08-18 16:26:04 -04:00
prubenda 58b51ebffc Adding kraken containerization with readme instructions 2020-07-07 11:59:34 -04:00
prubenda 52e232d0e7 Adding iterations or infinite run of kraken 2020-06-09 10:55:24 -04:00
Yashashree Suresh f5ae7818fd Added python linters and enabled travis CI to run linters
This commit adds python linters as it improves the code quality.
Travis CI is enabled to run linters on each PR to make sure it
follows the best practices and doesn't break the tool.
2020-05-14 09:08:59 -04:00