Added node scenarios to stop and terminate instance

This commit: - Adds a node scenario to stop and start an instance - Adds a node scenario to terminate an instance - Adds a node scenario to reboot an instance - Adds a node scenario to stop the kubelet - Adds a node scenario to crash the node
2026-08-25 09:27:36 +00:00 · 2020-08-27 16:50:42 -04:00
parent aac254ce45
commit 31f06b861a
11 changed files with 430 additions and 33 deletions
@@ -23,6 +23,8 @@ kraken:
        -    scenarios/etcd.yml
        -    scenarios/openshift-kube-apiserver.yml                           
        -    scenarios/openshift-apiserver.yml
+    node_scenarios:                                        # List of chaos node scenarios to load
+        -    scenarios/node_scenarios_example.yml

 tunings:
    wait_duration: 60                                      # Duration to wait between each chaos scenario 
@@ -57,7 +59,51 @@ The report is generated in the run directory and it contains the information abo
 [Cerberus](https://github.com/openshift-scale/cerberus) can be used to monitor the cluster under test and the aggregated go/no-go signal generated by it can be consumed by Kraken to determine pass/fail. This is to make sure the Kubernetes/OpenShift environments are healthy on a cluster level instead of just the targeted components level. It is highly recommended to turn on the Cerberus health check feature avaliable in Kraken after installing and setting up Cerberus. To do that, set cerberus_enabled to True and cerberus_url to the url where Cerberus publishes go/no-go signal in the config file.

 ### Kubernetes/OpenShift chaos scenarios supported
-Following are the components of Kubernetes/OpenShift for which a basic chaos scenario config exists today. It currently just supports pod based scenarios, we will be adding more soon. Adding a new pod based scenario is as simple as adding a new config under scenarios directory and defining it in the config.
+Kraken currently just supports pod and node based scenarios, we will be adding more soon.
+
+#### Node chaos scenarios
+Following node chaos scenarios are supported:
+
+1. **node_start_scenario**: scenario to stop the node instance.
+2. **node_stop_scenario**: scenario to stop the node instance.
+3. **node_stop_start_scenario**: scenario to stop and then start the node instance.
+4. **node_termination_scenario**: scenario to terminate the node instance.
+5. **node_reboot_scenario**: scenario to reboot the node instance.
+6. **stop_kubelet_scenario**: scenario to stop the kubelet of the node instance.
+7. **stop_start_kubelet_scenario**: scenario to stop and start the kubelet of the node instance.
+8. **node_crash_scenario**: scenario to crash the node instance.
+
+**NOTE**: If the node doesn't recover from the node_crash_scenario injection, reboot the node to get it back to Ready state.
+
+**NOTE**: node_start_scenario, node_stop_scenario, node_stop_start_scenario, node_termination_scenario, node_reboot_scenario and stop_start_kubelet_scenario are supported only on AWS as of now.
+
+**NOTE**: With AWS as the cloud type, make sure [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) is installed.
+
+Node scenarios can be injected by placing the node scenarios config files under node_scenarios option in the kraken config. Refer to [node_scenarios_example](https://github.com/openshift-scale/kraken/blob/master/scenarios/node_scenarios_example.yml) config file.
+
+
+```
+node_scenarios:
+  - actions:                                                        # node chaos scenarios to be injected
+    - node_stop_start_scenario
+    - stop_start_kubelet_scenario
+    - node_crash_scenario
+    node_name:                                                      # node on which scenario has to be injected
+    label_selector: node-role.kubernetes.io/worker                  # when node_name is not specified, a node with matching label_selector is selected for node chaos scenario injection
+    instance_kill_count: 1                                          # number of times to inject each scenario under actions
+    timeout: 120                                                    # duration to wait for completion of node scenario injection
+    cloud_type: aws                                                 # cloud type on which Kubernetes/OpenShift runs
+  - actions:
+    - node_reboot_scenario
+    node_name:
+    label_selector: node-role.kubernetes.io/infra
+    instance_kill_count: 1
+    timeout: 120
+    cloud_type: aws
+```
+
+#### Pod chaos scenarios
+Following are the components of Kubernetes/OpenShift for which a basic chaos scenario config exists today. Adding a new pod based scenario is as simple as adding a new config under scenarios directory and defining it in the config.

 Component                | Description                                                                                        | Working
 ------------------------ | ---------------------------------------------------------------------------------------------------| ------------------------- |
@@ -10,6 +10,9 @@ kraken:
          -    scenarios/post_action_openshift-apiserver.yml
        - -    scenarios/regex_openshift_pod_kill.yml
          -    scenarios/post_action_regex.py
+    node_scenarios:                                        # List of chaos node scenarios to load
+        -   scenarios/node_scenarios_example.yml
+
 cerberus:
    cerberus_enabled: False                                # Enable it when cerberus is previously installed
    cerberus_url:                                          # When cerberus_enabled is set to True, provide the url where cerberus publishes go/no-go signal
@@ -12,3 +12,10 @@ def invoke(command):
    except Exception as e:
        logging.error("Failed to run %s, error: %s" % (command, e))
    return out
+
+
+def run(command):
+    try:
+        subprocess.run(command, shell=True, universal_newlines=True, timeout=45)
+    except Exception:
+        pass
@@ -16,10 +16,13 @@ def initialize_clients(kubeconfig_path):


 # List nodes in the cluster
-def list_nodes():
+def list_nodes(label_selector=None):
    nodes = []
    try:
-        ret = cli.list_node(pretty=True)
+        if label_selector:
+            ret = cli.list_node(pretty=True, label_selector=label_selector)
+        else:
+            ret = cli.list_node(pretty=True)
    except ApiException as e:
        logging.error("Exception when calling CoreV1Api->list_node: %s\n" % e)
    for node in ret.items:
@@ -28,10 +31,13 @@ def list_nodes():


 # List nodes in the cluster that can be killed
-def list_killable_nodes():
+def list_killable_nodes(label_selector=None):
    nodes = []
    try:
-        ret = cli.list_node(pretty=True)
+        if label_selector:
+            ret = cli.list_node(pretty=True, label_selector=label_selector)
+        else:
+            ret = cli.list_node(pretty=True)
    except ApiException as e:
        logging.error("Exception when calling CoreV1Api->list_node: %s\n" % e)
    for node in ret.items:
@@ -63,6 +69,18 @@ def get_all_pods():
    return pods


+# Obtain node status
+def get_node_status(node):
+    try:
+        node_info = cli.read_node_status(node, pretty=True)
+    except ApiException as e:
+        logging.error("Exception when calling \
+                       CoreV1Api->read_node_status: %s\n" % e)
+    for condition in node_info.status.conditions:
+        if condition.type == "Ready":
+            return condition.status
+
+
 # Monitor the status of the cluster nodes and set the status to true or false
 def monitor_nodes():
    nodes = list_nodes()
@@ -0,0 +1,68 @@
+import sys
+import logging
+import kraken.invoke.command as runcommand
+import kraken.node_actions.common_node_functions as nodeaction
+
+
+class abstract_node_scenarios:
+
+    # Node scenario to start the node
+    def node_start_scenario(self, instance_kill_count, node, timeout):
+        pass
+
+    # Node scenario to stop the node
+    def node_stop_scenario(self, instance_kill_count, node, timeout):
+        pass
+
+    # Node scenario to stop and then start the node
+    def node_stop_start_scenario(self, instance_kill_count, node, timeout):
+        logging.info("Starting node_stop_start_scenario injection")
+        self.node_stop_scenario(instance_kill_count, node, timeout)
+        self.node_start_scenario(instance_kill_count, node, timeout)
+        logging.info("node_stop_start_scenario has been successfully injected!")
+
+    # Node scenario to terminate the node
+    def node_termination_scenario(self, instance_kill_count, node, timeout):
+        pass
+
+    # Node scenario to reboot the node
+    def node_reboot_scenario(self, instance_kill_count, node, timeout):
+        pass
+
+    # Node scenario to stop the kubelet
+    def stop_kubelet_scenario(self, instance_kill_count, node, timeout):
+        for _ in range(instance_kill_count):
+            try:
+                logging.info("Starting stop_kubelet_scenario injection")
+                logging.info("Stopping the kubelet of the node %s" % (node))
+                runcommand.run("oc debug node/" + node + " -- chroot /host systemctl stop kubelet")
+                nodeaction.wait_for_unknown_status(node, timeout)
+                logging.info("The kubelet of the node %s has been stopped" % (node))
+                logging.info("stop_kubelet_scenario has been successfuly injected!")
+            except Exception as e:
+                logging.error("Failed to stop the kubelet of the node. Encountered following "
+                              "exception: %s. Test Failed" % (e))
+                logging.error("stop_kubelet_scenario injection failed!")
+                sys.exit(1)
+
+    # Node scenario to stop and start the kubelet
+    def stop_start_kubelet_scenario(self, instance_kill_count, node, timeout):
+        logging.info("Starting stop_start_kubelet_scenario injection")
+        self.stop_kubelet_scenario(instance_kill_count, node, timeout)
+        self.node_reboot_scenario(instance_kill_count, node, timeout)
+        logging.info("stop_start_kubelet_scenario has been successfully injected!")
+
+    # Node scenario to crash the node
+    def node_crash_scenario(self, instance_kill_count, node, timeout):
+        for _ in range(instance_kill_count):
+            try:
+                logging.info("Starting node_crash_scenario injection")
+                logging.info("Crashing the node %s" % (node))
+                runcommand.invoke("oc debug node/" + node + " -- chroot /host "
+                                  "dd if=/dev/urandom of=/proc/sysrq-trigger")
+                logging.info("node_crash_scenario has been successfuly injected!")
+            except Exception as e:
+                logging.error("Failed to crash the node. Encountered following exception: %s. "
+                              "Test Failed" % (e))
+                logging.error("node_crash_scenario injection failed!")
+                sys.exit(1)
@@ -0,0 +1,142 @@
+import sys
+import time
+import boto3
+import logging
+import kraken.kubernetes.client as kubecli
+import kraken.node_actions.common_node_functions as nodeaction
+from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
+
+
+class AWS:
+    def __init__(self):
+        self.boto_client = boto3.client('ec2')
+        self.boto_instance = boto3.resource('ec2').Instance('id')
+
+    # Get the instance ID of the node
+    def get_instance_id(self, node):
+        return self.boto_client.describe_instances(
+            Filters=[{'Name': 'private-dns-name', 'Values': [node]}]
+        )['Reservations'][0]['Instances'][0]['InstanceId']
+
+    # Start the node instance
+    def start_instances(self, instance_id):
+        self.boto_client.start_instances(
+            InstanceIds=[instance_id]
+        )
+
+    # Stop the node instance
+    def stop_instances(self, instance_id):
+        self.boto_client.stop_instances(
+            InstanceIds=[instance_id]
+        )
+
+    # Terminate the node instance
+    def terminate_instances(self, instance_id):
+        self.boto_client.terminate_instances(
+            InstanceIds=[instance_id]
+        )
+
+    # Reboot the node instance
+    def reboot_instances(self, instance_id):
+        self.boto_client.reboot_instances(
+            InstanceIds=[instance_id]
+        )
+
+    # Wait until the node instance is running
+    def wait_until_running(self, instance_id):
+        self.boto_instance.wait_until_running(
+            InstanceIds=[instance_id]
+        )
+
+    # Wait until the node instance is stopped
+    def wait_until_stopped(self, instance_id):
+        self.boto_instance.wait_until_stopped(
+            InstanceIds=[instance_id]
+        )
+
+    # Wait until the node instance is terminated
+    def wait_until_terminated(self, instance_id):
+        self.boto_instance.wait_until_terminated(
+            InstanceIds=[instance_id]
+        )
+
+
+class aws_node_scenarios(abstract_node_scenarios):
+    def __init__(self):
+        self.aws = AWS()
+
+    # Node scenario to start the node
+    def node_start_scenario(self, instance_kill_count, node, timeout):
+        for _ in range(instance_kill_count):
+            try:
+                logging.info("Starting node_start_scenario injection")
+                instance_id = self.aws.get_instance_id(node)
+                logging.info("Starting the node %s with instance ID: %s " % (node, instance_id))
+                self.aws.start_instances(instance_id)
+                self.aws.wait_until_running(instance_id)
+                nodeaction.wait_for_ready_status(node, timeout)
+                logging.info("Node with instance ID: %s is in running state" % (instance_id))
+                logging.info("node_start_scenario has been successfully injected!")
+            except Exception as e:
+                logging.error("Failed to start node instance. Encountered following "
+                              "exception: %s. Test Failed" % (e))
+                logging.error("node_start_scenario injection failed!")
+                sys.exit(1)
+
+    # Node scenario to stop the node
+    def node_stop_scenario(self, instance_kill_count, node, timeout):
+        for _ in range(instance_kill_count):
+            try:
+                logging.info("Starting node_stop_scenario injection")
+                instance_id = self.aws.get_instance_id(node)
+                logging.info("Stopping the node %s with instance ID: %s " % (node, instance_id))
+                self.aws.stop_instances(instance_id)
+                self.aws.wait_until_stopped(instance_id)
+                logging.info("Node with instance ID: %s is in stopped state" % (instance_id))
+                nodeaction.wait_for_unknown_status(node, timeout)
+            except Exception as e:
+                logging.error("Failed to stop node instance. Encountered following exception: %s. "
+                              "Test Failed" % (e))
+                logging.error("node_stop_scenario injection failed!")
+                sys.exit(1)
+
+    # Node scenario to terminate the node
+    def node_termination_scenario(self, instance_kill_count, node, timeout):
+        for _ in range(instance_kill_count):
+            try:
+                logging.info("Starting node_termination_scenario injection")
+                instance_id = self.aws.get_instance_id(node)
+                logging.info("Terminating the node %s with instance ID: %s " % (node, instance_id))
+                self.aws.terminate_instances(instance_id)
+                self.aws.wait_until_terminated(instance_id)
+                for _ in range(timeout):
+                    if node not in kubecli.list_nodes():
+                        break
+                    time.sleep(1)
+                if node in kubecli.list_nodes():
+                    raise Exception("Node could not be terminated")
+                logging.info("Node with instance ID: %s has been terminated" % (instance_id))
+                logging.info("node_termination_scenario has been successfuly injected!")
+            except Exception as e:
+                logging.error("Failed to terminate node instance. Encountered following exception:"
+                              " %s. Test Failed" % (e))
+                logging.error("node_termination_scenario injection failed!")
+                sys.exit(1)
+
+    # Node scenario to reboot the node
+    def node_reboot_scenario(self, instance_kill_count, node, timeout):
+        for _ in range(instance_kill_count):
+            try:
+                logging.info("Starting node_reboot_scenario injection")
+                instance_id = self.aws.get_instance_id(node)
+                logging.info("Rebooting the node %s with instance ID: %s " % (node, instance_id))
+                self.aws.reboot_instances(instance_id)
+                nodeaction.wait_for_unknown_status(node, timeout)
+                nodeaction.wait_for_ready_status(node, timeout)
+                logging.info("Node with instance ID: %s has been rebooted" % (instance_id))
+                logging.info("node_reboot_scenario has been successfuly injected!")
+            except Exception as e:
+                logging.error("Failed to reboot node instance. Encountered following exception:"
+                              " %s. Test Failed" % (e))
+                logging.error("node_reboot_scenario injection failed!")
+                sys.exit(1)
@@ -0,0 +1,37 @@
+import time
+import random
+import logging
+import kraken.kubernetes.client as kubecli
+import kraken.invoke.command as runcommand
+
+
+# Pick a random node with specified label selector
+def get_node(node_name, label_selector):
+    if node_name in kubecli.list_killable_nodes():
+        return node_name
+    else:
+        logging.info("Node with provided node_name does not exist or the node might "
+                     "be in NotReady state.")
+    nodes = kubecli.list_killable_nodes(label_selector)
+    if not nodes:
+        raise Exception("Ready nodes with the provided label selector do not exist")
+    logging.info("Ready nodes with the label selector %s: %s" % (label_selector, nodes))
+    number_of_nodes = len(nodes)
+    node = nodes[random.randint(0, number_of_nodes - 1)]
+    return node
+
+
+# Wait till node status becomes Ready
+def wait_for_ready_status(node, timeout):
+    runcommand.invoke("kubectl wait --for=condition=Ready "
+                      "node/" + node + " --timeout=" + str(timeout) + "s")
+
+
+# Wait till node status becomes NotReady
+def wait_for_unknown_status(node, timeout):
+    for _ in range(timeout):
+        if kubecli.get_node_status(node) == "Unknown":
+            break
+        time.sleep(1)
+    if kubecli.get_node_status(node) != "Unknown":
+        raise Exception("Node condition status isn't Unknown")
@@ -2,3 +2,4 @@ datetime
 pyfiglet
 powerfulseal==3.0.0rc11
 requests
+boto3
@@ -1,15 +1,52 @@
 #!/usr/bin/env python

-import sys
 import os
+import sys
 import time
-import optparse
-import logging
 import yaml
+import logging
+import optparse
 import requests
+import pyfiglet
 import kraken.kubernetes.client as kubecli
 import kraken.invoke.command as runcommand
-import pyfiglet
+import kraken.node_actions.common_node_functions as nodeaction
+from kraken.node_actions.aws_node_scenarios import aws_node_scenarios
+
+
+# Get the node scenarios object of specfied cloud type
+def get_node_scenario_object(node_scenario):
+    if node_scenario['cloud_type'] == 'aws':
+        return aws_node_scenarios()
+
+
+# Inject the specified node scenario
+def inject_node_scenario(action, node_scenario, node_scenario_object):
+    # Get the node scenario configurations
+    instance_kill_count = node_scenario.get("instance_kill_count", 1)
+    node_name = node_scenario.get("node_name", "")
+    label_selector = node_scenario.get("label_selector", "")
+    timeout = node_scenario.get("timeout", 120)
+
+    # Get the node to apply the scenario
+    node = nodeaction.get_node(node_name, label_selector)
+
+    if action == "node_start_scenario":
+        node_scenario_object.node_start_scenario(instance_kill_count, node, timeout)
+    elif action == "node_stop_scenario":
+        node_scenario_object.node_stop_scenario(instance_kill_count, node, timeout)
+    elif action == "node_stop_start_scenario":
+        node_scenario_object.node_stop_start_scenario(instance_kill_count, node, timeout)
+    elif action == "node_termination_scenario":
+        node_scenario_object.node_termination_scenario(instance_kill_count, node, timeout)
+    elif action == "node_reboot_scenario":
+        node_scenario_object.node_reboot_scenario(instance_kill_count, node, timeout)
+    elif action == "stop_kubelet_scenario":
+        node_scenario_object.stop_kubelet_scenario(instance_kill_count, node, timeout)
+    elif action == "stop_start_kubelet_scenario":
+        node_scenario_object.stop_start_kubelet_scenario(instance_kill_count, node, timeout)
+    elif action == "node_crash_scenario":
+        node_scenario_object.node_crash_scenario(instance_kill_count, node, timeout)


 # Get cerberus status
@@ -125,11 +162,12 @@ def main(cfg):
    if os.path.isfile(cfg):
        with open(cfg, 'r') as f:
            config = yaml.full_load(f)
-        kubeconfig_path = config["kraken"]["kubeconfig_path"]
-        scenarios = config["kraken"]["scenarios"]
-        wait_duration = config["tunings"]["wait_duration"]
-        iterations = config["tunings"]["iterations"]
-        daemon_mode = config["tunings"]['daemon_mode']
+        kubeconfig_path = config["kraken"].get("kubeconfig_path", "")
+        scenarios = config["kraken"].get("scenarios", [])
+        node_scenarios = config["kraken"].get("node_scenarios", [])
+        wait_duration = config["tunings"].get("wait_duration", 60)
+        iterations = config["tunings"].get("iterations", 1)
+        daemon_mode = config["tunings"].get("daemon_mode", False)

        # Initialize clients
        if not os.path.isfile(kubeconfig_path):
@@ -153,11 +191,11 @@ def main(cfg):
        # Set the number of iterations to loop to infinity if daemon mode is
        # enabled or else set it to the provided iterations count in the config
        if daemon_mode:
-            logging.info("Daemon mode enabled, kraken will cause chaos forever")
+            logging.info("Daemon mode enabled, kraken will cause chaos forever\n")
            logging.info("Ignoring the iterations set")
            iterations = float('inf')
        else:
-            logging.info("Daemon mode not enabled, will run through %s iterations"
+            logging.info("Daemon mode not enabled, will run through %s iterations\n"
                         % str(iterations))
            iterations = int(iterations)

@@ -166,24 +204,44 @@ def main(cfg):
        while (int(iteration) < iterations):
            # Inject chaos scenarios specified in the config
            logging.info("Executing scenarios for iteration " + str(iteration))
-            try:
-                # Loop to run the scenarios starts here
-                for scenario in scenarios:
-                    pre_action_output = run_post_action(kubeconfig_path, scenario[1])
-                    runcommand.invoke("powerfulseal autonomous --use-pod-delete-instead-of-ssh-kill"
-                                      " --policy-file %s --kubeconfig %s --no-cloud"
-                                      " --inventory-kubernetes --headless"
-                                      % (scenario[0], kubeconfig_path))
+            if scenarios:
+                try:
+                    # Loop to run the scenarios starts here
+                    for scenario in scenarios:
+                        pre_action_output = run_post_action(kubeconfig_path, scenario[1])
+                        runcommand.invoke("powerfulseal autonomous --use-pod-delete-instead-of-ssh-kill"  # noqa
+                                          " --policy-file %s --kubeconfig %s --no-cloud"
+                                          " --inventory-kubernetes --headless"
+                                          % (scenario[0], kubeconfig_path))
+
+                        logging.info("Scenario: %s has been successfully injected!" % (scenario[0]))
+                        logging.info("Waiting for the specified duration: %s" % (wait_duration))
+                        time.sleep(wait_duration)
+                        failed_post_scenarios = post_actions(kubeconfig_path, scenario,
+                                                             failed_post_scenarios,
+                                                             pre_action_output)
+                        publish_kraken_status(config, failed_post_scenarios)
+                except Exception as e:
+                    logging.error("Failed to run scenario: %s. Encountered the following "
+                                  "exception: %s" % (scenario[0], e))
+
+            # Inject node chaos scenarios specified in the config
+            if node_scenarios:
+                for node_scenario_config in node_scenarios:
+                    with open(node_scenario_config, 'r') as f:
+                        node_scenario_config = yaml.full_load(f)
+                        for node_scenario in node_scenario_config['node_scenarios']:
+                            node_scenario_object = get_node_scenario_object(node_scenario)
+                            if node_scenario['actions']:
+                                for action in node_scenario['actions']:
+                                    inject_node_scenario(action, node_scenario,
+                                                         node_scenario_object)
+                                    logging.info("Waiting for the specified duration: %s"
+                                                 % (wait_duration))
+                                    time.sleep(wait_duration)
+                                    cerberus_integration(config)
+                                    logging.info("")

-                    logging.info("Scenario: %s has been successfully injected!" % (scenario[0]))
-                    logging.info("Waiting for the specified duration: %s" % (wait_duration))
-                    time.sleep(wait_duration)
-                    failed_post_scenarios = post_actions(kubeconfig_path, scenario,
-                                                         failed_post_scenarios, pre_action_output)
-                    publish_kraken_status(config, failed_post_scenarios)
-            except Exception as e:
-                logging.error("Failed to run scenario: %s. Encountered the following exception: %s"
-                              % (scenario[0], e))
            iteration += 1
            logging.info("")
        if failed_post_scenarios:
@@ -0,0 +1,17 @@
+node_scenarios:
+  - actions:                                                        # node chaos scenarios to be injected
+    - node_stop_start_scenario
+    - stop_start_kubelet_scenario
+    - node_crash_scenario
+    node_name:                                                      # node on which scenario has to be injected
+    label_selector: node-role.kubernetes.io/worker                  # when node_name is not specified, a node with matching label_selector is selected for node chaos scenario injection
+    instance_kill_count: 1                                          # number of times to inject each scenario under actions
+    timeout: 120                                                    # duration to wait for completion of node scenario injection
+    cloud_type: aws                                                 # cloud type on which Kubernetes/OpenShift runs
+  - actions:
+    - node_reboot_scenario
+    node_name:
+    label_selector: node-role.kubernetes.io/infra
+    instance_kill_count: 1
+    timeout: 120
+    cloud_type: aws