mirror of
https://github.com/krkn-chaos/krkn.git
synced 2026-08-25 09:27:36 +00:00
Chaos Recommendation Utility (#508)
* application profiling based chaos recommendation * deleted unused dir * Update requirements.txt Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update config.ini Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update Makefile Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update Dockerfile Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update README.md Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> --------- Signed-off-by: Mudit Verma <mudiverm@in.ibm.com>
This commit is contained in:
Mudit Verma
@@ -0,0 +1,9 @@
|
||||
FROM python:3.9
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY . .
|
||||
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
CMD [ "python", "chaos-recommender.py", "-p"]
|
||||
@@ -0,0 +1,6 @@
|
||||
#to build docker image
|
||||
build:
|
||||
docker build -t engineered_chaos .
|
||||
# to run
|
||||
run:
|
||||
docker run -it --rm engineered_chaos
|
||||
@@ -0,0 +1,69 @@
|
||||
# Chaos Recommendation Tool
|
||||
|
||||
This tool, designed for Redhat Kraken, operates through the command line and offers recommendations for chaos testing. It suggests probable chaos test cases that can disrupt application services by analyzing their behavior and assessing their susceptibility to specific fault types.
|
||||
|
||||
This tool profiles an application and gathers telemetry data such as CPU, Memory, and Network usage, analyzing it to suggest probable chaos scenarios. For optimal results, it is recommended to activate the utility while the application is under load.
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
- Openshift Or Kubernetes Environment where the application is hosted
|
||||
- Access to the telemetry data via the exposed Prometheus endpoint
|
||||
- Python3
|
||||
|
||||
## Usage
|
||||
|
||||
1. To run
|
||||
|
||||
```
|
||||
python3 chaos-recommender.py
|
||||
```
|
||||
|
||||
2. Follow the prompts to provide the required information.
|
||||
|
||||
## Configuration
|
||||
|
||||
You can customize the default values by editing the `config.ini` file. The configuration file contains the following options:
|
||||
|
||||
- `[General]`
|
||||
- `application`: Specify the application name.
|
||||
- `namespace`: Specify the namespace name. If you want to profile
|
||||
- `labels`: Specify the labels (not used).
|
||||
- `kubeconfig`: Specify the location of the kubeconfig file (not used).
|
||||
|
||||
- `[Options]`
|
||||
- `prometheus_endpoint`: Specify the prometheus endpoint (must).
|
||||
- `auth_token`: Auth token to connect to prometheus endpoint (must).
|
||||
- `scrape_duration`: For how long data should be fetched, e.g., '1m' (must).
|
||||
- `chaos_library`: "kraken" (currently it only supports kraken). One can make modifications in kraken_chaos_tests.txt though.
|
||||
|
||||
You can also provide the input values through command-line arguments. The following options are available:
|
||||
|
||||
- `-p`, `--prompt`: Prompt all options on console.
|
||||
|
||||
If you provide the input values through command-line arguments, the corresponding config file inputs would be ignored.
|
||||
|
||||
## Docker
|
||||
|
||||
To run chaos recommendation to via Docker. please follow the steps below.
|
||||
|
||||
- Build a docker image `make build`
|
||||
- Run the tool `make run` or alternatively `docker run -it --rm <IMAGE_NAME:TAG> python3 chaos-recommender.py -p`
|
||||
|
||||
PS: Please note that either one should provide populated config.ini during the image build, or use -p flag to ask for a prompt when running in docker.
|
||||
|
||||
## How it works
|
||||
|
||||
After obtaining telemetry data, sourced either locally or from Prometheus, the tool conducts a comprehensive data analysis to detect anomalies. Employing the Z-score method and heatmaps, it identifies outliers by evaluating CPU, memory, and network usage against established limits. Services with Z-scores surpassing a specified threshold are categorized as outliers. This categorization classifies services as network, CPU, or memory-sensitive, consequently leading to the recommendation of relevant test cases.
|
||||
|
||||
## Customizing Thresholds and Options
|
||||
|
||||
You can customize the thresholds and options used for data analysis by modifying the `analysis.py` file. For example, you can adjust the threshold for identifying outliers by changing the value of the `threshold` variable in the `identify_outliers` function.
|
||||
|
||||
## Additional Files
|
||||
|
||||
- `config.ini`: The configuration file containing default values for application, namespace, labels, and kubeconfig.
|
||||
- `requirements.txt`: The file listing the required dependencies for the project.
|
||||
- `Dockerfile`: The Dockerfile used to build the Docker image for the project.
|
||||
- `Makefile`: The file containing commands to build and run the project using `make`.
|
||||
|
||||
Happy Chaos!
|
||||
@@ -0,0 +1,95 @@
|
||||
import pandas as pd
|
||||
import kraken_tests
|
||||
import time
|
||||
|
||||
threshold = .7 # Adjust the threshold as needed
|
||||
heatmap_cpu_threshold = .5
|
||||
heatmap_mem_threshold = .5
|
||||
|
||||
KRAKEN_TESTS_PATH = "./kraken_chaos_tests.txt"
|
||||
|
||||
#Placeholder, this should be done with topology
|
||||
def return_critical_services():
|
||||
return ["web", "cart"]
|
||||
|
||||
|
||||
def load_telemetry_data(file_path):
|
||||
data = pd.read_csv(file_path, delimiter=r"\s+")
|
||||
return data
|
||||
|
||||
def calculate_zscores(data):
|
||||
zscores = pd.DataFrame()
|
||||
zscores["Service"] = data["service"]
|
||||
zscores["CPU"] = (data["CPU"] - data["CPU"].mean()) / data["CPU"].std()
|
||||
zscores["Memory"] = (data["MEM"] - data["MEM"].mean()) / data["MEM"].std()
|
||||
zscores["Network"] = (data["NETWORK"] - data["NETWORK"].mean()) / data["NETWORK"].std()
|
||||
#print("wdfdsfsdfsdfssd")
|
||||
#print(zscores)
|
||||
return zscores
|
||||
|
||||
def identify_outliers(data):
|
||||
outliers_cpu = data[data["CPU"] > threshold]["Service"].tolist()
|
||||
outliers_memory = data[data["Memory"] > threshold]["Service"].tolist()
|
||||
outliers_network = data[data["Network"] > threshold]["Service"].tolist()
|
||||
|
||||
return outliers_cpu, outliers_memory, outliers_network
|
||||
|
||||
|
||||
def get_services_above_heatmap_threshold(dataframe, cpu_threshold, mem_threshold):
|
||||
# Filter the DataFrame based on CPU_HEATMAP and MEM_HEATMAP thresholds
|
||||
filtered_df = dataframe[((dataframe['CPU']/dataframe['CPU_LIMITS']) > cpu_threshold)]
|
||||
# Get the lists of services
|
||||
cpu_services = filtered_df['service'].tolist()
|
||||
|
||||
filtered_df = dataframe[((dataframe['MEM']/dataframe['MEM_LIMITS']) > mem_threshold)]
|
||||
mem_services = filtered_df['service'].tolist()
|
||||
|
||||
return cpu_services, mem_services
|
||||
|
||||
|
||||
def analysis(file_path):
|
||||
# Load the telemetry data from file
|
||||
data = load_telemetry_data(file_path)
|
||||
|
||||
# Calculate Z-scores for CPU, Memory, and Network columns
|
||||
zscores = calculate_zscores(data)
|
||||
|
||||
# Identify outliers
|
||||
outliers_cpu, outliers_memory, outliers_network = identify_outliers(zscores)
|
||||
cpu_services, mem_services = get_services_above_heatmap_threshold(data, heatmap_cpu_threshold, heatmap_mem_threshold)
|
||||
|
||||
# Display the identified outliers
|
||||
print("======================== Profiling ==================================")
|
||||
print("CPU outliers:", outliers_cpu)
|
||||
print("Memory outliers:", outliers_memory)
|
||||
print("Network outliers:", outliers_network)
|
||||
print("===================== HeatMap Analysis ==============================")
|
||||
|
||||
if cpu_services:
|
||||
print("Services with CPU_HEATMAP above threshold:", cpu_services)
|
||||
else:
|
||||
print("There are no services that are using siginificant CPU compared to their assigned limits (infinite in case no limits are set).")
|
||||
if mem_services:
|
||||
print("Services with MEM_HEATMAP above threshold:", mem_services)
|
||||
else:
|
||||
print("There are no services that are using siginificant MEMORY compared to their assigned limits (infinite in case no limits are set).")
|
||||
time.sleep(2)
|
||||
print("======================= Recommendations =============================")
|
||||
if cpu_services:
|
||||
print("Recommended tests for " + str(cpu_services) + " :\n" + str(kraken_tests.get_entries_by_category(KRAKEN_TESTS_PATH, "CPU")) )
|
||||
print("\n")
|
||||
if mem_services:
|
||||
print("Recommended tests for " + str(mem_services) + " :\n" + str(kraken_tests.get_entries_by_category(KRAKEN_TESTS_PATH, "MEM")) )
|
||||
print("\n")
|
||||
|
||||
if outliers_network:
|
||||
print("Recommended tests for " + str(outliers_network) + " :\n" + str(kraken_tests.get_entries_by_category(KRAKEN_TESTS_PATH, "NETWORK")) )
|
||||
print("\n")
|
||||
|
||||
#print("Recommended tests for " + str(return_critical_services()) + " :\n" + str(kraken_tests.get_entries_by_category(KRAKEN_TESTS_PATH, "GENERIC")) )
|
||||
print("\n")
|
||||
print("Please check data in utilisation.txt for further analysis")
|
||||
|
||||
#if __name__ == "__main__":
|
||||
# file_path = "./utilisation.txt" # Replace with the actual file path
|
||||
# analysis(file_path)
|
||||
@@ -0,0 +1,65 @@
|
||||
import argparse
|
||||
import configparser
|
||||
import analysis
|
||||
import prometheus
|
||||
|
||||
def parse_arguments():
|
||||
parser = argparse.ArgumentParser(description="Command-line tool")
|
||||
parser.add_argument("-p", "--prompt", action="store_true", help="Prompt for input")
|
||||
return parser.parse_args()
|
||||
|
||||
def read_configuration():
|
||||
config = configparser.ConfigParser()
|
||||
config.read("./config.ini")
|
||||
|
||||
application = config.get("General", "application", fallback="")
|
||||
namespace = config.get("General", "namespace", fallback="")
|
||||
labels = config.get("General", "labels", fallback="")
|
||||
kubeconfig = config.get("General", "kubeconfig", fallback="~/.kube/config.yaml")
|
||||
|
||||
prometheus_endpoint = config.get("Options", "prometheus_endpoint", fallback="")
|
||||
auth_token = config.get("Options", "auth_token", fallback="")
|
||||
scrape_duration = config.get("Options", "scrape_duration", fallback="1m")
|
||||
chaos_library = config.get("Options", "chaos_library", fallback="kraken")
|
||||
|
||||
return application, namespace, labels, kubeconfig, prometheus_endpoint, auth_token, scrape_duration, chaos_library
|
||||
|
||||
def prompt_input(prompt, default_value):
|
||||
user_input = input(f"{prompt} [{default_value}]: ")
|
||||
if user_input.strip():
|
||||
return user_input
|
||||
return default_value
|
||||
|
||||
def main():
|
||||
args = parse_arguments()
|
||||
|
||||
application, namespace, labels, kubeconfig, prometheus_endpoint, auth_token, scrape_duration, chaos_library = read_configuration()
|
||||
|
||||
if args.prompt:
|
||||
application = prompt_input("Application name", application)
|
||||
namespace = prompt_input("Namespace", namespace)
|
||||
labels = prompt_input("Labels", labels)
|
||||
kubeconfig = prompt_input("Kubeconfig file location", kubeconfig)
|
||||
prometheus_endpoint = prompt_input("Prometheus endpoint", prometheus_endpoint)
|
||||
auth_token = prompt_input("Auth Token for Prometheus", auth_token)
|
||||
scrape_duration = prompt_input("Scrape duration", scrape_duration)
|
||||
chaos_library = prompt_input("Chaos library", chaos_library)
|
||||
|
||||
print("============================INPUTS===================================")
|
||||
print(f"Application: {application}")
|
||||
print(f"Namespace: {namespace}")
|
||||
print(f"Labels: {labels}")
|
||||
print(f"Kubeconfig: {kubeconfig}")
|
||||
print(f"Prometheus endpoint: {prometheus_endpoint}")
|
||||
print(f"Scrape duration: {scrape_duration}")
|
||||
print(f"Chaos library: {chaos_library}")
|
||||
print("=====================================================================")
|
||||
print("Starting Analysis ...")
|
||||
print("Fetching the Telemetry data")
|
||||
|
||||
file_path = prometheus.fetch_utilization_from_prometheus(prometheus_endpoint, auth_token, namespace, scrape_duration)
|
||||
|
||||
analysis.analysis(file_path)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,11 @@
|
||||
[General]
|
||||
application = openshift-etcd
|
||||
namespace = openshift-etcd
|
||||
labels = app=openshift-etcd
|
||||
kubeconfig = ~/.kube/config.yaml
|
||||
|
||||
[Options]
|
||||
prometheus_endpoint = <Prometheus_Endpoint>
|
||||
auth_token = <Auth_Token>
|
||||
scrape_duration = 10m
|
||||
chaos_library = "kraken"
|
||||
@@ -0,0 +1,20 @@
|
||||
[CPU]
|
||||
node_cpu_hog
|
||||
|
||||
[NETWORK]
|
||||
application_outage
|
||||
node_network_chaos
|
||||
pod_network_chaos
|
||||
|
||||
[MEM]
|
||||
node_memory_hog
|
||||
pvc_disk_fill
|
||||
|
||||
[GENERIC]
|
||||
pod_failure
|
||||
container_failure
|
||||
node_failure
|
||||
zone_outage
|
||||
time_skew
|
||||
namespace_failure
|
||||
power_outage
|
||||
@@ -0,0 +1,30 @@
|
||||
def get_entries_by_category(filename, category):
|
||||
# Read the file
|
||||
with open(filename, 'r') as file:
|
||||
content = file.read()
|
||||
|
||||
# Split the content into sections based on the square brackets
|
||||
sections = content.split('\n\n')
|
||||
|
||||
# Define the categories
|
||||
valid_categories = ['CPU', 'NETWORK', 'MEM', 'GENERIC']
|
||||
|
||||
# Validate the provided category
|
||||
if category not in valid_categories:
|
||||
return []
|
||||
|
||||
# Find the section corresponding to the specified category
|
||||
target_section = None
|
||||
for section in sections:
|
||||
if section.startswith(f"[{category}]"):
|
||||
target_section = section
|
||||
break
|
||||
|
||||
# If the category section was not found, return an empty list
|
||||
if target_section is None:
|
||||
return []
|
||||
|
||||
# Extract the entries from the category section
|
||||
entries = [entry.strip() for entry in target_section.split('\n') if entry and not entry.startswith('[')]
|
||||
|
||||
return entries
|
||||
@@ -0,0 +1,118 @@
|
||||
import random
|
||||
from prometheus_api_client import PrometheusConnect
|
||||
import pandas as pd
|
||||
from functools import reduce
|
||||
import urllib3
|
||||
|
||||
|
||||
saved_metrics_path = "./utilisation.txt"
|
||||
duration = "10m"
|
||||
|
||||
def convert_data_to_dataframe(data, label):
|
||||
df = pd.DataFrame()
|
||||
df['service'] = [item['metric']['pod'] for item in data]
|
||||
df[label] = [item['value'][1] for item in data]
|
||||
|
||||
return df
|
||||
|
||||
|
||||
def get_value_from_data(data, service):
|
||||
df = pd.DataFrame()
|
||||
df['service'] = [item['metric']['pod'] for item in data]
|
||||
df[label] = [item['value'][1] for item in data]
|
||||
|
||||
return df
|
||||
|
||||
|
||||
def convert_data(data, service):
|
||||
|
||||
result = {}
|
||||
for entry in data:
|
||||
pod_name = entry['metric']['pod']
|
||||
value = entry['value'][1]
|
||||
result[pod_name] = value
|
||||
#print(result)
|
||||
return result.get(service, '100000000000') # for those pods whose limits are not defined they can take as much resources, there assigning a very high value
|
||||
|
||||
def save_utilization_to_file(cpu_data, cpu_limits_result, mem_data, mem_limits_result, network_data, filename):
|
||||
|
||||
#print(cpu_data)
|
||||
df_cpu = convert_data_to_dataframe(cpu_data, "CPU")
|
||||
|
||||
|
||||
merged_df = pd.DataFrame(columns=['service','CPU','CPU_LIMITS','MEM','MEM_LIMITS','NETWORK'])
|
||||
|
||||
|
||||
services = df_cpu.service.unique()
|
||||
|
||||
print(services)
|
||||
|
||||
for s in services:
|
||||
|
||||
new_row = {"service": s, "CPU" : convert_data(cpu_data, s),
|
||||
"CPU_LIMITS" : convert_data(cpu_limits_result, s),
|
||||
"MEM" : convert_data(mem_data, s), "MEM_LIMITS" : convert_data(mem_limits_result, s),
|
||||
"NETWORK" : convert_data(network_data, s)}
|
||||
merged_df = merged_df.append(new_row, ignore_index=True)
|
||||
|
||||
|
||||
# Convert columns to string
|
||||
merged_df['CPU'] = merged_df['CPU'].astype(str)
|
||||
merged_df['MEM'] = merged_df['MEM'].astype(str)
|
||||
merged_df['CPU_LIMITS'] = merged_df['CPU_LIMITS'].astype(str)
|
||||
merged_df['MEM_LIMITS'] = merged_df['MEM_LIMITS'].astype(str)
|
||||
merged_df['NETWORK'] = merged_df['NETWORK'].astype(str)
|
||||
|
||||
# Extract integer part before the decimal point
|
||||
merged_df['CPU'] = merged_df['CPU'].str.split('.').str[0]
|
||||
merged_df['MEM'] = merged_df['MEM'].str.split('.').str[0]
|
||||
merged_df['CPU_LIMITS'] = merged_df['CPU_LIMITS'].str.split('.').str[0]
|
||||
merged_df['MEM_LIMITS'] = merged_df['MEM_LIMITS'].str.split('.').str[0]
|
||||
merged_df['NETWORK'] = merged_df['NETWORK'].str.split('.').str[0]
|
||||
|
||||
merged_df.to_csv(filename, sep='\t', index=False)
|
||||
|
||||
def fetch_utilization_from_prometheus(prometheus_endpoint, auth_token, namespace, n):
|
||||
urllib3.disable_warnings()
|
||||
prometheus = PrometheusConnect(url=prometheus_endpoint, headers={'Authorization':'Bearer {}'.format(auth_token)}, disable_ssl=True)
|
||||
|
||||
# Fetch CPU utilization
|
||||
cpu_query = 'sum (rate (container_cpu_usage_seconds_total{image!="", namespace="%s"}[%s])) by (pod) *1000' % (namespace,duration)
|
||||
print(cpu_query)
|
||||
cpu_result = prometheus.custom_query(cpu_query)
|
||||
cpu_data = cpu_result
|
||||
|
||||
|
||||
cpu_limits_query = '(sum by (pod) (kube_pod_container_resource_limits{resource="cpu", namespace="%s"}))*1000' %(namespace)
|
||||
print(cpu_limits_query)
|
||||
cpu_limits_result = prometheus.custom_query(cpu_limits_query)
|
||||
|
||||
|
||||
mem_query = 'sum by (pod) (avg_over_time(container_memory_usage_bytes{image!="", namespace="%s"}[%s]))' % (namespace, duration)
|
||||
print(mem_query)
|
||||
mem_result = prometheus.custom_query(mem_query)
|
||||
mem_data = mem_result
|
||||
|
||||
mem_limits_query = 'sum by (pod) (kube_pod_container_resource_limits{resource="memory", namespace="%s"}) ' %(namespace)
|
||||
print(mem_limits_query)
|
||||
mem_limits_result = prometheus.custom_query(mem_limits_query)
|
||||
|
||||
|
||||
network_query = 'sum by (pod) ((avg_over_time(container_network_transmit_bytes_total{namespace="%s"}[%s])) + \
|
||||
(avg_over_time(container_network_receive_bytes_total{namespace="%s"}[%s])))' % (namespace, duration, namespace, duration)
|
||||
network_result = prometheus.custom_query(network_query)
|
||||
print(network_query)
|
||||
network_data = network_result
|
||||
|
||||
|
||||
save_utilization_to_file(cpu_data, cpu_limits_result, mem_data, mem_limits_result, network_data, saved_metrics_path)
|
||||
return saved_metrics_path
|
||||
|
||||
|
||||
# Example usage
|
||||
#prometheus_endpoint = "http://localhost:9090"
|
||||
#namespace ="robot-shop"
|
||||
#n = 1 # Number of minutes
|
||||
|
||||
#fetch_utilization_from_prometheus(prometheus_endpoint, namespace, n)
|
||||
#save_utilization_to_file(cpu_data, mem_data, network_data, 'utilization.txt')
|
||||
@@ -0,0 +1,2 @@
|
||||
pandas<2.0.0
|
||||
prometheus-api-client
|
||||
Reference in New Issue
Block a user