krkn integration of telemetry events collection (#523 )

* function package refactoring in krkn-lib * cluster events collection flag * krkn-lib version bump requirements * dockerfile bump
Bump telemetry archive default size to 500MB
2026-02-19 20:40:33 +00:00 · 2023-10-31 14:31:33 -04:00 · 2023-10-30 12:50:04 -04:00 · 2023-10-30 17:31:48 +01:00 · 2023-10-30 15:51:09 +01:00 · 2023-10-27 18:09:46 +02:00
63 changed files with 1761 additions and 537 deletions
--- a/.github/workflows/functional_tests.yaml
+++ b/.github/workflows/functional_tests.yaml
@@ -34,7 +34,7 @@ jobs:
      - name: Check out Kraken
        uses: actions/checkout@v3
      - name: Checkout Pull Request
-        run: hub pr checkout ${{ github.event.issue.number }}
+        run: gh pr checkout ${{ github.event.issue.number }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      - name: Install OC CLI
--- a/README.md
+++ b/README.md
@@ -1,5 +1,6 @@
 # Krkn aka Kraken
 [![Docker Repository on Quay](https://quay.io/repository/redhat-chaos/krkn/status "Docker Repository on Quay")](https://quay.io/repository/redhat-chaos/krkn?tab=tags&tag=latest)
+![Workflow-Status](https://github.com/redhat-chaos/krkn/actions/workflows/docker-image.yml/badge.svg)

 ![Krkn logo](media/logo.png)

@@ -29,6 +30,8 @@ The guide is hosted at https://redhat-chaos.github.io/krkn.
 ### How to Get Started
 Instructions on how to setup, configure and run Kraken can be found at [Installation](docs/installation.md).

+You may consider utilizing the chaos recommendation tool prior to initiating the chaos runs to profile the application service(s) under test. This tool discovers a list of Krkn scenarios with a high probability of causing failures or disruptions to your application service(s). The tool can be accessed at [Chaos-Recommender](utils/chaos_recommender/README.md).
+
 See the [getting started doc](docs/getting_started.md) on support on how to get started with your own custom scenario or editing current scenarios for your specific usage.

 After installation, refer back to the below sections for supported scenarios and how to tweak the kraken config to load them on your cluster.
@@ -65,7 +68,7 @@ Scenario type               | Kubernetes    | OpenShift
 [Time Scenarios](docs/time_scenarios.md) | :x: | :heavy_check_mark: |
 [Hog Scenarios: CPU, Memory](docs/arcaflow_scenarios.md) | :heavy_check_mark: | :heavy_check_mark: |
 [Cluster Shut Down Scenarios](docs/cluster_shut_down_scenarios.md) | :heavy_check_mark: | :heavy_check_mark: |
-[Namespace Scenarios](docs/namespace_scenarios.md) | :heavy_check_mark: | :heavy_check_mark: |
+[Service Disruption Scenarios](docs/service_disruption_scenarios.md.md) | :heavy_check_mark: | :heavy_check_mark: |
 [Zone Outage Scenarios](docs/zone_outage.md) | :heavy_check_mark: | :heavy_check_mark: |
 [Application_outages](docs/application_outages.md) | :heavy_check_mark: | :heavy_check_mark: |
 [PVC scenario](docs/pvc_scenario.md) | :heavy_check_mark: | :heavy_check_mark: |
@@ -106,11 +109,13 @@ Information on enabling and leveraging this feature can be found [here](docs/SLO

 Kraken supports injecting faults into [Open Cluster Management (OCM)](https://open-cluster-management.io/) and [Red Hat Advanced Cluster Management for Kubernetes (ACM)](https://www.redhat.com/en/technologies/management/advanced-cluster-management) managed clusters through [ManagedCluster Scenarios](docs/managedcluster_scenarios.md).

+
 ### Blogs and other useful resources
 - Blog post on introduction to Kraken: https://www.openshift.com/blog/introduction-to-kraken-a-chaos-tool-for-openshift/kubernetes
 - Discussion and demo on how Kraken can be leveraged to ensure OpenShift is reliable, performant and scalable: https://www.youtube.com/watch?v=s1PvupI5sD0&ab_channel=OpenShift
 - Blog post emphasizing the importance of making Chaos part of Performance and Scale runs to mimic the production environments: https://www.openshift.com/blog/making-chaos-part-of-kubernetes/openshift-performance-and-scalability-tests
 - Blog post on findings from Chaos test runs: https://cloud.redhat.com/blog/openshift/kubernetes-chaos-stories
+- Discussion with CNCF TAG App Delivery on Krkn workflow, features and addition to CNCF sandbox: [Github](https://github.com/cncf/sandbox/issues/44), [Tracker](https://github.com/cncf/tag-app-delivery/issues/465), [recording](https://www.youtube.com/watch?v=nXQkBFK_MWc&t=722s)


 ### Roadmap
@@ -128,5 +133,5 @@ Please read [this file]((CI/README.md#adding-a-test-case)) for more information

 ### Community
 Key Members(slack_usernames/full name): paigerube14/Paige Rubendall, mffiedler/Mike Fiedler, ravielluri/Naga Ravi Chaitanya Elluri.
-* [**#sig-scalability on Kubernetes Slack**](https://kubernetes.slack.com)
+* [**#krkn on Kubernetes Slack**](https://kubernetes.slack.com)
 * [**#forum-chaos on CoreOS Slack internal to Red Hat**](https://coreos.slack.com)
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -2,10 +2,14 @@

 Following are a list of enhancements that we are planning to work on adding support in Krkn. Of course any help/contributions are greatly appreciated.

- [Ability to run multiple chaos scenarios in parallel under load to mimic real world outages](https://github.com/redhat-chaos/krkn/issues/424)
- [Centralized storage for chaos experiments artifacts](https://github.com/redhat-chaos/krkn/issues/423)
- [Support for causing DNS outages](https://github.com/redhat-chaos/krkn/issues/394)
- [Support for pod level network traffic shaping](https://github.com/redhat-chaos/krkn/issues/393)
- [Ability to visualize the metrics that are being captured by Kraken and stored in Elasticsearch](https://github.com/redhat-chaos/krkn/issues/124)
- Support for running all the scenarios of Kraken on Kubernetes distribution - see https://github.com/redhat-chaos/krkn/issues/185, https://github.com/redhat-chaos/krkn/issues/186
- Continue to improve [Chaos Testing Guide](https://redhat-chaos.github.io/krkn) in terms of adding best practices, test environment recommendations and scenarios to make sure the OpenShift platform, as well the applications running on top it, are resilient and performant under chaotic conditions.
+- [ ] [Ability to run multiple chaos scenarios in parallel under load to mimic real world outages](https://github.com/redhat-chaos/krkn/issues/424)
+- [x] [Centralized storage for chaos experiments artifacts](https://github.com/redhat-chaos/krkn/issues/423)
+- [ ] [Support for causing DNS outages](https://github.com/redhat-chaos/krkn/issues/394)
+- [x] [Chaos recommender](https://github.com/redhat-chaos/krkn/tree/main/utils/chaos-recommender) to suggest scenarios having probability of impacting the service under test using profiling results 
+- [ ] Chaos AI integration to improve and automate test coverage
+- [x] [Support for pod level network traffic shaping](https://github.com/redhat-chaos/krkn/issues/393)
+- [ ] [Ability to visualize the metrics that are being captured by Kraken and stored in Elasticsearch](https://github.com/redhat-chaos/krkn/issues/124)
+- [ ] Support for running all the scenarios of Kraken on Kubernetes distribution - see https://github.com/redhat-chaos/krkn/issues/185, https://github.com/redhat-chaos/krkn/issues/186
+- [ ] Continue to improve [Chaos Testing Guide](https://redhat-chaos.github.io/krkn) in terms of adding best practices, test environment recommendations and scenarios to make sure the OpenShift platform, as well the applications running on top it, are resilient and performant under chaotic conditions.
+- [ ] [Switch documentation references to Kubernetes](https://github.com/redhat-chaos/krkn/issues/495)
+- [ ] [OCP and Kubernetes functionalities segregation](https://github.com/redhat-chaos/krkn/issues/497)
--- a/config/alerts
+++ b/config/alerts
@@ -16,6 +16,42 @@
  description: etcd leader changes observed
  severity: warning

+- expr: (last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m]))*100 > 95
+  description: etcd cluster database is running full. 
+  severity: critical
+  
+- expr: (last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) < 0.5
+  description: etcd database size in use is less than 50% of the actual allocated storage.
+  severity: warning  
+
+- expr: rate(etcd_server_proposals_failed_total{job=~".*etcd.*"}[15m]) > 5
+  description: etcd cluster has high number of proposal failures.
+  severity: warning 
+
+- expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~".*etcd.*"}[5m])) > 0.15
+  description: etcd cluster member communication is slow.
+  severity: warning 
+
+- expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~".*etcd.*", grpc_method!="Defragment", grpc_type="unary"}[5m])) without(grpc_type)) > 0.15
+  description: etcd grpc requests are slow.
+  severity: critical
+      
+- expr: 100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*", grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[5m])) without (grpc_type, grpc_code) / sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) without (grpc_type, grpc_code) > 5
+  description: etcd cluster has high number of failed grpc requests.
+  severity: critical      
+      
+- expr: etcd_server_has_leader{job=~".*etcd.*"} == 0
+  description: etcd cluster has no leader.
+  severity: warning    
+    
+- expr: sum(up{job=~".*etcd.*"} == bool 1) without (instance) < ((count(up{job=~".*etcd.*"}) without (instance) + 1) / 2)
+  description: etcd cluster has insufficient number of members.
+  severity: warning      
+
+- expr: max without (endpoint) ( sum without (instance) (up{job=~".*etcd.*"} == bool 0) or count without (To) ( sum without (instance) (rate(etcd_network_peer_sent_failures_total{job=~".*etcd.*"}[120s])) > 0.01 )) > 0
+  description: etcd cluster members are down.
+  severity: warning  
+
 # API server
 - expr: avg_over_time(histogram_quantile(0.99, sum(irate(apiserver_request_duration_seconds_bucket{apiserver="kube-apiserver", verb=~"POST|PUT|DELETE|PATCH", subresource!~"log|exec|portforward|attach|proxy"}[2m])) by (le, resource, verb))[10m:]) > 1
  description: 10 minutes avg. 99th mutating API call latency for {{$labels.verb}}/{{$labels.resource}} higher than 1 second. {{$value}}s
--- a/config/config.yaml
+++ b/config/config.yaml
@@ -6,9 +6,42 @@ kraken:
    signal_state: RUN                                      # Will wait for the RUN signal when set to PAUSE before running the scenarios, refer docs/signal.md for more details
    signal_address: 0.0.0.0                                # Signal listening address
    port: 8081                                             # Signal port
-    chaos_scenarios:                                       # List of policies/chaos scenarios to load
+    chaos_scenarios:
+        # List of policies/chaos scenarios to load
+        - arcaflow_scenarios:
+            - scenarios/arcaflow/cpu-hog/input.yaml
+            - scenarios/arcaflow/memory-hog/input.yaml
+            - scenarios/arcaflow/io-hog/input.yaml
        - application_outages:
-              - scenarios/openshift/app_outage.yaml
+            - scenarios/openshift/app_outage.yaml
+        - container_scenarios:                             # List of chaos pod scenarios to load
+            - -    scenarios/openshift/container_etcd.yml
+        - plugin_scenarios:
+            - scenarios/openshift/etcd.yml
+            - scenarios/openshift/regex_openshift_pod_kill.yml
+            - scenarios/openshift/vmware_node_scenarios.yml
+            - scenarios/openshift/network_chaos_ingress.yml
+            - scenarios/openshift/prom_kill.yml
+        - node_scenarios:                                  # List of chaos node scenarios to load
+            -   scenarios/openshift/node_scenarios_example.yml
+        - plugin_scenarios:
+            - scenarios/openshift/openshift-apiserver.yml
+            - scenarios/openshift/openshift-kube-apiserver.yml
+        - time_scenarios:                                  # List of chaos time scenarios to load
+            - scenarios/openshift/time_scenarios_example.yml
+        - cluster_shut_down_scenarios:
+            - - scenarios/openshift/cluster_shut_down_scenario.yml
+              - scenarios/openshift/post_action_shut_down.py
+        - service_disruption_scenarios:
+             - - scenarios/openshift/regex_namespace.yaml
+             - - scenarios/openshift/ingress_namespace.yaml
+               - scenarios/openshift/post_action_namespace.py
+        - zone_outages:
+            - scenarios/openshift/zone_outage.yaml
+        - pvc_scenarios:
+            - scenarios/openshift/pvc_scenario.yaml
+        - network_chaos:
+            - scenarios/openshift/network_chaos.yaml

 cerberus:
    cerberus_enabled: False                                # Enable it when cerberus is previously installed
@@ -43,9 +76,17 @@ telemetry:
    archive_path: /tmp                                      # local path where the archive files will be temporarly stored
    max_retries: 0                                          # maximum number of upload retries (if 0 will retry forever)
    run_tag: ''                                             # if set, this will be appended to the run folder in the bucket (useful to group the runs)
-    archive_size: 10000                                     # the size of the prometheus data archive size in KB. The lower the size of archive is
+    archive_size: 500000                                     # the size of the prometheus data archive size in KB. The lower the size of archive is
                                                            # the higher the number of archive files will be produced and uploaded (and processed by backup_threads
                                                            # simultaneously).
                                                            # For unstable/slow connection is better to keep this value low
                                                            # increasing the number of backup_threads, in this way, on upload failure, the retry will happen only on the
-                                                            # failed chunk without affecting the whole upload.
+                                                            # failed chunk without affecting the whole upload.
+    logs_backup: True
+    logs_filter_patterns:
+     - "(\\w{3}\\s\\d{1,2}\\s\\d{2}:\\d{2}:\\d{2}\\.\\d+).+"         # Sep 9 11:20:36.123425532
+     - "kinit (\\d+/\\d+/\\d+\\s\\d{2}:\\d{2}:\\d{2})\\s+"          # kinit 2023/09/15 11:20:36 log
+     - "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z).+"      # 2023-09-15T11:20:36.123425532Z log
+    oc_cli_path: /usr/bin/oc                                # optional, if not specified will be search in $PATH
+    events_backup: True                                     # enables/disables cluster events collection
+
--- a/config/config_performance.yaml
+++ b/config/config_performance.yaml
@@ -13,6 +13,7 @@ kraken:
        -   plugin_scenarios:                                 # List of chaos pod scenarios to load
            - scenarios/openshift/etcd.yml
            - scenarios/openshift/regex_openshift_pod_kill.yml
+            - scenarios/openshift/prom_kill.yml
        -   node_scenarios:                                # List of chaos node scenarios to load
            -   scenarios/openshift/node_scenarios_example.yml
        -   plugin_scenarios:
@@ -20,13 +21,10 @@ kraken:
            - scenarios/openshift/openshift-kube-apiserver.yml
        -   time_scenarios:                                # List of chaos time scenarios to load
            - scenarios/openshift/time_scenarios_example.yml
-        -   litmus_scenarios:                              # List of litmus scenarios to load
-            - - https://hub.litmuschaos.io/api/chaos/1.10.0?file=charts/generic/node-cpu-hog/rbac.yaml
-              - scenarios/openshift/node_cpu_hog_engine.yaml
        -   cluster_shut_down_scenarios:
            - - scenarios/openshift/cluster_shut_down_scenario.yml
              - scenarios/openshift/post_action_shut_down.py
-        -   namespace_scenarios:
+        -   service_disruption_scenarios:
            - scenarios/openshift/regex_namespace.yaml
            - scenarios/openshift/ingress_namespace.yaml
        -   zone_outages:
@@ -60,3 +58,27 @@ tunings:
    wait_duration: 60                                      # Duration to wait between each chaos scenario
    iterations: 1                                          # Number of times to execute the scenarios
    daemon_mode: False                                     # Iterations are set to infinity which means that the kraken will cause chaos forever
+
+telemetry:
+    enabled: False                                           # enable/disables the telemetry collection feature
+    api_url: https://ulnmf9xv7j.execute-api.us-west-2.amazonaws.com/production #telemetry service endpoint
+    username: username                                      # telemetry service username
+    password: password                                      # telemetry service password
+    prometheus_backup: True                                 # enables/disables prometheus data collection
+    full_prometheus_backup: False                           # if is set to False only the /prometheus/wal folder will be downloaded.
+    backup_threads: 5                                       # number of telemetry download/upload threads
+    archive_path: /tmp                                      # local path where the archive files will be temporarly stored
+    max_retries: 0                                          # maximum number of upload retries (if 0 will retry forever)
+    run_tag: ''                                             # if set, this will be appended to the run folder in the bucket (useful to group the runs)
+    archive_size: 500000                                     # the size of the prometheus data archive size in KB. The lower the size of archive is
+                                                            # the higher the number of archive files will be produced and uploaded (and processed by backup_threads
+                                                            # simultaneously).
+                                                            # For unstable/slow connection is better to keep this value low
+                                                            # increasing the number of backup_threads, in this way, on upload failure, the retry will happen only on the
+                                                            # failed chunk without affecting the whole upload.
+    logs_backup: True
+    logs_filter_patterns:
+     - "(\\w{3}\\s\\d{1,2}\\s\\d{2}:\\d{2}:\\d{2}\\.\\d+).+"         # Sep 9 11:20:36.123425532
+     - "kinit (\\d+/\\d+/\\d+\\s\\d{2}:\\d{2}:\\d{2})\\s+"          # kinit 2023/09/15 11:20:36 log
+     - "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z).+"      # 2023-09-15T11:20:36.123425532Z log
+    oc_cli_path: /usr/bin/oc                                # optional, if not specified will be search in $PATH
--- a/config/metrics-aggregated.yaml
+++ b/config/metrics-aggregated.yaml
@@ -139,6 +139,39 @@ metrics:
  - query: histogram_quantile(0.99,sum(rate(etcd_request_duration_seconds_bucket[2m])) by (le,operation,apiserver)) > 0
    metricName: P99APIEtcdRequestLatency

+  - query: sum(grpc_server_started_total{namespace="openshift-etcd",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{namespace="openshift-etcd",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})
+    metricName: ActiveWatchStreams
+
+  - query: sum(grpc_server_started_total{namespace="openshift-etcd",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{namespace="openshift-etcd",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})
+    metricName: ActiveLeaseStreams
+
+  - query: sum(rate(etcd_debugging_snap_save_total_duration_seconds_sum{namespace="openshift-etcd"}[2m]))
+    metricName: snapshotSaveLatency
+
+  - query: sum(rate(etcd_server_heartbeat_send_failures_total{namespace="openshift-etcd"}[2m]))
+    metricName: HeartBeatFailures
+
+  - query: sum(rate(etcd_server_health_failures{namespace="openshift-etcd"}[2m]))
+    metricName: HealthFailures
+
+  - query: sum(rate(etcd_server_slow_apply_total{namespace="openshift-etcd"}[2m]))
+    metricName: SlowApplies
+
+  - query: sum(rate(etcd_server_slow_read_indexes_total{namespace="openshift-etcd"}[2m]))
+    metricName: SlowIndexRead
+
+  - query: sum(etcd_server_proposals_pending)
+    metricName: PendingProposals
+
+  - query: histogram_quantile(1.0, sum(rate(etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket[1m])) by (le, instance))
+    metricName: CompactionMaxPause
+
+  - query: sum by (instance) (apiserver_storage_objects)
+    metricName: etcdTotalObjectCount 
+
+  - query: topk(500, max by(resource) (apiserver_storage_objects))
+    metricName: etcdTopObectCount    
+
 # Cluster metrics
  - query: count(kube_namespace_created)
    metricName: namespaceCount
--- a/config/recommender_config.yaml
+++ b/config/recommender_config.yaml
@@ -0,0 +1,29 @@
+application: openshift-etcd
+namespace: openshift-etcd
+labels: app=openshift-etcd
+kubeconfig: ~/.kube/config.yaml
+prometheus_endpoint: <Prometheus_Endpoint>
+auth_token: <Auth_Token>
+scrape_duration: 10m
+chaos_library: "kraken"
+log_level: INFO
+
+# for output purpose only do not change if not needed
+chaos_tests:
+  GENERIC:
+    - pod_failure
+    - container_failure
+    - node_failure
+    - zone_outage
+    - time_skew
+    - namespace_failure
+    - power_outage
+  CPU:
+    - node_cpu_hog
+  NETWORK:
+    - application_outage
+    - node_network_chaos
+    - pod_network_chaos
+  MEM:
+    - node_memory_hog
+    - pvc_disk_fill
--- a/containers/Dockerfile
+++ b/containers/Dockerfile
@@ -14,7 +14,7 @@ COPY --from=azure-cli /usr/local/bin/az /usr/bin/az
 # Install dependencies
 RUN yum install -y git python39 python3-pip jq gettext wget && \
    python3.9 -m pip install -U pip && \
-    git clone https://github.com/redhat-chaos/krkn.git --branch v1.4.1 /root/kraken && \
+    git clone https://github.com/redhat-chaos/krkn.git --branch v1.5.0 /root/kraken && \
    mkdir -p /root/.kube && cd /root/kraken && \
    pip3.9 install -r requirements.txt && \
    pip3.9 install virtualenv && \
--- a/containers/Dockerfile-ppc64le
+++ b/containers/Dockerfile-ppc64le
@@ -14,7 +14,7 @@ COPY --from=azure-cli /usr/local/bin/az /usr/bin/az
 # Install dependencies
 RUN yum install -y git python39 python3-pip jq gettext wget && \
    python3.9 -m pip install -U pip && \
-    git clone https://github.com/redhat-chaos/krkn.git --branch v1.4.1 /root/kraken && \
+    git clone https://github.com/redhat-chaos/krkn.git --branch v1.5.0 /root/kraken && \
    mkdir -p /root/.kube && cd /root/kraken && \
    pip3.9 install -r requirements.txt && \
    pip3.9 install virtualenv && \
--- a/containers/README.md
+++ b/containers/README.md
@@ -1,27 +1,20 @@

 ### Kraken image

-  
-
 Container image gets automatically built by quay.io at [Kraken image](https://quay.io/redhat-chaos/krkn).

-  

 ### Run containerized version

 Refer [instructions](https://github.com/redhat-chaos/krkn/blob/main/docs/installation.md#run-containerized-version) for information on how to run the containerized version of kraken.

-  
-  

 ### Run Custom Kraken Image

 Refer to [instructions](https://github.com/redhat-chaos/krkn/blob/main/containers/build_own_image-README.md) for information on how to run a custom containerized version of kraken using podman.

-  
-  

-### Kraken as a KubeApp
+### Kraken as a KubeApp ( Unsupported and not recommended )

 #### GENERAL NOTES:

@@ -50,4 +43,4 @@ To run containerized Kraken as a Kubernetes/OpenShift Deployment, follow these s
 8. Create a ConfigMap named scenarios-kube-config using `kubectl create configmap scenarios-kube-config --from-file=<path_to_kraken>/scenarios/kube` 
 9. Create a service account to run the kraken pod `kubectl create serviceaccount useroot`.
 10.  In Openshift, add privileges to service account and execute `oc adm policy add-scc-to-user privileged -z useroot`.
-11.  Create a Job using `kubectl apply -f <path_to_kraken>/containers/kraken.yml` and monitor the status using `oc get jobs` and `oc get pods`.
+11.  Create a Job using `kubectl apply -f <path_to_kraken>/containers/kraken.yml` and monitor the status using `oc get jobs` and `oc get pods`.
--- a/containers/build_own_image-README.md
+++ b/containers/build_own_image-README.md
@@ -1,13 +1,13 @@
 # Building your own Kraken image

-1. Git clone the Kraken repository using `git clone https://github.com/openshift-scale/kraken.git`.
+1. Git clone the Kraken repository using `git clone https://github.com/redhat-chaos/krkn.git`.
 2. Modify the python code and yaml files to address your needs.
 3. Execute `podman build -t <new_image_name>:latest .` in the containers directory within kraken to build an image from a Dockerfile.
 4. Execute `podman run --detach --name <container_name> <new_image_name>:latest` to start a container based on your new image.

 # Building the Kraken image on IBM Power (ppc64le)

-1. Git clone the Kraken repository using `git clone https://github.com/cloud-bulldozer/kraken.git` on an IBM Power Systems server.
+1. Git clone the Kraken repository using `git clone https://github.com/redhat-chaos/krkn.git` on an IBM Power Systems server.
 2. Modify the python code and yaml files to address your needs.
 3. Execute `podman build -t <new_image_name>:latest -f Dockerfile-ppc64le` in the containers directory within kraken to build an image from the Dockerfile for Power.
 4. Execute `podman run --detach --name <container_name> <new_image_name>:latest` to start a container based on your new image.
--- a/docs/arcaflow_scenarios.md
+++ b/docs/arcaflow_scenarios.md
@@ -7,6 +7,7 @@ The engine uses containers to execute plugins and runs them either locally in Do
 #### Hog scenarios:
 - [CPU Hog](arcaflow_scenarios/cpu_hog.md)
 - [Memory Hog](arcaflow_scenarios/memory_hog.md)
+- [I/O Hog](arcaflow_scenarios/io_hog.md)


 ### Prequisites
@@ -64,4 +65,6 @@ Each step is represented by a container that will be executed from the deployer
 Note that we provide the scenarios as a template, but they can be manipulated to define more complex workflows.
 To have more details regarding the arcaflow workflows architecture and syntax it is suggested to refer to the [Arcaflow Documentation](https://arcalot.io/arcaflow/).

-
+This edit is no longer in quay image
+Working on fix in ticket: https://issues.redhat.com/browse/CHAOS-494
+This will effect all versions 4.12 and higher of OpenShift
--- a/docs/arcaflow_scenarios/io_hog.md
+++ b/docs/arcaflow_scenarios/io_hog.md
@@ -0,0 +1,21 @@
+# I/O Hog
+This scenario is based on the arcaflow [arcaflow-plugin-stressng](https://github.com/arcalot/arcaflow-plugin-stressng) plugin. 
+The purpose of this scenario is to create disk pressure on a particular node of the Kubernetes/OpenShift cluster for a time span.
+The scenario allows to attach a node path to the pod as a `hostPath` volume.
+To enable this plugin add the pointer to the scenario input file `scenarios/arcaflow/io-hog/input.yaml` as described in the 
+Usage section.
+This scenario takes a list of objects named `input_list` with the following properties:
+
+- **kubeconfig :** *string* the kubeconfig needed by the deployer to deploy the sysbench plugin in the target cluster
+- **namespace :** *string* the namespace where the scenario container will be deployed
+**Note:** this parameter will be automatically filled by kraken if the `kubeconfig_path` property is correctly set
+- **node_selector :** *key-value map* the node label that will be used as `nodeSelector` by the pod to target a specific cluster node
+- **duration :** *string* stop  stress  test  after  N  seconds.  One  can  also specify the units of time in seconds, minutes, hours, days or years with the suffix s, m, h, d or y.
+- **target_pod_folder :** *string* the path in the pod where the volume is mounted
+- **target_pod_volume :** *object* the `hostPath` volume definition in the [Kubernetes/OpenShift](https://docs.openshift.com/container-platform/3.11/install_config/persistent_storage/using_hostpath.html) format, that will be attached to the pod as a volume
+- **io_write_bytes :** *string* writes N bytes for each hdd process. The size can be expressed as % of free space on the file system or in units of Bytes, KBytes, MBytes and GBytes using the suffix b, k, m or g
+- **io_block_size :** *string* size of each write in bytes. Size can be from 1 byte to 4m.
+
+To perform several load tests in the same run simultaneously (eg. stress two or more nodes in the same run) add another item
+to the `input_list` with the same properties (and eventually different values eg. different node_selectors 
+to schedule the pod on different nodes). To reduce (or increase) the parallelism change the value `parallelism` in `workload.yaml` file 
--- a/docs/container_scenarios.md
+++ b/docs/container_scenarios.md
@@ -8,14 +8,14 @@ The following are the components of Kubernetes/OpenShift for which a basic chaos

 ```
 scenarios:
- name: "<Name of scenario>"
+- name: "<name of scenario>"
  namespace: "<specific namespace>" # can specify "*" if you want to find in all namespaces
  label_selector: "<label of pod(s)>"
  container_name: "<specific container name>"  # This is optional, can take out and will kill all containers in all pods found under namespace and label
  pod_names:  # This is optional, can take out and will select all pods with given namespace and label
  - <pod_name>
  count: <number of containers to disrupt, default=1>
-  action: <Action to run. For example kill 1 ( hang up ) or kill 9. Default is set to kill 1>
+  action: <kill signal to run. For example 1 ( hang up ) or 9. Default is set to 1>
  expected_recovery_time: <number of seconds to wait for container to be running again> (defaults to 120seconds)
 ```

--- a/docs/installation.md
+++ b/docs/installation.md
@@ -3,14 +3,19 @@
 The following ways are supported to run Kraken:

 - Standalone python program through Git.
- Containerized version using either Podman or Docker as the runtime.
- Kubernetes or OpenShift deployment.
+- Containerized version using either Podman or Docker as the runtime via [Krkn-hub](https://github.com/redhat-chaos/krkn-hub)
+- Kubernetes or OpenShift deployment ( unsupported )

 **NOTE**: It is recommended to run Kraken external to the cluster ( Standalone or Containerized ) hitting the Kubernetes/OpenShift API as running it internal to the cluster might be disruptive to itself and also might not report back the results if the chaos leads to cluster's API server instability.

 **NOTE**: To run Kraken on Power (ppc64le) architecture, build and run a containerized version by following the
 instructions given [here](https://github.com/redhat-chaos/krkn/blob/main/containers/build_own_image-README.md).

+**NOTE**: Helper functions for interactions in Krkn are part of [krkn-lib](https://github.com/redhat-chaos/krkn-lib). 
+Please feel free to reuse and expand them as you see fit when adding a new scenario or expanding 
+the capabilities of the current supported scenarios.
+
+
 ### Git

 #### Clone the repository
@@ -35,26 +40,12 @@ $ python3.9 run_kraken.py --config <config_file_location>
 ```

 ### Run containerized version
-Assuming that the latest docker ( 17.05 or greater with multi-build support ) is installed on the host, run:
-```
-$ docker pull quay.io/redhat-chaos/krkn:latest
-$ docker run --name=kraken --net=host -v <path_to_kubeconfig>:/root/.kube/config:Z -v <path_to_kraken_config>:/root/kraken/config/config.yaml:Z -d quay.io/redhat-chaos/krkn:latest
-$ docker run --name=kraken --net=host -v <path_to_kubeconfig>:/root/.kube/config:Z -v <path_to_kraken_config>:/root/kraken/config/config.yaml:Z -v <path_to_scenarios_directory>:/root/kraken/scenarios:Z -d quay.io/redhat-chaos/krkn:latest #custom or tweaked scenario configs
-$ docker logs -f kraken
-```
+[Krkn-hub](https://github.com/redhat-chaos/krkn-hub) is a wrapper that allows running Krkn chaos scenarios via podman or docker runtime with scenario parameters/configuration defined as environment variables.

-Similarly, podman can be used to achieve the same:
-```
-$ podman pull quay.io/redhat-chaos/krkn
-$ podman run --name=kraken --net=host -v <path_to_kubeconfig>:/root/.kube/config:Z -v <path_to_kraken_config>:/root/kraken/config/config.yaml:Z -d quay.io/redhat-chaos/krkn:latest
-$ podman run --name=kraken --net=host -v <path_to_kubeconfig>:/root/.kube/config:Z -v <path_to_kraken_config>:/root/kraken/config/config.yaml:Z -v <path_to_scenarios_directory>:/root/kraken/scenarios:Z -d quay.io/redhat-chaos/krkn:latest #custom or tweaked scenario configs
-$ podman logs -f kraken
-```
-
-If you want to build your own kraken image see [here](https://github.com/redhat-chaos/krkn/blob/main/containers/build_own_image-README.md)
+Refer [instructions](https://github.com/redhat-chaos/krkn-hub#supported-chaos-scenarios) to get started.


-### Run Kraken as a Kubernetes deployment
+### Run Kraken as a Kubernetes deployment ( unsupported option - standalone or containerized deployers are recommended )
 Refer [Instructions](https://github.com/redhat-chaos/krkn/blob/main/containers/README.md) on how to deploy and run Kraken as a Kubernetes/OpenShift deployment.


--- a/docs/service_disruption_scenarios.md
+++ b/docs/service_disruption_scenarios.md
@@ -1,6 +1,6 @@
-###  Delete Namespace Scenarios
+###  Service Disruption Scenarios (Previously Delete Namespace Scenario)

-Using this type of scenario configuration one is able to delete a specific namespace, or a namespace matching a certain regex string.
+Using this type of scenario configuration one is able to delete crucial objects in a specific namespace, or a namespace matching a certain regex string.

 Configuration Options:

@@ -27,12 +27,20 @@ scenarios:
  sleep: 15
 ```

-**NOTE:** Many openshift namespaces have finalizers built that protect the namespace from being fully deleted: see documentation [here](https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/).
-The namespaces that do have finalizers enabled will be in left in a terminating state but all the pods running on that namespace will get deleted.
+
+### Steps
+
+This scenario will select a namespace (or multiple) dependent on the configuration and will kill all of the below object types in that namespace and will wait for them to be Running in the post action 
+1. Services 
+2. Daemonsets
+3. Statefulsets
+4. Replicasets
+5. Deployments 
+

 #### Post Action

-In all scenarios we do a post chaos check to wait and verify the specific component.
+We do a post chaos check to wait and verify the specific objects in each namespace are Ready

 Here there are two options:

@@ -47,8 +55,8 @@ See [scenarios/post_action_namespace.py](https://github.com/cloud-bulldozer/krak
 ```


-2. Allow kraken to wait and check the killed namespaces become 'Active' again. Kraken keeps a list of the specific
-namespaces that were killed to verify all that were affected recover properly.
+1. Allow kraken to wait and check all killed objects in the namespaces become 'Running' again. Kraken keeps a list of the specific
+objects in namespaces that were killed to verify all that were affected recover properly.

 ```
 wait_time: <seconds to wait for namespace to recover>
--- a/kraken/application_outage/actions.py
+++ b/kraken/application_outage/actions.py
@@ -4,11 +4,14 @@ import time
 import kraken.cerberus.setup as cerberus
 from jinja2 import Template
 import kraken.invoke.command as runcommand
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import get_yaml_item_value, log_exception
+

 # Reads the scenario config, applies and deletes a network policy to
 # block the traffic for the specified duration
-def run(scenarios_list, config, wait_duration, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+def run(scenarios_list, config, wait_duration, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    failed_post_scenarios = ""
    scenario_telemetries: list[ScenarioTelemetry] = []
    failed_scenarios = []
@@ -22,10 +25,18 @@ def run(scenarios_list, config, wait_duration, telemetry: KrknTelemetry) -> (lis
                with open(app_outage_config, "r") as f:
                    app_outage_config_yaml = yaml.full_load(f)
                    scenario_config = app_outage_config_yaml["application_outage"]
-                    pod_selector = scenario_config.get("pod_selector", "{}")
-                    traffic_type = scenario_config.get("block", "[Ingress, Egress]")
-                    namespace = scenario_config.get("namespace", "")
-                    duration = scenario_config.get("duration", 60)
+                    pod_selector = get_yaml_item_value(
+                        scenario_config, "pod_selector", "{}"
+                    )
+                    traffic_type = get_yaml_item_value(
+                        scenario_config, "block", "[Ingress, Egress]"
+                    )
+                    namespace = get_yaml_item_value(
+                        scenario_config, "namespace", ""
+                    )
+                    duration = get_yaml_item_value(
+                        scenario_config, "duration", 60
+                    )

                    start_time = int(time.time())

@@ -66,7 +77,7 @@ spec:
            except Exception as e :
                scenario_telemetry.exitStatus = 1
                failed_scenarios.append(app_outage_config)
-                telemetry.log_exception(app_outage_config)
+                log_exception(app_outage_config)
            else:
                scenario_telemetry.exitStatus = 0
            scenario_telemetry.endTimeStamp = time.time()
--- a/kraken/arcaflow_plugin/arcaflow_plugin.py
+++ b/kraken/arcaflow_plugin/arcaflow_plugin.py
@@ -1,17 +1,16 @@
 import time
-
 import arcaflow
 import os
 import yaml
 import logging
-import sys
 from pathlib import Path
 from typing import List
 from .context_auth import ContextAuth
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry


-def run(scenarios_list: List[str], kubeconfig_path: str, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+def run(scenarios_list: List[str], kubeconfig_path: str, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    scenario_telemetries: list[ScenarioTelemetry] = []
    failed_post_scenarios = []
    for scenario in scenarios_list:
--- a/kraken/chaos_recommender/init.py
+++ b/kraken/chaos_recommender/init.py
@@ -0,0 +1,3 @@
+from .analysis import *
+from .kraken_tests import *
+from .prometheus import *
--- a/kraken/chaos_recommender/analysis.py
+++ b/kraken/chaos_recommender/analysis.py
@@ -0,0 +1,90 @@
+import logging
+
+import pandas as pd
+import kraken.chaos_recommender.kraken_tests as kraken_tests
+import time
+
+threshold = .7  # Adjust the threshold as needed
+heatmap_cpu_threshold = .5
+heatmap_mem_threshold = .5
+
+KRAKEN_TESTS_PATH = "./kraken_chaos_tests.txt"
+
+#Placeholder, this should be done with topology
+def return_critical_services():
+    return ["web", "cart"]
+
+
+def load_telemetry_data(file_path):
+    data = pd.read_csv(file_path, delimiter=r"\s+")
+    return data
+
+def calculate_zscores(data):
+    zscores = pd.DataFrame()
+    zscores["Service"] = data["service"]
+    zscores["CPU"] = (data["CPU"] - data["CPU"].mean()) / data["CPU"].std()
+    zscores["Memory"] = (data["MEM"] - data["MEM"].mean()) / data["MEM"].std()
+    zscores["Network"] = (data["NETWORK"] - data["NETWORK"].mean()) / data["NETWORK"].std()
+    return zscores
+
+def identify_outliers(data):
+    outliers_cpu = data[data["CPU"] > threshold]["Service"].tolist()
+    outliers_memory = data[data["Memory"] > threshold]["Service"].tolist()
+    outliers_network = data[data["Network"] > threshold]["Service"].tolist()
+
+    return outliers_cpu, outliers_memory, outliers_network
+
+
+def get_services_above_heatmap_threshold(dataframe, cpu_threshold, mem_threshold):
+    # Filter the DataFrame based on CPU_HEATMAP and MEM_HEATMAP thresholds
+    filtered_df = dataframe[((dataframe['CPU']/dataframe['CPU_LIMITS']) > cpu_threshold)]
+    # Get the lists of services
+    cpu_services = filtered_df['service'].tolist()
+
+    filtered_df = dataframe[((dataframe['MEM']/dataframe['MEM_LIMITS']) > mem_threshold)]
+    mem_services = filtered_df['service'].tolist()
+
+    return cpu_services, mem_services
+
+
+def analysis(file_path, chaos_tests_config):
+    # Load the telemetry data from file
+    data = load_telemetry_data(file_path)
+
+    # Calculate Z-scores for CPU, Memory, and Network columns
+    zscores = calculate_zscores(data)
+
+    # Identify outliers
+    outliers_cpu, outliers_memory, outliers_network = identify_outliers(zscores)
+    cpu_services, mem_services = get_services_above_heatmap_threshold(data, heatmap_cpu_threshold, heatmap_mem_threshold)
+
+    # Display the identified outliers
+    logging.info("======================== Profiling ==================================")
+    logging.info(f"CPU outliers: {outliers_cpu}")
+    logging.info(f"Memory outliers: {outliers_memory}")
+    logging.info(f"Network outliers: {outliers_network}")
+    logging.info("===================== HeatMap Analysis ==============================")
+
+    if cpu_services:
+        logging.info("Services with CPU_HEATMAP above threshold:", cpu_services)
+    else:
+        logging.info("There are no services that are using siginificant CPU compared to their assigned limits (infinite in case no limits are set).")
+    if mem_services:
+        logging.info("Services with MEM_HEATMAP above threshold:", mem_services)
+    else:
+        logging.info("There are no services that are using siginificant MEMORY compared to their assigned limits (infinite in case no limits are set).")
+    time.sleep(2)
+    logging.info("======================= Recommendations =============================")
+    if cpu_services:
+        logging.info(f"Recommended tests for {str(cpu_services)}  :\n {chaos_tests_config['CPU']}")
+        logging.info("\n")
+    if mem_services:
+        logging.info(f"Recommended tests for {str(mem_services)}  :\n {chaos_tests_config['MEM']}")
+        logging.info("\n")
+
+    if outliers_network:
+        logging.info(f"Recommended tests for  str(outliers_network)  :\n {chaos_tests_config['NETWORK']}")
+        logging.info("\n")
+
+    logging.info("\n")
+    logging.info("Please check data in utilisation.txt for further analysis")
--- a/kraken/chaos_recommender/kraken_tests.py
+++ b/kraken/chaos_recommender/kraken_tests.py
@@ -0,0 +1,30 @@
+def get_entries_by_category(filename, category):
+    # Read the file
+    with open(filename, 'r') as file:
+        content = file.read()
+
+    # Split the content into sections based on the square brackets
+    sections = content.split('\n\n')
+
+    # Define the categories
+    valid_categories = ['CPU', 'NETWORK', 'MEM', 'GENERIC']
+
+    # Validate the provided category
+    if category not in valid_categories:
+        return []
+
+    # Find the section corresponding to the specified category
+    target_section = None
+    for section in sections:
+        if section.startswith(f"[{category}]"):
+            target_section = section
+            break
+
+    # If the category section was not found, return an empty list
+    if target_section is None:
+        return []
+
+    # Extract the entries from the category section
+    entries = [entry.strip() for entry in target_section.split('\n') if entry and not entry.startswith('[')]
+
+    return entries
--- a/kraken/chaos_recommender/prometheus.py
+++ b/kraken/chaos_recommender/prometheus.py
@@ -0,0 +1,96 @@
+import logging
+
+import pandas
+from prometheus_api_client import PrometheusConnect
+import pandas as pd
+import urllib3
+
+
+saved_metrics_path = "./utilisation.txt"
+
+def convert_data_to_dataframe(data, label):
+    df = pd.DataFrame()
+    df['service'] = [item['metric']['pod'] for item in data]
+    df[label] = [item['value'][1] for item in data]
+
+    return df
+
+
+def convert_data(data, service):
+
+    result = {}
+    for entry in data:
+        pod_name = entry['metric']['pod']
+        value = entry['value'][1]
+        result[pod_name] = value
+    return result.get(service, '100000000000') # for those pods whose limits are not defined they can take as much resources, there assigning a very high value
+
+def save_utilization_to_file(cpu_data, cpu_limits_result, mem_data, mem_limits_result, network_data, filename):
+    df_cpu = convert_data_to_dataframe(cpu_data, "CPU")
+    merged_df = pd.DataFrame(columns=['service','CPU','CPU_LIMITS','MEM','MEM_LIMITS','NETWORK'])
+    services = df_cpu.service.unique()
+    logging.info(services)
+
+    for s in services:
+
+        new_row_df = pd.DataFrame( {"service": s, "CPU" : convert_data(cpu_data, s),
+                    "CPU_LIMITS" : convert_data(cpu_limits_result, s),
+                    "MEM" : convert_data(mem_data, s), "MEM_LIMITS" : convert_data(mem_limits_result, s),
+                    "NETWORK" : convert_data(network_data, s)}, index=[0])
+        merged_df = pd.concat([merged_df, new_row_df], ignore_index=True)
+
+
+
+    # Convert columns to string
+    merged_df['CPU'] = merged_df['CPU'].astype(str)
+    merged_df['MEM'] = merged_df['MEM'].astype(str)
+    merged_df['CPU_LIMITS'] = merged_df['CPU_LIMITS'].astype(str)
+    merged_df['MEM_LIMITS'] = merged_df['MEM_LIMITS'].astype(str)
+    merged_df['NETWORK'] = merged_df['NETWORK'].astype(str)
+
+    # Extract integer part before the decimal point
+    merged_df['CPU'] = merged_df['CPU'].str.split('.').str[0]
+    merged_df['MEM'] = merged_df['MEM'].str.split('.').str[0]
+    merged_df['CPU_LIMITS'] = merged_df['CPU_LIMITS'].str.split('.').str[0]
+    merged_df['MEM_LIMITS'] = merged_df['MEM_LIMITS'].str.split('.').str[0]
+    merged_df['NETWORK'] = merged_df['NETWORK'].str.split('.').str[0]
+
+    merged_df.to_csv(filename, sep='\t', index=False)
+
+def fetch_utilization_from_prometheus(prometheus_endpoint, auth_token, namespace, scrape_duration):
+    urllib3.disable_warnings()
+    prometheus = PrometheusConnect(url=prometheus_endpoint, headers={'Authorization':'Bearer {}'.format(auth_token)}, disable_ssl=True)
+
+    # Fetch CPU utilization
+    cpu_query = 'sum (rate (container_cpu_usage_seconds_total{image!="", namespace="%s"}[%s])) by (pod) *1000' % (namespace,scrape_duration)
+    logging.info(cpu_query)
+    cpu_result = prometheus.custom_query(cpu_query)
+    cpu_data = cpu_result
+
+
+    cpu_limits_query = '(sum by (pod) (kube_pod_container_resource_limits{resource="cpu", namespace="%s"}))*1000' %(namespace)
+    logging.info(cpu_limits_query)
+    cpu_limits_result = prometheus.custom_query(cpu_limits_query)
+
+
+    mem_query = 'sum by (pod) (avg_over_time(container_memory_usage_bytes{image!="", namespace="%s"}[%s]))' % (namespace, scrape_duration)
+    logging.info(mem_query)
+    mem_result = prometheus.custom_query(mem_query)
+    mem_data = mem_result
+
+    mem_limits_query = 'sum by (pod) (kube_pod_container_resource_limits{resource="memory", namespace="%s"})  ' %(namespace)
+    logging.info(mem_limits_query)
+    mem_limits_result = prometheus.custom_query(mem_limits_query)
+
+
+    network_query = 'sum by (pod) ((avg_over_time(container_network_transmit_bytes_total{namespace="%s"}[%s])) + \
+    (avg_over_time(container_network_receive_bytes_total{namespace="%s"}[%s])))' % (namespace, scrape_duration, namespace, scrape_duration)
+    network_result = prometheus.custom_query(network_query)
+    logging.info(network_query)
+    network_data = network_result
+
+
+    save_utilization_to_file(cpu_data, cpu_limits_result, mem_data, mem_limits_result, network_data, saved_metrics_path)
+    return saved_metrics_path
+
+
--- a/kraken/litmus/common_litmus.py
+++ b/kraken/litmus/common_litmus.py
@@ -1,14 +1,13 @@
 import kraken.invoke.command as runcommand
-import krkn_lib_kubernetes
 import logging
 import time
 import sys
 import requests
 import yaml
 import kraken.cerberus.setup as cerberus
+from krkn_lib.k8s import KrknKubernetes

-
-# krkn_lib_kubernetes
+# krkn_lib
 # Inject litmus scenarios defined in the config
 def run(
        scenarios_list,
@@ -16,7 +15,7 @@ def run(
        litmus_uninstall,
        wait_duration,
        litmus_namespace,
-        kubecli: krkn_lib_kubernetes.KrknLibKubernetes
+        kubecli: KrknKubernetes
 ):
    # Loop to run the scenarios starts here
    for l_scenario in scenarios_list:
@@ -94,8 +93,8 @@ def deploy_all_experiments(version_string, namespace):
    )


-# krkn_lib_kubernetes
-def wait_for_initialized(engine_name, experiment_name, namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def wait_for_initialized(engine_name, experiment_name, namespace, kubecli: KrknKubernetes):

    chaos_engine = kubecli.get_litmus_chaos_object(kind='chaosengine', name=engine_name,
                                                   namespace=namespace).engineStatus
@@ -119,13 +118,13 @@ def wait_for_initialized(engine_name, experiment_name, namespace, kubecli: krkn_
    return True


-# krkn_lib_kubernetes
+# krkn_lib
 def wait_for_status(
        engine_name,
        expected_status,
        experiment_name,
        namespace,
-        kubecli: krkn_lib_kubernetes.KrknLibKubernetes
+        kubecli: KrknKubernetes
 ):

    if expected_status == "running":
@@ -156,8 +155,8 @@ def wait_for_status(


 # Check status of experiment
-# krkn_lib_kubernetes
-def check_experiment(engine_name, experiment_name, namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def check_experiment(engine_name, experiment_name, namespace, kubecli: KrknKubernetes):

    wait_response = wait_for_status(engine_name, "running", experiment_name, namespace, kubecli)

@@ -183,8 +182,8 @@ def check_experiment(engine_name, experiment_name, namespace, kubecli: krkn_lib_


 # Delete all chaos engines in a given namespace
-# krkn_lib_kubernetes
-def delete_chaos_experiments(namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def delete_chaos_experiments(namespace, kubecli: KrknKubernetes):

    if kubecli.check_if_namespace_exists(namespace):
        chaos_exp_exists = runcommand.invoke_no_exit("kubectl get chaosexperiment")
@@ -194,8 +193,8 @@ def delete_chaos_experiments(namespace, kubecli: krkn_lib_kubernetes.KrknLibKube


 # Delete all chaos engines in a given namespace
-# krkn_lib_kubernetes
-def delete_chaos(namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def delete_chaos(namespace, kubecli:KrknKubernetes):

    if kubecli.check_if_namespace_exists(namespace):
        logging.info("Deleting all litmus run objects")
@@ -209,8 +208,8 @@ def delete_chaos(namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
        logging.info(namespace + " namespace doesn't exist")


-# krkn_lib_kubernetes
-def uninstall_litmus(version, litmus_namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def uninstall_litmus(version, litmus_namespace, kubecli: KrknKubernetes):

    if kubecli.check_if_namespace_exists(litmus_namespace):
        logging.info("Uninstalling Litmus operator")
--- a/kraken/managedcluster_scenarios/common_managedcluster_functions.py
+++ b/kraken/managedcluster_scenarios/common_managedcluster_functions.py
@@ -1,14 +1,14 @@
 import random
 import logging
-import krkn_lib_kubernetes
+from krkn_lib.k8s import KrknKubernetes

-# krkn_lib_kubernetes
+# krkn_lib
 # Pick a random managedcluster with specified label selector
 def get_managedcluster(
        managedcluster_name,
        label_selector,
        instance_kill_count,
-        kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+        kubecli: KrknKubernetes):

    if managedcluster_name in kubecli.list_killable_managedclusters():
        return [managedcluster_name]
@@ -30,12 +30,12 @@ def get_managedcluster(


 # Wait until the managedcluster status becomes Available
-# krkn_lib_kubernetes
-def wait_for_available_status(managedcluster, timeout, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def wait_for_available_status(managedcluster, timeout, kubecli: KrknKubernetes):
    kubecli.watch_managedcluster_status(managedcluster, "True", timeout)


 # Wait until the managedcluster status becomes Not Available
-# krkn_lib_kubernetes
-def wait_for_unavailable_status(managedcluster, timeout, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def wait_for_unavailable_status(managedcluster, timeout, kubecli: KrknKubernetes):
    kubecli.watch_managedcluster_status(managedcluster, "Unknown", timeout)
--- a/kraken/managedcluster_scenarios/managedcluster_scenarios.py
+++ b/kraken/managedcluster_scenarios/managedcluster_scenarios.py
@@ -4,19 +4,17 @@ import time
 import logging
 import sys
 import yaml
-import html
-import krkn_lib_kubernetes
 import kraken.managedcluster_scenarios.common_managedcluster_functions as common_managedcluster_functions
-
+from krkn_lib.k8s import KrknKubernetes

 class GENERAL:
    def __init__(self):
        pass

-# krkn_lib_kubernetes
+# krkn_lib
 class managedcluster_scenarios():
-    kubecli: krkn_lib_kubernetes.KrknLibKubernetes
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    kubecli: KrknKubernetes
+    def __init__(self, kubecli: KrknKubernetes):
        self.kubecli = kubecli
        self.general = GENERAL()

--- a/kraken/managedcluster_scenarios/run.py
+++ b/kraken/managedcluster_scenarios/run.py
@@ -1,20 +1,20 @@
 import yaml
 import logging
 import time
-import krkn_lib_kubernetes
 from kraken.managedcluster_scenarios.managedcluster_scenarios import managedcluster_scenarios
 import kraken.managedcluster_scenarios.common_managedcluster_functions as common_managedcluster_functions
 import kraken.cerberus.setup as cerberus
-
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.utils.functions import get_yaml_item_value

 # Get the managedcluster scenarios object of specfied cloud type
-# krkn_lib_kubernetes
-def get_managedcluster_scenario_object(managedcluster_scenario, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def get_managedcluster_scenario_object(managedcluster_scenario, kubecli: KrknKubernetes):
    return managedcluster_scenarios(kubecli)

 # Run defined scenarios
-# krkn_lib_kubernetes
-def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def run(scenarios_list, config, wait_duration, kubecli: KrknKubernetes):
    for managedcluster_scenario_config in scenarios_list:
        with open(managedcluster_scenario_config, "r") as f:
            managedcluster_scenario_config = yaml.full_load(f)
@@ -32,14 +32,22 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn


 # Inject the specified managedcluster scenario
-# krkn_lib_kubernetes
-def inject_managedcluster_scenario(action, managedcluster_scenario, managedcluster_scenario_object, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def inject_managedcluster_scenario(action, managedcluster_scenario, managedcluster_scenario_object, kubecli: KrknKubernetes):
    # Get the managedcluster scenario configurations
-    run_kill_count = managedcluster_scenario.get("runs", 1)
-    instance_kill_count = managedcluster_scenario.get("instance_count", 1)
-    managedcluster_name = managedcluster_scenario.get("managedcluster_name", "")
-    label_selector = managedcluster_scenario.get("label_selector", "")
-    timeout = managedcluster_scenario.get("timeout", 120)
+    run_kill_count = get_yaml_item_value(
+        managedcluster_scenario, "runs", 1
+    )
+    instance_kill_count = get_yaml_item_value(
+        managedcluster_scenario, "instance_count", 1
+    )
+    managedcluster_name = get_yaml_item_value(
+        managedcluster_scenario, "managedcluster_name", ""
+    )
+    label_selector = get_yaml_item_value(
+        managedcluster_scenario, "label_selector", ""
+    )
+    timeout = get_yaml_item_value(managedcluster_scenario, "timeout", 120)
    # Get the managedcluster to apply the scenario
    if managedcluster_name:
        managedcluster_name_list = managedcluster_name.split(",")
--- a/kraken/namespace_actions/common_namespace_functions.py
+++ b/kraken/namespace_actions/common_namespace_functions.py
@@ -1,131 +0,0 @@
-import time
-import random
-import logging
-import krkn_lib_kubernetes
-import kraken.cerberus.setup as cerberus
-import kraken.post_actions.actions as post_actions
-import yaml
-import sys
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
-
-# krkn_lib_kubernetes
-def run(
-        scenarios_list,
-        config,
-        wait_duration,
-        failed_post_scenarios,
-        kubeconfig_path,
-        kubecli: krkn_lib_kubernetes.KrknLibKubernetes,
-        telemetry: KrknTelemetry
-) -> (list[str], list[ScenarioTelemetry]):
-    scenario_telemetries: list[ScenarioTelemetry] = []
-    failed_scenarios = []
-    for scenario_config in scenarios_list:
-        scenario_telemetry = ScenarioTelemetry()
-        scenario_telemetry.scenario = scenario_config[0]
-        scenario_telemetry.startTimeStamp = time.time()
-        telemetry.set_parameters_base64(scenario_telemetry, scenario_config[0])
-        try:
-            if len(scenario_config) > 1:
-                pre_action_output = post_actions.run(kubeconfig_path, scenario_config[1])
-            else:
-                pre_action_output = ""
-            with open(scenario_config[0], "r") as f:
-                scenario_config_yaml = yaml.full_load(f)
-                for scenario in scenario_config_yaml["scenarios"]:
-                    scenario_namespace = scenario.get("namespace", "")
-                    scenario_label = scenario.get("label_selector", "")
-                    if scenario_namespace is not None and scenario_namespace.strip() != "":
-                        if scenario_label is not None and scenario_label.strip() != "":
-                            logging.error("You can only have namespace or label set in your namespace scenario")
-                            logging.error(
-                                "Current scenario config has namespace '%s' and label selector '%s'"
-                                % (scenario_namespace, scenario_label)
-                            )
-                            logging.error(
-                                "Please set either namespace to blank ('') or label_selector to blank ('') to continue"
-                            )
-                            # removed_exit
-                            # sys.exit(1)
-                            raise RuntimeError()
-                    delete_count = scenario.get("delete_count", 1)
-                    run_count = scenario.get("runs", 1)
-                    run_sleep = scenario.get("sleep", 10)
-                    wait_time = scenario.get("wait_time", 30)
-                    killed_namespaces = []
-                    start_time = int(time.time())
-                    for i in range(run_count):
-                        namespaces = kubecli.check_namespaces([scenario_namespace], scenario_label)
-                        for j in range(delete_count):
-                            if len(namespaces) == 0:
-                                logging.error(
-                                    "Couldn't delete %s namespaces, not enough namespaces matching %s with label %s"
-                                    % (str(run_count), scenario_namespace, str(scenario_label))
-                                )
-                                # removed_exit
-                                # sys.exit(1)
-                                raise RuntimeError()
-                            selected_namespace = namespaces[random.randint(0, len(namespaces) - 1)]
-                            killed_namespaces.append(selected_namespace)
-                            try:
-                                kubecli.delete_namespace(selected_namespace)
-                                logging.info("Delete on namespace %s was successful" % str(selected_namespace))
-                            except Exception as e:
-                                logging.info("Delete on namespace %s was unsuccessful" % str(selected_namespace))
-                                logging.info("Namespace action error: " + str(e))
-                                # removed_exit
-                                # sys.exit(1)
-                                raise RuntimeError()
-                            namespaces.remove(selected_namespace)
-                            logging.info("Waiting %s seconds between namespace deletions" % str(run_sleep))
-                            time.sleep(run_sleep)
-
-                            logging.info("Waiting for the specified duration: %s" % wait_duration)
-                            time.sleep(wait_duration)
-                            if len(scenario_config) > 1:
-                                try:
-                                    failed_post_scenarios = post_actions.check_recovery(
-                                        kubeconfig_path, scenario_config, failed_post_scenarios, pre_action_output
-                                    )
-                                except Exception as e:
-                                    logging.error("Failed to run post action checks: %s" % e)
-                                    # removed_exit
-                                    # sys.exit(1)
-                                    raise RuntimeError()
-                            else:
-                                failed_post_scenarios = check_active_namespace(killed_namespaces, wait_time, kubecli)
-                    end_time = int(time.time())
-                    cerberus.publish_kraken_status(config, failed_post_scenarios, start_time, end_time)
-        except (Exception, RuntimeError):
-            scenario_telemetry.exitStatus = 1
-            failed_scenarios.append(scenario_config[0])
-            telemetry.log_exception(scenario_config[0])
-        else:
-            scenario_telemetry.exitStatus = 0
-        scenario_telemetry.endTimeStamp = time.time()
-        scenario_telemetries.append(scenario_telemetry)
-    return failed_scenarios, scenario_telemetries
-
-# krkn_lib_kubernetes
-def check_active_namespace(killed_namespaces, wait_time, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
-    active_namespace = []
-    timer = 0
-    while timer < wait_time and killed_namespaces:
-        for namespace_name in killed_namespaces:
-            if namespace_name in kubecli.list_namespaces():
-                response = kubecli.get_namespace_status(namespace_name).strip()
-                if response != "Active":
-                    continue
-                else:
-                    active_namespace.append(namespace_name)
-        killed_namespaces = set(killed_namespaces) - set(active_namespace)
-        if len(killed_namespaces) == 0:
-            return []
-
-        timer += 5
-        time.sleep(5)
-        logging.info("Waiting 5 seconds for namespaces to become active")
-
-    logging.error("Namespaces are still not active after waiting " + str(wait_time) + "seconds")
-    logging.error("Non active namespaces " + str(killed_namespaces))
-    return killed_namespaces
--- a/kraken/network_chaos/actions.py
+++ b/kraken/network_chaos/actions.py
@@ -1,18 +1,20 @@
 import yaml
 import logging
 import time
-import sys
 import os
 import random
-import krkn_lib_kubernetes
-from jinja2 import Environment, FileSystemLoader
 import kraken.cerberus.setup as cerberus
 import kraken.node_actions.common_node_functions as common_node_functions
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from jinja2 import Environment, FileSystemLoader
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import get_yaml_item_value, log_exception

-# krkn_lib_kubernetes
+
+# krkn_lib
 # Reads the scenario config and introduces traffic variations in Node's host network interface.
-def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+def run(scenarios_list, config, wait_duration, kubecli: KrknKubernetes, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    failed_post_scenarios = ""
    logging.info("Runing the Network Chaos tests")
    failed_post_scenarios = ""
@@ -28,13 +30,26 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn
                param_lst = ["latency", "loss", "bandwidth"]
                test_config = yaml.safe_load(file)
                test_dict = test_config["network_chaos"]
-                test_duration = int(test_dict.get("duration", 300))
-                test_interface = test_dict.get("interfaces", [])
-                test_node = test_dict.get("node_name", "")
-                test_node_label = test_dict.get("label_selector", "node-role.kubernetes.io/master")
-                test_execution = test_dict.get("execution", "serial")
-                test_instance_count = test_dict.get("instance_count", 1)
-                test_egress = test_dict.get("egress", {"bandwidth": "100mbit"})
+                test_duration = int(
+                    get_yaml_item_value(test_dict, "duration", 300)
+                )
+                test_interface = get_yaml_item_value(
+                    test_dict, "interfaces", []
+                )
+                test_node = get_yaml_item_value(test_dict, "node_name", "")
+                test_node_label = get_yaml_item_value(
+                    test_dict, "label_selector",
+                    "node-role.kubernetes.io/master"
+                )
+                test_execution = get_yaml_item_value(
+                    test_dict, "execution", "serial"
+                )
+                test_instance_count = get_yaml_item_value(
+                    test_dict, "instance_count", 1
+                )
+                test_egress = get_yaml_item_value(
+                    test_dict, "egress", {"bandwidth": "100mbit"}
+                )
                if test_node:
                    node_name_list = test_node.split(",")
                else:
@@ -101,15 +116,15 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn
        except (RuntimeError, Exception):
            scenario_telemetry.exitStatus = 1
            failed_scenarios.append(net_config)
-            telemetry.log_exception(net_config)
+            log_exception(net_config)
        else:
            scenario_telemetry.exitStatus = 0
        scenario_telemetries.append(scenario_telemetry)
    return failed_scenarios, scenario_telemetries


-# krkn_lib_kubernetes
-def verify_interface(test_interface, nodelst, template, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def verify_interface(test_interface, nodelst, template, kubecli: KrknKubernetes):
    pod_index = random.randint(0, len(nodelst) - 1)
    pod_body = yaml.safe_load(template.render(nodename=nodelst[pod_index]))
    logging.info("Creating pod to query interface on node %s" % nodelst[pod_index])
@@ -134,16 +149,16 @@ def verify_interface(test_interface, nodelst, template, kubecli: krkn_lib_kubern
        kubecli.delete_pod("fedtools", "default")


-# krkn_lib_kubernetes
-def get_job_pods(api_response, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def get_job_pods(api_response, kubecli: KrknKubernetes):
    controllerUid = api_response.metadata.labels["controller-uid"]
    pod_label_selector = "controller-uid=" + controllerUid
    pods_list = kubecli.list_pods(label_selector=pod_label_selector, namespace="default")
    return pods_list[0]


-# krkn_lib_kubernetes
-def wait_for_job(joblst, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, timeout=300):
+# krkn_lib
+def wait_for_job(joblst, kubecli: KrknKubernetes, timeout=300):
    waittime = time.time() + timeout
    count = 0
    joblen = len(joblst)
@@ -161,8 +176,8 @@ def wait_for_job(joblst, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, timeout
            time.sleep(5)


-# krkn_lib_kubernetes
-def delete_job(joblst, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def delete_job(joblst, kubecli: KrknKubernetes):
    for jobname in joblst:
        try:
            api_response = kubecli.get_job_status(jobname, namespace="default")
--- a/kraken/node_actions/abstract_node_scenarios.py
+++ b/kraken/node_actions/abstract_node_scenarios.py
@@ -2,12 +2,12 @@ import sys
 import logging
 import kraken.invoke.command as runcommand
 import kraken.node_actions.common_node_functions as nodeaction
-import  krkn_lib_kubernetes
+from krkn_lib.k8s import KrknKubernetes

-# krkn_lib_kubernetes
+# krkn_lib
 class abstract_node_scenarios:
-    kubecli: krkn_lib_kubernetes.KrknLibKubernetes
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    kubecli: KrknKubernetes
+    def __init__(self, kubecli: KrknKubernetes):
        self.kubecli = kubecli
    # Node scenario to start the node
    def node_start_scenario(self, instance_kill_count, node, timeout):
--- a/kraken/node_actions/alibaba_node_scenarios.py
+++ b/kraken/node_actions/alibaba_node_scenarios.py
@@ -1,14 +1,14 @@
 import sys
 import time
-import krkn_lib_kubernetes
+import logging
+import kraken.node_actions.common_node_functions as nodeaction
+import os
+import json
 from aliyunsdkcore.client import AcsClient
 from aliyunsdkecs.request.v20140526 import DescribeInstancesRequest, DeleteInstanceRequest
 from aliyunsdkecs.request.v20140526 import StopInstanceRequest, StartInstanceRequest, RebootInstanceRequest
-import logging
-import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-import os
-import json
+from krkn_lib.k8s import KrknKubernetes


 class Alibaba:
@@ -180,9 +180,9 @@ class Alibaba:
        logging.info("ECS %s is released" % instance_id)
        return True

-# krkn_lib_kubernetes
+# krkn_lib
 class alibaba_node_scenarios(abstract_node_scenarios):
-    def __init__(self,kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self,kubecli: KrknKubernetes):
        self.alibaba = Alibaba()

    # Node scenario to start the node
--- a/kraken/node_actions/aws_node_scenarios.py
+++ b/kraken/node_actions/aws_node_scenarios.py
@@ -2,10 +2,9 @@ import sys
 import time
 import boto3
 import logging
-import krkn_lib_kubernetes
 import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-
+from krkn_lib.k8s import KrknKubernetes

 class AWS:
    def __init__(self):
@@ -166,9 +165,9 @@ class AWS:
            # sys.exit(1)
            raise RuntimeError()

-# krkn_lib_kubernetes
+# krkn_lib
 class aws_node_scenarios(abstract_node_scenarios):
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self, kubecli: KrknKubernetes):
        super().__init__(kubecli)
        self.aws = AWS()

--- a/kraken/node_actions/az_node_scenarios.py
+++ b/kraken/node_actions/az_node_scenarios.py
@@ -1,13 +1,14 @@
-import sys
+
 import time
-from azure.mgmt.compute import ComputeManagementClient
-from azure.identity import DefaultAzureCredential
+import yaml
+import kraken.invoke.command as runcommand
 import logging
-import krkn_lib_kubernetes
 import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-import kraken.invoke.command as runcommand
-import yaml
+from azure.mgmt.compute import ComputeManagementClient
+from azure.identity import DefaultAzureCredential
+from krkn_lib.k8s import KrknKubernetes
+


 class Azure:
@@ -129,9 +130,9 @@ class Azure:
                logging.info("Vm %s is terminated" % vm_name)
                return True

-# krkn_lib_kubernetes
+# krkn_lib
 class azure_node_scenarios(abstract_node_scenarios):
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self, kubecli: KrknKubernetes):
        super().__init__(kubecli)
        logging.info("init in azure")
        self.azure = Azure()
--- a/kraken/node_actions/bm_node_scenarios.py
+++ b/kraken/node_actions/bm_node_scenarios.py
@@ -1,6 +1,5 @@
 import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-import krkn_lib_kubernetes
 import logging
 import openshift as oc
 import pyipmi
@@ -8,7 +7,7 @@ import pyipmi.interfaces
 import sys
 import time
 import traceback
-
+from krkn_lib.k8s import KrknKubernetes

 class BM:
    def __init__(self, bm_info, user, passwd):
@@ -105,9 +104,9 @@ class BM:
        while self.get_ipmi_connection(bmc_addr, node_name).get_chassis_status().power_on:
            time.sleep(1)

-# krkn_lib_kubernetes
+# krkn_lib
 class bm_node_scenarios(abstract_node_scenarios):
-    def __init__(self, bm_info, user, passwd, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self, bm_info, user, passwd, kubecli: KrknKubernetes):
        super().__init__(kubecli)
        self.bm = BM(bm_info, user, passwd)

--- a/kraken/node_actions/common_node_functions.py
+++ b/kraken/node_actions/common_node_functions.py
@@ -2,14 +2,13 @@ import time
 import random
 import logging
 import paramiko
-import krkn_lib_kubernetes
 import kraken.invoke.command as runcommand
-
+from krkn_lib.k8s import KrknKubernetes
 node_general = False


 # Pick a random node with specified label selector
-def get_node(node_name, label_selector, instance_kill_count, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def get_node(node_name, label_selector, instance_kill_count, kubecli: KrknKubernetes):
    if node_name in kubecli.list_killable_nodes():
        return [node_name]
    elif node_name:
@@ -29,21 +28,21 @@ def get_node(node_name, label_selector, instance_kill_count, kubecli: krkn_lib_k
    return nodes_to_return


-# krkn_lib_kubernetes
+# krkn_lib
 # Wait until the node status becomes Ready
-def wait_for_ready_status(node, timeout, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def wait_for_ready_status(node, timeout, kubecli: KrknKubernetes):
    resource_version = kubecli.get_node_resource_version(node)
    kubecli.watch_node_status(node, "True", timeout, resource_version)

-# krkn_lib_kubernetes
+# krkn_lib
 # Wait until the node status becomes Not Ready
-def wait_for_not_ready_status(node, timeout, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def wait_for_not_ready_status(node, timeout, kubecli: KrknKubernetes):
    resource_version = kubecli.get_node_resource_version(node)
    kubecli.watch_node_status(node, "False", timeout, resource_version)

-# krkn_lib_kubernetes
+# krkn_lib
 # Wait until the node status becomes Unknown
-def wait_for_unknown_status(node, timeout, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def wait_for_unknown_status(node, timeout, kubecli: KrknKubernetes):
    resource_version = kubecli.get_node_resource_version(node)
    kubecli.watch_node_status(node, "Unknown", timeout, resource_version)

--- a/kraken/node_actions/docker_node_scenarios.py
+++ b/kraken/node_actions/docker_node_scenarios.py
@@ -1,10 +1,9 @@
 import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-import krkn_lib_kubernetes
 import logging
 import sys
 import docker
-
+from krkn_lib.k8s import KrknKubernetes

 class Docker:
    def __init__(self):
@@ -37,7 +36,7 @@ class Docker:


 class docker_node_scenarios(abstract_node_scenarios):
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self, kubecli: KrknKubernetes):
        super().__init__(kubecli)
        self.docker = Docker()

--- a/kraken/node_actions/gcp_node_scenarios.py
+++ b/kraken/node_actions/gcp_node_scenarios.py
@@ -1,13 +1,12 @@
 import sys
 import time
 import logging
-import krkn_lib_kubernetes
 import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
 from googleapiclient import discovery
 from oauth2client.client import GoogleCredentials
 import kraken.invoke.command as runcommand
-
+from krkn_lib.k8s import KrknKubernetes

 class GCP:
    def __init__(self):
@@ -143,9 +142,9 @@ class GCP:
            return True


-# krkn_lib_kubernetes
+# krkn_lib
 class gcp_node_scenarios(abstract_node_scenarios):
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self, kubecli: KrknKubernetes):
        super().__init__(kubecli)
        self.gcp = GCP()

--- a/kraken/node_actions/general_cloud_node_scenarios.py
+++ b/kraken/node_actions/general_cloud_node_scenarios.py
@@ -1,15 +1,14 @@
 import logging
-import krkn_lib_kubernetes
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-
+from krkn_lib.k8s import KrknKubernetes

 class GENERAL:
    def __init__(self):
        pass

-# krkn_lib_kubernetes
+# krkn_lib
 class general_node_scenarios(abstract_node_scenarios):
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes ):
+    def __init__(self, kubecli: KrknKubernetes):
        super().__init__(kubecli)
        self.general = GENERAL()

--- a/kraken/node_actions/openstack_node_scenarios.py
+++ b/kraken/node_actions/openstack_node_scenarios.py
@@ -1,11 +1,10 @@
 import sys
 import time
 import logging
-import krkn_lib_kubernetes
 import kraken.invoke.command as runcommand
 import kraken.node_actions.common_node_functions as nodeaction
 from kraken.node_actions.abstract_node_scenarios import abstract_node_scenarios
-
+from krkn_lib.k8s import KrknKubernetes

 class OPENSTACKCLOUD:
    def __init__(self):
@@ -93,9 +92,9 @@ class OPENSTACKCLOUD:
                    return node_name
                counter += 1

-# krkn_lib_kubernetes
+# krkn_lib
 class openstack_node_scenarios(abstract_node_scenarios):
-    def __init__(self, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+    def __init__(self, kubecli: KrknKubernetes):
        self.openstackcloud = OPENSTACKCLOUD()

    # Node scenario to start the node
--- a/kraken/node_actions/run.py
+++ b/kraken/node_actions/run.py
@@ -2,7 +2,6 @@ import yaml
 import logging
 import sys
 import time
-import krkn_lib_kubernetes
 from kraken.node_actions.aws_node_scenarios import aws_node_scenarios
 from kraken.node_actions.general_cloud_node_scenarios import general_node_scenarios
 from kraken.node_actions.az_node_scenarios import azure_node_scenarios
@@ -13,14 +12,15 @@ from kraken.node_actions.bm_node_scenarios import bm_node_scenarios
 from kraken.node_actions.docker_node_scenarios import docker_node_scenarios
 import kraken.node_actions.common_node_functions as common_node_functions
 import kraken.cerberus.setup as cerberus
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
-
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
 node_general = False


 # Get the node scenarios object of specfied cloud type
-# krkn_lib_kubernetes
-def get_node_scenario_object(node_scenario, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def get_node_scenario_object(node_scenario, kubecli: KrknKubernetes):
    if "cloud_type" not in node_scenario.keys() or node_scenario["cloud_type"] == "generic":
        global node_general
        node_general = True
@@ -52,8 +52,8 @@ def get_node_scenario_object(node_scenario, kubecli: krkn_lib_kubernetes.KrknLib


 # Run defined scenarios
-# krkn_lib_kubernetes
-def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+# krkn_lib
+def run(scenarios_list, config, wait_duration, kubecli: KrknKubernetes, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    scenario_telemetries: list[ScenarioTelemetry] = []
    failed_scenarios = []
    for node_scenario_config in scenarios_list:
@@ -78,7 +78,7 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn
                        except (RuntimeError, Exception) as e:
                            scenario_telemetry.exitStatus = 1
                            failed_scenarios.append(node_scenario_config)
-                            telemetry.log_exception(node_scenario_config)
+                            log_exception(node_scenario_config)
                        else:
                            scenario_telemetry.exitStatus = 0

@@ -89,16 +89,20 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn


 # Inject the specified node scenario
-def inject_node_scenario(action, node_scenario, node_scenario_object, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def inject_node_scenario(action, node_scenario, node_scenario_object, kubecli: KrknKubernetes):
    generic_cloud_scenarios = ("stop_kubelet_scenario", "node_crash_scenario")
    # Get the node scenario configurations
-    run_kill_count = node_scenario.get("runs", 1)
-    instance_kill_count = node_scenario.get("instance_count", 1)
-    node_name = node_scenario.get("node_name", "")
-    label_selector = node_scenario.get("label_selector", "")
-    timeout = node_scenario.get("timeout", 120)
-    service = node_scenario.get("service", "")
-    ssh_private_key = node_scenario.get("ssh_private_key", "~/.ssh/id_rsa")
+    run_kill_count = get_yaml_item_value(node_scenario, "runs", 1)
+    instance_kill_count = get_yaml_item_value(
+        node_scenario, "instance_count", 1
+    )
+    node_name = get_yaml_item_value(node_scenario, "node_name", "")
+    label_selector = get_yaml_item_value(node_scenario, "label_selector", "")
+    timeout = get_yaml_item_value(node_scenario, "timeout", 120)
+    service = get_yaml_item_value(node_scenario, "service", "")
+    ssh_private_key = get_yaml_item_value(
+        node_scenario, "ssh_private_key", "~/.ssh/id_rsa"
+    )
    # Get the node to apply the scenario
    if node_name:
        node_name_list = node_name.split(",")
--- a/kraken/plugins/init.py
+++ b/kraken/plugins/init.py
@@ -13,7 +13,9 @@ from kraken.plugins.run_python_plugin import run_python_file
 from kraken.plugins.network.ingress_shaping import network_chaos
 from kraken.plugins.pod_network_outage.pod_network_outage_plugin import pod_outage
 from kraken.plugins.pod_network_outage.pod_network_outage_plugin import pod_egress_shaping
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import log_exception


@dataclasses.dataclass
@@ -221,25 +223,25 @@ PLUGINS = Plugins(
            [
                "error"
            ]
-        )       
+        )
    ]
 )


-def run(scenarios: List[str], kubeconfig_path: str, kraken_config: str, failed_post_scenarios: List[str], wait_duration: int, telemetry: KrknTelemetry) -> (List[str], list[ScenarioTelemetry]):
+def run(scenarios: List[str], kubeconfig_path: str, kraken_config: str, failed_post_scenarios: List[str], wait_duration: int, telemetry: KrknTelemetryKubernetes) -> (List[str], list[ScenarioTelemetry]):
    scenario_telemetries: list[ScenarioTelemetry] = []
    for scenario in scenarios:
        scenario_telemetry = ScenarioTelemetry()
        scenario_telemetry.scenario = scenario
        scenario_telemetry.startTimeStamp = time.time()
        telemetry.set_parameters_base64(scenario_telemetry, scenario)
-        logging.info('scenario '+ str(scenario))
+        logging.info('scenario ' + str(scenario))
        try:
            PLUGINS.run(scenario, kubeconfig_path, kraken_config)
        except Exception as e:
            scenario_telemetry.exitStatus = 1
            failed_post_scenarios.append(scenario)
-            telemetry.log_exception(scenario)
+            log_exception(scenario)
        else:
            scenario_telemetry.exitStatus = 0
            logging.info("Waiting for the specified duration: %s" % (wait_duration))
--- a/kraken/plugins/pod_network_outage/pod_network_outage_plugin.py
+++ b/kraken/plugins/pod_network_outage/pod_network_outage_plugin.py
@@ -10,7 +10,7 @@ import re
 from dataclasses import dataclass, field
 from traceback import format_exc
 from jinja2 import Environment, FileSystemLoader
-from krkn_lib_kubernetes import KrknLibKubernetes
+from krkn_lib.k8s import KrknKubernetes
 from arcaflow_plugin_sdk import plugin, validation
 from kubernetes import client
 from kubernetes.client.api.apiextensions_v1_api import ApiextensionsV1Api
@@ -19,7 +19,7 @@ from . import cerberus


 def get_test_pods(
-    pod_name: str, pod_label: str, namespace: str, kubecli: KrknLibKubernetes
+    pod_name: str, pod_label: str, namespace: str, kubecli: KrknKubernetes
 ) -> typing.List[str]:
    """
    Function that returns a list of pods to apply network policy
@@ -35,7 +35,7 @@ def get_test_pods(
        namepsace (string)
            - namespace in which the pod is present

-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

    Returns:
@@ -54,12 +54,12 @@ def get_test_pods(
        return pods_list


-def get_job_pods(kubecli: KrknLibKubernetes, api_response):
+def get_job_pods(kubecli: KrknKubernetes, api_response):
    """
    Function that gets the pod corresponding to the job

    Args:
-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

        api_response
@@ -78,12 +78,12 @@ def get_job_pods(kubecli: KrknLibKubernetes, api_response):
    return pods_list[0]


-def delete_jobs(kubecli: KrknLibKubernetes, job_list: typing.List[str]):
+def delete_jobs(kubecli: KrknKubernetes, job_list: typing.List[str]):
    """
    Function that deletes jobs

    Args:
-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

        job_list (List of strings)
@@ -109,7 +109,7 @@ def delete_jobs(kubecli: KrknLibKubernetes, job_list: typing.List[str]):


 def wait_for_job(
-    job_list: typing.List[str], kubecli: KrknLibKubernetes, timeout: int = 300
+    job_list: typing.List[str], kubecli: KrknKubernetes, timeout: int = 300
 ) -> None:
    """
    Function that waits for a list of jobs to finish within a time period
@@ -118,7 +118,7 @@ def wait_for_job(
        job_list (List of strings)
            - The list of jobs to check for completion

-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

        timeout (int)
@@ -195,7 +195,7 @@ def apply_outage_policy(
    direction: str,
    duration: str,
    bridge_name: str,
-    kubecli: KrknLibKubernetes,
+    kubecli: KrknKubernetes,
 ) -> typing.List[str]:
    """
    Function that applies filters(ingress or egress) to block traffic.
@@ -278,7 +278,7 @@ def apply_net_policy(
    network_params: typing.Dict[str, str],
    duration: str,
    bridge_name: str,
-    kubecli: KrknLibKubernetes,
+    kubecli: KrknKubernetes,
    test_execution: str,
 ) -> typing.List[str]:
    """
@@ -312,7 +312,7 @@ def apply_net_policy(
        bridge_name (string):
            - bridge to which  filter rules need to be applied

-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

        test_execution (String)
@@ -393,7 +393,7 @@ def get_egress_cmd(


 def list_bridges(
-    node: str, pod_template, kubecli: KrknLibKubernetes
+    node: str, pod_template, kubecli: KrknKubernetes
 ) -> typing.List[str]:
    """
    Function that returns a list of bridges on the node
@@ -406,7 +406,7 @@ def list_bridges(
            - The YAML template used to instantiate a pod to query
              the node's interface

-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

    Returns:
@@ -437,7 +437,7 @@ def list_bridges(


 def check_cookie(
-    node: str, pod_template, br_name, cookie, kubecli: KrknLibKubernetes
+    node: str, pod_template, br_name, cookie, kubecli: KrknKubernetes
 ) -> str:
    """
    Function to check for matching flow rules
@@ -496,7 +496,7 @@ def check_cookie(


 def get_pod_interface(
-    node: str, ip: str, pod_template, br_name, kubecli: KrknLibKubernetes
+    node: str, ip: str, pod_template, br_name, kubecli: KrknKubernetes
 ) -> str:
    """
    Function to query the pod interface on a node
@@ -515,7 +515,7 @@ def get_pod_interface(
        br_name (string):
            - bridge against which the flows rules need to be checked

-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

    Returns
@@ -576,7 +576,7 @@ def get_pod_interface(


 def check_bridge_interface(
-    node_name: str, pod_template, bridge_name: str, kubecli: KrknLibKubernetes
+    node_name: str, pod_template, bridge_name: str, kubecli: KrknKubernetes
 ) -> bool:
    """
    Function  is used to check if the required OVS or OVN bridge is found in
@@ -593,7 +593,7 @@ def check_bridge_interface(
        bridge_name (string):
            - bridge name to check for in the node.

-        kubecli (KrknLibKubernetes)
+        kubecli (KrknKubernetes)
            - Object to interact with Kubernetes Python client

    Returns:
@@ -815,7 +815,7 @@ def pod_outage(
        node_dict = {}
        label_set = set()

-        kubecli = KrknLibKubernetes(kubeconfig_path=params.kubeconfig_path)
+        kubecli = KrknKubernetes(kubeconfig_path=params.kubeconfig_path)
        api_ext = client.ApiextensionsV1Api(kubecli.api_client)
        custom_obj = client.CustomObjectsApi(kubecli.api_client)

@@ -1073,7 +1073,7 @@ def pod_egress_shaping(
        param_lst = ["latency", "loss", "bandwidth"]
        mod_lst = [i for i in param_lst if i in params.network_params]

-        kubecli = KrknLibKubernetes(kubeconfig_path=params.kubeconfig_path)
+        kubecli = KrknKubernetes(kubeconfig_path=params.kubeconfig_path)
        api_ext = client.ApiextensionsV1Api(kubecli.api_client)
        custom_obj = client.CustomObjectsApi(kubecli.api_client)

--- a/kraken/pod_scenarios/setup.py
+++ b/kraken/pod_scenarios/setup.py
@@ -1,16 +1,17 @@
 import logging
-
-from arcaflow_plugin_sdk import serialization
-import arcaflow_plugin_kill_pod
-
-import kraken.cerberus.setup as cerberus
-import kraken.post_actions.actions as post_actions
-import krkn_lib_kubernetes
 import time
 import yaml
 import sys
 import random
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+import arcaflow_plugin_kill_pod
+import kraken.cerberus.setup as cerberus
+import kraken.post_actions.actions as post_actions
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from arcaflow_plugin_sdk import serialization
+from krkn_lib.utils.functions import get_yaml_item_value, log_exception
+

 # Run pod based scenarios
 def run(kubeconfig_path, scenarios_list, config, failed_post_scenarios, wait_duration):
@@ -66,14 +67,15 @@ def run(kubeconfig_path, scenarios_list, config, failed_post_scenarios, wait_dur
        cerberus.publish_kraken_status(config, failed_post_scenarios, start_time, end_time)
    return failed_post_scenarios

-# krkn_lib_kubernetes
+
+# krkn_lib
 def container_run(kubeconfig_path,
                  scenarios_list,
                  config,
                  failed_post_scenarios,
                  wait_duration,
-                  kubecli: krkn_lib_kubernetes.KrknLibKubernetes,
-                  telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+                  kubecli: KrknKubernetes,
+                  telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):

    failed_scenarios = []
    scenario_telemetries: list[ScenarioTelemetry] = []
@@ -116,7 +118,7 @@ def container_run(kubeconfig_path,
                    cerberus.publish_kraken_status(config, failed_post_scenarios, start_time, end_time)
                except (RuntimeError, Exception):
                    failed_scenarios.append(container_scenario_config[0])
-                    telemetry.log_exception(container_scenario_config[0])
+                    log_exception(container_scenario_config[0])
                    scenario_telemetry.exitStatus = 1
                    # removed_exit
                    # sys.exit(1)
@@ -128,15 +130,22 @@ def container_run(kubeconfig_path,
    return failed_scenarios, scenario_telemetries


-
-def container_killing_in_pod(cont_scenario, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
-    scenario_name = cont_scenario.get("name", "")
-    namespace = cont_scenario.get("namespace", "*")
-    label_selector = cont_scenario.get("label_selector", None)
-    pod_names = cont_scenario.get("pod_names", [])
-    container_name = cont_scenario.get("container_name", "")
-    kill_action = cont_scenario.get("action", "kill 1")
-    kill_count = cont_scenario.get("count", 1)
+def container_killing_in_pod(cont_scenario, kubecli: KrknKubernetes):
+    scenario_name = get_yaml_item_value(cont_scenario, "name", "")
+    namespace = get_yaml_item_value(cont_scenario, "namespace", "*")
+    label_selector = get_yaml_item_value(cont_scenario, "label_selector", None)
+    pod_names = get_yaml_item_value(cont_scenario, "pod_names", [])
+    container_name = get_yaml_item_value(cont_scenario, "container_name", "")
+    kill_action = get_yaml_item_value(cont_scenario, "action", 1)
+    kill_count = get_yaml_item_value(cont_scenario, "count", 1)
+    if not isinstance(kill_action, int):
+        logging.error("Please make sure the action parameter defined in the "
+                      "config is an integer")
+        raise RuntimeError()
+    if (kill_action < 1) or (kill_action > 15):
+        logging.error("Only 1-15 kill signals are supported.")
+        raise RuntimeError()
+    kill_action = "kill " + str(kill_action)
    if type(pod_names) != list:
        logging.error("Please make sure your pod_names are in a list format")
        # removed_exit
@@ -196,7 +205,7 @@ def container_killing_in_pod(cont_scenario, kubecli: krkn_lib_kubernetes.KrknLib
    return killed_container_list


-def retry_container_killing(kill_action, podname, namespace, container_name, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def retry_container_killing(kill_action, podname, namespace, container_name, kubecli: KrknKubernetes):
    i = 0
    while i < 5:
        logging.info("Killing container %s in pod %s (ns %s)" % (str(container_name), str(podname), str(namespace)))
@@ -213,7 +222,7 @@ def retry_container_killing(kill_action, podname, namespace, container_name, kub
            continue


-def check_failed_containers(killed_container_list, wait_time, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+def check_failed_containers(killed_container_list, wait_time, kubecli: KrknKubernetes):

    container_ready = []
    timer = 0
--- a/kraken/pvc/pvc_scenario.py
+++ b/kraken/pvc/pvc_scenario.py
@@ -1,16 +1,17 @@
 import logging
 import random
 import re
-import sys
 import time
-import krkn_lib_kubernetes
 import yaml
-
 from ..cerberus import setup as cerberus
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import get_yaml_item_value, log_exception

-# krkn_lib_kubernetes
-def run(scenarios_list, config, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+
+# krkn_lib
+def run(scenarios_list, config, kubecli: KrknKubernetes, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    """
    Reads the scenario config and creates a temp file to fill up the PVC
    """
@@ -27,13 +28,21 @@ def run(scenarios_list, config, kubecli: krkn_lib_kubernetes.KrknLibKubernetes,
                with open(app_config, "r") as f:
                    config_yaml = yaml.full_load(f)
                    scenario_config = config_yaml["pvc_scenario"]
-                    pvc_name = scenario_config.get("pvc_name", "")
-                    pod_name = scenario_config.get("pod_name", "")
-                    namespace = scenario_config.get("namespace", "")
-                    target_fill_percentage = scenario_config.get(
-                        "fill_percentage", "50"
+                    pvc_name = get_yaml_item_value(
+                        scenario_config, "pvc_name", ""
+                    )
+                    pod_name = get_yaml_item_value(
+                        scenario_config, "pod_name", ""
+                    )
+                    namespace = get_yaml_item_value(
+                        scenario_config, "namespace", ""
+                    )
+                    target_fill_percentage = get_yaml_item_value(
+                        scenario_config, "fill_percentage", "50"
+                    )
+                    duration = get_yaml_item_value(
+                        scenario_config, "duration", 60
                    )
-                    duration = scenario_config.get("duration", 60)

                    logging.info(
                        "Input params:\n"
@@ -307,7 +316,7 @@ def run(scenarios_list, config, kubecli: krkn_lib_kubernetes.KrknLibKubernetes,
        except (RuntimeError, Exception):
            scenario_telemetry.exitStatus = 1
            failed_scenarios.append(app_config)
-            telemetry.log_exception(app_config)
+            log_exception(app_config)
        else:
            scenario_telemetry.exitStatus = 0
        scenario_telemetries.append(scenario_telemetry)
@@ -317,7 +326,7 @@ def run(scenarios_list, config, kubecli: krkn_lib_kubernetes.KrknLibKubernetes,



-# krkn_lib_kubernetes
+# krkn_lib
 def remove_temp_file(
    file_name,
    full_path,
@@ -326,7 +335,7 @@ def remove_temp_file(
    container_name,
    mount_path,
    file_size_kb,
-    kubecli: krkn_lib_kubernetes.KrknLibKubernetes
+    kubecli: KrknKubernetes
 ):
    command = "rm -f %s" % (str(full_path))
    logging.debug("Remove temp file from the PVC command:\n %s" % command)
--- a/kraken/service_disruption/init.py
+++ b/kraken/service_disruption/init.py
--- a/kraken/service_disruption/common_service_disruption_functions.py
+++ b/kraken/service_disruption/common_service_disruption_functions.py
@@ -0,0 +1,325 @@
+import time
+import random
+import logging
+import kraken.cerberus.setup as cerberus
+import kraken.post_actions.actions as post_actions
+import yaml
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import get_yaml_item_value, log_exception
+
+
+def delete_objects(kubecli, namespace):
+
+    services = delete_all_services_namespace(kubecli, namespace)
+    daemonsets = delete_all_daemonset_namespace(kubecli, namespace)
+    statefulsets = delete_all_statefulsets_namespace(kubecli, namespace)
+    replicasets = delete_all_replicaset_namespace(kubecli, namespace)
+    deployments = delete_all_deployment_namespace(kubecli, namespace)
+
+    objects = { "daemonsets": daemonsets,
+                "deployments": deployments,
+                "replicasets": replicasets,
+                "statefulsets": statefulsets,
+                "services": services
+                }
+
+    return objects
+
+
+def get_list_running_pods(kubecli: KrknKubernetes, namespace: str):
+    running_pods = []
+    pods = kubecli.list_pods(namespace)
+    for pod in pods:
+        pod_status = kubecli.get_pod_info(pod, namespace)
+        if pod_status and pod_status.status == "Running":
+            running_pods.append(pod)
+    logging.info('all running pods ' + str(running_pods))
+    return running_pods
+
+
+def delete_all_deployment_namespace(kubecli: KrknKubernetes, namespace: str):
+    """
+    Delete all the deployments in the specified namespace
+
+    :param kubecli: krkn kubernetes python package
+    :param namespace: namespace
+    """
+    try:
+        deployments = kubecli.get_deployment_ns(namespace)
+        for deployment in deployments:
+            logging.info("Deleting deployment" + deployment)
+            kubecli.delete_deployment(deployment, namespace)
+    except Exception as e:
+        logging.error(
+            "Exception when calling delete_all_deployment_namespace: %s\n",
+            str(e),
+        )
+        raise e
+
+    return deployments
+
+
+def delete_all_daemonset_namespace(kubecli: KrknKubernetes, namespace: str):
+    """
+    Delete all the daemonset in the specified namespace
+
+    :param kubecli: krkn kubernetes python package
+    :param namespace: namespace
+    """
+    try:
+        daemonsets = kubecli.get_daemonset(namespace)
+        for daemonset in daemonsets:
+            logging.info("Deleting daemonset" + daemonset)
+            kubecli.delete_daemonset(daemonset, namespace)
+    except Exception as e:
+        logging.error(
+            "Exception when calling delete_all_daemonset_namespace: %s\n",
+            str(e),
+        )
+        raise e
+
+    return daemonsets
+
+
+def delete_all_statefulsets_namespace(kubecli: KrknKubernetes, namespace: str):
+    """
+    Delete all the statefulsets in the specified namespace
+
+
+    :param kubecli: krkn kubernetes python package
+    :param namespace: namespace
+    """
+    try:
+        statefulsets = kubecli.get_all_statefulset(namespace)
+        for statefulset in statefulsets:
+            logging.info("Deleting statefulsets" + statefulsets)
+            kubecli.delete_statefulset(statefulset, namespace)
+    except Exception as e:
+        logging.error(
+            "Exception when calling delete_all_statefulsets_namespace: %s\n",
+            str(e),
+        )
+        raise e
+
+    return statefulsets
+
+
+def delete_all_replicaset_namespace(kubecli: KrknKubernetes, namespace: str):
+    """
+    Delete all the replicasets in the specified namespace
+
+    :param kubecli: krkn kubernetes python package
+    :param namespace: namespace
+    """
+    try:
+        replicasets = kubecli.get_all_replicasets(namespace)
+        for replicaset in replicasets:
+            logging.info("Deleting replicaset" + replicaset)
+            kubecli.delete_replicaset(replicaset, namespace)
+    except Exception as e:
+        logging.error(
+            "Exception when calling delete_all_replicaset_namespace: %s\n",
+            str(e),
+        )
+        raise e
+
+    return replicasets
+
+def delete_all_services_namespace(kubecli: KrknKubernetes, namespace: str):
+    """
+    Delete all the services in the specified namespace
+
+
+    :param kubecli: krkn kubernetes python package
+    :param namespace: namespace
+    """
+    try:
+        services = kubecli.get_all_services(namespace)
+        for service in services:
+            logging.info("Deleting services" + service)
+            kubecli.delete_services(service, namespace)
+    except Exception as e:
+        logging.error(
+            "Exception when calling delete_all_services_namespace: %s\n",
+            str(e),
+        )
+        raise e
+
+    return services
+
+
+# krkn_lib
+def run(
+        scenarios_list,
+        config,
+        wait_duration,
+        failed_post_scenarios,
+        kubeconfig_path,
+        kubecli: KrknKubernetes,
+        telemetry: KrknTelemetryKubernetes
+) -> (list[str], list[ScenarioTelemetry]):
+    scenario_telemetries: list[ScenarioTelemetry] = []
+    failed_scenarios = []
+    for scenario_config in scenarios_list:
+        scenario_telemetry = ScenarioTelemetry()
+        scenario_telemetry.scenario = scenario_config[0]
+        scenario_telemetry.startTimeStamp = time.time()
+        telemetry.set_parameters_base64(scenario_telemetry, scenario_config[0])
+        try:
+            if len(scenario_config) > 1:
+                pre_action_output = post_actions.run(kubeconfig_path, scenario_config[1])
+            else:
+                pre_action_output = ""
+            with open(scenario_config[0], "r") as f:
+                scenario_config_yaml = yaml.full_load(f)
+                for scenario in scenario_config_yaml["scenarios"]:
+                    scenario_namespace = get_yaml_item_value(
+                        scenario, "namespace", ""
+                    )
+                    scenario_label = get_yaml_item_value(
+                        scenario, "label_selector", ""
+                    )
+                    if scenario_namespace is not None and scenario_namespace.strip() != "":
+                        if scenario_label is not None and scenario_label.strip() != "":
+                            logging.error("You can only have namespace or label set in your namespace scenario")
+                            logging.error(
+                                "Current scenario config has namespace '%s' and label selector '%s'"
+                                % (scenario_namespace, scenario_label)
+                            )
+                            logging.error(
+                                "Please set either namespace to blank ('') or label_selector to blank ('') to continue"
+                            )
+                            # removed_exit
+                            # sys.exit(1)
+                            raise RuntimeError()
+                    delete_count = get_yaml_item_value(
+                        scenario, "delete_count", 1
+                    )
+                    run_count = get_yaml_item_value(scenario, "runs", 1)
+                    run_sleep = get_yaml_item_value(scenario, "sleep", 10)
+                    wait_time = get_yaml_item_value(scenario, "wait_time", 30)
+
+                    logging.info(str(scenario_namespace) + str(scenario_label) + str(delete_count) + str(run_count) + str(run_sleep) + str(wait_time))
+                    logging.info("done")
+                    start_time = int(time.time())
+                    for i in range(run_count):
+                        killed_namespaces = {}
+                        namespaces = kubecli.check_namespaces([scenario_namespace], scenario_label)
+                        for j in range(delete_count):
+                            if len(namespaces) == 0:
+                                logging.error(
+                                    "Couldn't delete %s namespaces, not enough namespaces matching %s with label %s"
+                                    % (str(run_count), scenario_namespace, str(scenario_label))
+                                )
+                                # removed_exit
+                                # sys.exit(1)
+                                raise RuntimeError()
+                            selected_namespace = namespaces[random.randint(0, len(namespaces) - 1)]
+                            logging.info('Delete objects in selected namespace: ' + selected_namespace )
+                            try:
+                                # delete all pods in namespace
+                                objects = delete_objects(kubecli,selected_namespace)
+                                killed_namespaces[selected_namespace] = objects
+                                logging.info("Deleted all objects in namespace %s was successful" % str(selected_namespace))
+                            except Exception as e:
+                                logging.info("Delete all objects in namespace %s was unsuccessful" % str(selected_namespace))
+                                logging.info("Namespace action error: " + str(e))
+                                raise RuntimeError()
+                            namespaces.remove(selected_namespace)
+                            logging.info("Waiting %s seconds between namespace deletions" % str(run_sleep))
+                            time.sleep(run_sleep)
+
+                        logging.info("Waiting for the specified duration: %s" % wait_duration)
+                        time.sleep(wait_duration)
+                        if len(scenario_config) > 1:
+                            try:
+                                failed_post_scenarios = post_actions.check_recovery(
+                                    kubeconfig_path, scenario_config, failed_post_scenarios, pre_action_output
+                                )
+                            except Exception as e:
+                                logging.error("Failed to run post action checks: %s" % e)
+                                # removed_exit
+                                # sys.exit(1)
+                                raise RuntimeError()
+                        else:
+                            failed_post_scenarios = check_all_running_deployment(killed_namespaces, wait_time, kubecli)
+
+                    end_time = int(time.time())
+                    cerberus.publish_kraken_status(config, failed_post_scenarios, start_time, end_time)
+        except (Exception, RuntimeError):
+            scenario_telemetry.exitStatus = 1
+            failed_scenarios.append(scenario_config[0])
+            log_exception(scenario_config[0])
+        else:
+            scenario_telemetry.exitStatus = 0
+        scenario_telemetry.endTimeStamp = time.time()
+        scenario_telemetries.append(scenario_telemetry)
+    return failed_scenarios, scenario_telemetries
+
+
+def check_all_running_pods(kubecli: KrknKubernetes, namespace_name, wait_time):
+
+    timer = 0
+    while timer < wait_time:
+        pod_list = kubecli.list_pods(namespace_name)
+        pods_running = 0
+        for pod in pod_list:
+            pod_info = kubecli.get_pod_info(pod, namespace_name)
+            if pod_info.status != "Running" and pod_info.status != "Succeeded":
+                logging.info("Pods %s still not running or completed" % pod_info.name)
+                break
+            pods_running += 1
+        if len(pod_list) == pods_running:
+            break
+        timer += 5
+        time.sleep(5)
+        logging.info("Waiting 5 seconds for pods to become active")
+
+# krkn_lib
+def check_all_running_deployment(killed_namespaces, wait_time, kubecli: KrknKubernetes):
+
+    timer = 0
+    while timer < wait_time and killed_namespaces:
+        still_missing_ns = killed_namespaces.copy()
+        for namespace_name, objects in killed_namespaces.items():
+            still_missing_obj = objects.copy()
+            for obj_name, obj_list in objects.items():
+                if "deployments" == obj_name:
+                    deployments = kubecli.get_deployment_ns(namespace_name)
+                    if len(obj_list) == len(deployments):
+                        still_missing_obj.pop(obj_name)
+                elif "replicasets" == obj_name:
+                    replicasets = kubecli.get_all_replicasets(namespace_name)
+                    if len(obj_list) == len(replicasets):
+                        still_missing_obj.pop(obj_name)
+                elif "statefulsets" == obj_name:
+                    statefulsets = kubecli.get_all_statefulset(namespace_name)
+                    if len(obj_list) == len(statefulsets):
+                        still_missing_obj.pop(obj_name)
+                elif "services" == obj_name:
+                    services = kubecli.get_all_services(namespace_name)
+                    if len(obj_list) == len(services):
+                        still_missing_obj.pop(obj_name)
+                elif "daemonsets" == obj_name:
+                    daemonsets = kubecli.get_daemonset(namespace_name)
+                    if len(obj_list) == len(daemonsets):
+                        still_missing_obj.pop(obj_name)
+            logging.info("Still missing objects " + str(still_missing_obj))
+            killed_namespaces[namespace_name] = still_missing_obj.copy()
+            if len(killed_namespaces[namespace_name].keys()) == 0:
+                logging.info("Wait for pods to become running for namespace: " + namespace_name)
+                check_all_running_pods(kubecli, namespace_name, wait_time)
+                still_missing_ns.pop(namespace_name)
+        killed_namespaces = still_missing_ns
+        if len(killed_namespaces.keys()) == 0:
+            return []
+
+        timer += 10
+        time.sleep(10)
+        logging.info("Waiting 10 seconds for objects in namespaces to become active")
+
+    logging.error("Objects are still not ready after waiting " + str(wait_time) + "seconds")
+    logging.error("Non active namespaces " + str(killed_namespaces))
+    return killed_namespaces
--- a/kraken/shut_down/common_shut_down_func.py
+++ b/kraken/shut_down/common_shut_down_func.py
@@ -1,19 +1,18 @@
 #!/usr/bin/env python
-import os
-import sys
 import yaml
 import logging
 import time
-import krkn_lib_kubernetes
 from multiprocessing.pool import ThreadPool
-
 from ..cerberus import setup as cerberus
 from ..post_actions import actions as post_actions
 from ..node_actions.aws_node_scenarios import AWS
 from ..node_actions.openstack_node_scenarios import OPENSTACKCLOUD
 from ..node_actions.az_node_scenarios import Azure
 from ..node_actions.gcp_node_scenarios import GCP
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import log_exception

 def multiprocess_nodes(cloud_object_function, nodes):
    try:
@@ -40,8 +39,8 @@ def multiprocess_nodes(cloud_object_function, nodes):


 # Inject the cluster shut down scenario
-# krkn_lib_kubernetes
-def cluster_shut_down(shut_down_config, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def cluster_shut_down(shut_down_config, kubecli: KrknKubernetes):
    runs = shut_down_config["runs"]
    shut_down_duration = shut_down_config["shut_down_duration"]
    cloud_type = shut_down_config["cloud_type"]
@@ -128,9 +127,9 @@ def cluster_shut_down(shut_down_config, kubecli: krkn_lib_kubernetes.KrknLibKube

        logging.info("Successfully injected cluster_shut_down scenario!")

-# krkn_lib_kubernetes
+# krkn_lib

-def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+def run(scenarios_list, config, wait_duration, kubecli: KrknKubernetes, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    failed_post_scenarios = []
    failed_scenarios = []
    scenario_telemetries: list[ScenarioTelemetry] = []
@@ -167,7 +166,7 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn
                )

            except (RuntimeError, Exception):
-                telemetry.log_exception(shut_down_config[0])
+                log_exception(shut_down_config[0])
                failed_scenarios.append(shut_down_config[0])
                scenario_telemetry.exitStatus = 1
            else:
--- a/kraken/time_actions/common_time_functions.py
+++ b/kraken/time_actions/common_time_functions.py
@@ -2,16 +2,18 @@ import datetime
 import time
 import logging
 import re
-import sys
 import yaml
 import random
-import krkn_lib_kubernetes
 from ..cerberus import setup as cerberus
 from ..invoke import command as runcommand
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import get_yaml_item_value, log_exception

-# krkn_lib_kubernetes
-def pod_exec(pod_name, command, namespace, container_name, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+
+# krkn_lib
+def pod_exec(pod_name, command, namespace, container_name, kubecli:KrknKubernetes):
    for i in range(5):
        response = kubecli.exec_cmd_in_pod(
            command,
@@ -40,8 +42,8 @@ def node_debug(node_name, command):
    return response


-# krkn_lib_kubernetes
-def get_container_name(pod_name, namespace, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, container_name=""):
+# krkn_lib
+def get_container_name(pod_name, namespace, kubecli:KrknKubernetes, container_name=""):

    container_names = kubecli.get_containers_in_pod(pod_name, namespace)
    if container_name != "":
@@ -63,9 +65,9 @@ def get_container_name(pod_name, namespace, kubecli: krkn_lib_kubernetes.KrknLib
        return container_name


-# krkn_lib_kubernetes
-def skew_time(scenario, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
-    skew_command = "date --set "
+# krkn_lib
+def skew_time(scenario, kubecli:KrknKubernetes):
+    skew_command = "date --date "
    if scenario["action"] == "skew_date":
        skewed_date = "00-01-01"
        skew_command += skewed_date
@@ -88,7 +90,7 @@ def skew_time(scenario, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
        return "node", node_names

    elif "pod" in scenario["object_type"]:
-        container_name = scenario.get("container_name", "")
+        container_name = get_yaml_item_value(scenario, "container_name", "")
        pod_names = []
        if "object_name" in scenario.keys() and scenario["object_name"]:
            for name in scenario["object_name"]:
@@ -231,8 +233,8 @@ def string_to_date(obj_datetime):
        return datetime.datetime(datetime.MINYEAR, 1, 1)


-# krkn_lib_kubernetes
-def check_date_time(object_type, names, kubecli: krkn_lib_kubernetes.KrknLibKubernetes):
+# krkn_lib
+def check_date_time(object_type, names, kubecli:KrknKubernetes):
    skew_command = "date"
    not_reset = []
    max_retries = 30
@@ -307,8 +309,8 @@ def check_date_time(object_type, names, kubecli: krkn_lib_kubernetes.KrknLibKube
    return not_reset


-# krkn_lib_kubernetes
-def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.KrknLibKubernetes, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]):
+# krkn_lib
+def run(scenarios_list, config, wait_duration, kubecli:KrknKubernetes, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]):
    failed_scenarios = []
    scenario_telemetries: list[ScenarioTelemetry] = []
    for time_scenario_config in scenarios_list:
@@ -338,7 +340,7 @@ def run(scenarios_list, config, wait_duration, kubecli: krkn_lib_kubernetes.Krkn
                    )
        except (RuntimeError, Exception):
            scenario_telemetry.exitStatus = 1
-            telemetry.log_exception(time_scenario_config)
+            log_exception(time_scenario_config)
            failed_scenarios.append(time_scenario_config)
        else:
            scenario_telemetry.exitStatus = 0
--- a/kraken/zone_outage/actions.py
+++ b/kraken/zone_outage/actions.py
@@ -3,9 +3,11 @@ import logging
 import time
 from ..node_actions.aws_node_scenarios import AWS
 from ..cerberus import setup as cerberus
-from krkn_lib_kubernetes import ScenarioTelemetry, KrknTelemetry
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.models.telemetry import ScenarioTelemetry
+from krkn_lib.utils.functions import log_exception

-def run(scenarios_list, config, wait_duration, telemetry: KrknTelemetry) -> (list[str], list[ScenarioTelemetry]) :
+def run(scenarios_list, config, wait_duration, telemetry: KrknTelemetryKubernetes) -> (list[str], list[ScenarioTelemetry]) :
    """
    filters the subnet of interest and applies the network acl
    to create zone outage
@@ -110,7 +112,7 @@ def run(scenarios_list, config, wait_duration, telemetry: KrknTelemetry) -> (lis
        except (RuntimeError, Exception):
            scenario_telemetry.exitStatus = 1
            failed_scenarios.append(zone_outage_config)
-            telemetry.log_exception(zone_outage_config)
+            log_exception(zone_outage_config)
        else:
            scenario_telemetry.exitStatus = 0
        scenario_telemetry.endTimeStamp = time.time()
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,40 +1,41 @@
-coverage
-datetime
-pyfiglet
 PyYAML>=5.1
-requests
-boto3
-google-api-python-client
-azure-mgmt-compute
-azure-keyvault
-azure-identity
-kubernetes
-oauth2client>=4.1.3
-python-openstackclient
-gitpython
-paramiko
-setuptools==65.5.1
-openshift-client
-python-ipmi
-podman-compose
-docker-compose
-docker
-jinja2==3.0.3
-itsdangerous==2.0.1
-werkzeug==2.2.3
-lxml >= 4.3.0
-pyVmomi >= 6.7
-zope.interface==5.4.0
 aliyun-python-sdk-core==2.13.36
 aliyun-python-sdk-ecs==4.24.25
-arcaflow-plugin-sdk>=0.9.0
-wheel
-service_identity
-git+https://github.com/vmware/vsphere-automation-sdk-python.git@v8.0.0.0
+arcaflow >= 0.8.0
+arcaflow-plugin-sdk >= 0.10.0
+azure-identity
+azure-keyvault
+azure-mgmt-compute
+boto3==1.28.61
+coverage
+datetime
+docker
+docker-compose
 git+https://github.com/redhat-chaos/arcaflow-plugin-kill-pod.git
-arcaflow >= 0.4.1
-prometheus_api_client
+git+https://github.com/vmware/vsphere-automation-sdk-python.git@v8.0.0.0
+gitpython
+google-api-python-client
 ibm_cloud_sdk_core
 ibm_vpc
+itsdangerous==2.0.1
+jinja2==3.0.3
+krkn-lib>=1.4.1
+kubernetes
+lxml >= 4.3.0
+oauth2client>=4.1.3
+openshift-client
+paramiko
+podman-compose
+prometheus_api_client
+pyVmomi >= 6.7
+pyfiglet
 pytest
-krkn-lib-kubernetes >= 0.1.3
+python-ipmi
+python-openstackclient
+requests
+service_identity
+setuptools==65.5.1
+werkzeug==3.0.1
+wheel
+zope.interface==5.4.0
+pandas<2.0.0
--- a/run_kraken.py
+++ b/run_kraken.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-
+import json
 import os
 import sys
 import yaml
@@ -12,7 +12,7 @@ import kraken.litmus.common_litmus as common_litmus
 import kraken.time_actions.common_time_functions as time_actions
 import kraken.performance_dashboards.setup as performance_dashboards
 import kraken.pod_scenarios.setup as pod_scenarios
-import kraken.namespace_actions.common_namespace_functions as namespace_actions
+import kraken.service_disruption.common_service_disruption_functions as service_disruption
 import kraken.shut_down.common_shut_down_func as shut_down
 import kraken.node_actions.run as nodeaction
 import kraken.managedcluster_scenarios.run as managedcluster_scenarios
@@ -25,7 +25,14 @@ import kraken.arcaflow_plugin as arcaflow_plugin
 import server as server
 import kraken.prometheus.client as promcli
 from kraken import plugins
-from krkn_lib_kubernetes import KrknLibKubernetes, KrknTelemetry, ChaosRunTelemetry, SafeLogger
+
+from krkn_lib.k8s import KrknKubernetes
+from krkn_lib.ocp import KrknOpenshift
+from krkn_lib.telemetry.k8s import KrknTelemetryKubernetes
+from krkn_lib.telemetry.ocp import KrknTelemetryOpenshift
+from krkn_lib.models.telemetry import ChaosRunTelemetry
+from krkn_lib.utils import SafeLogger
+from krkn_lib.utils.functions import get_yaml_item_value

 KUBE_BURNER_URL = (
    "https://github.com/cloud-bulldozer/kube-burner/"
@@ -45,50 +52,78 @@ def main(cfg):
        with open(cfg, "r") as f:
            config = yaml.full_load(f)
        global kubeconfig_path, wait_duration, kraken_config
-        distribution = config["kraken"].get("distribution", "openshift")
+        distribution = get_yaml_item_value(
+            config["kraken"], "distribution", "openshift"
+        )
        kubeconfig_path = os.path.expanduser(
-            config["kraken"].get("kubeconfig_path", "")
+            get_yaml_item_value(config["kraken"], "kubeconfig_path", "")
        )
        kraken_config = cfg
-        chaos_scenarios = config["kraken"].get("chaos_scenarios", [])
-        publish_running_status = config["kraken"].get("publish_kraken_status", False)
-        port = config["kraken"].get("port")
-        signal_address = config["kraken"].get("signal_address")
-        run_signal = config["kraken"].get("signal_state", "RUN")
-        litmus_install = config["kraken"].get("litmus_install", True)
-        litmus_version = config["kraken"].get("litmus_version", "v1.9.1")
-        litmus_uninstall = config["kraken"].get("litmus_uninstall", False)
-        litmus_uninstall_before_run = config["kraken"].get(
-            "litmus_uninstall_before_run", True
+        chaos_scenarios = get_yaml_item_value(
+            config["kraken"], "chaos_scenarios", []
        )
-        wait_duration = config["tunings"].get("wait_duration", 60)
-        iterations = config["tunings"].get("iterations", 1)
-        daemon_mode = config["tunings"].get("daemon_mode", False)
-        deploy_performance_dashboards = config["performance_monitoring"].get(
-            "deploy_dashboards", False
+        publish_running_status = get_yaml_item_value(
+            config["kraken"], "publish_kraken_status", False
        )
-        dashboard_repo = config["performance_monitoring"].get(
-            "repo", "https://github.com/cloud-bulldozer/performance-dashboards.git"
+        port = get_yaml_item_value(config["kraken"], "port", 8081)
+        signal_address = get_yaml_item_value(
+            config["kraken"], "signal_address", "0.0.0.0")
+        run_signal = get_yaml_item_value(
+            config["kraken"], "signal_state", "RUN"
        )
-        capture_metrics = config["performance_monitoring"].get("capture_metrics", False)
-        kube_burner_url = config["performance_monitoring"].get(
-            "kube_burner_binary_url",
+        litmus_install = get_yaml_item_value(
+            config["kraken"], "litmus_install", False
+        )
+        litmus_version = get_yaml_item_value(
+            config["kraken"], "litmus_version", "v1.9.1"
+        )
+        litmus_uninstall = get_yaml_item_value(
+            config["kraken"], "litmus_uninstall", True
+        )
+        litmus_uninstall_before_run = get_yaml_item_value(
+            config["kraken"], "litmus_uninstall_before_run", True
+        )
+        wait_duration = get_yaml_item_value(
+            config["tunings"], "wait_duration", 60
+        )
+        iterations = get_yaml_item_value(config["tunings"], "iterations", 1)
+        daemon_mode = get_yaml_item_value(
+            config["tunings"], "daemon_mode", False
+        )
+        deploy_performance_dashboards = get_yaml_item_value(
+            config["performance_monitoring"], "deploy_dashboards", False
+        )
+        dashboard_repo = get_yaml_item_value(
+            config["performance_monitoring"], "repo",
+            "https://github.com/cloud-bulldozer/performance-dashboards.git"
+        )
+        capture_metrics = get_yaml_item_value(
+            config["performance_monitoring"], "capture_metrics", False
+        )
+        kube_burner_url = get_yaml_item_value(
+            config["performance_monitoring"], "kube_burner_binary_url",
            KUBE_BURNER_URL.format(version=KUBE_BURNER_VERSION),
        )
-        config_path = config["performance_monitoring"].get(
-            "config_path", "config/kube_burner.yaml"
+        config_path = get_yaml_item_value(
+            config["performance_monitoring"], "config_path",
+            "config/kube_burner.yaml"
        )
-        metrics_profile = config["performance_monitoring"].get(
-            "metrics_profile_path", "config/metrics-aggregated.yaml"
+        metrics_profile = get_yaml_item_value(
+            config["performance_monitoring"], "metrics_profile_path",
+            "config/metrics-aggregated.yaml"
        )
-        prometheus_url = config["performance_monitoring"].get("prometheus_url", "")
+        prometheus_url = config["performance_monitoring"].get("prometheus_url")
        prometheus_bearer_token = config["performance_monitoring"].get(
-            "prometheus_bearer_token", ""
+            "prometheus_bearer_token"
+        )
+        run_uuid = config["performance_monitoring"].get("uuid")
+        enable_alerts = get_yaml_item_value(
+            config["performance_monitoring"], "enable_alerts", False
+        )
+        alert_profile = config["performance_monitoring"].get("alert_profile")
+        check_critical_alerts = get_yaml_item_value(
+            config["performance_monitoring"], "check_critical_alerts", False
        )
-        run_uuid = config["performance_monitoring"].get("uuid", "")
-        enable_alerts = config["performance_monitoring"].get("enable_alerts", False)
-        alert_profile = config["performance_monitoring"].get("alert_profile", "")
-        check_critical_alerts = config["performance_monitoring"].get("check_critical_alerts", False)                          

        # Initialize clients
        if (not os.path.isfile(kubeconfig_path) and
@@ -119,12 +154,15 @@ def main(cfg):
            kubeconfig_path
            os.environ["KUBECONFIG"] = str(kubeconfig_path)
            # krkn-lib-kubernetes init
-            kubecli = KrknLibKubernetes(kubeconfig_path=kubeconfig_path)
+            kubecli = KrknKubernetes(kubeconfig_path=kubeconfig_path)
+            ocpcli = KrknOpenshift(kubeconfig_path=kubeconfig_path)
        except:
            kubecli.initialize_clients(None)

        # KrknTelemetry init
-        telemetry = KrknTelemetry(safe_logger, kubecli)
+        telemetry_k8s = KrknTelemetryKubernetes(safe_logger, kubecli)
+        telemetry_ocp = KrknTelemetryOpenshift(safe_logger, ocpcli)
+

        # find node kraken might be running on
        kubecli.find_kraken_node()
@@ -150,7 +188,9 @@ def main(cfg):

        # Cluster info
        logging.info("Fetching cluster info")
-        cv = kubecli.get_clusterversion_string()
+        cv = ""
+        if config["kraken"]["distribution"] == "openshift":
+            cv = ocpcli.get_clusterversion_string()
        if cv != "":
            logging.info(cv)
        else:
@@ -219,7 +259,7 @@ def main(cfg):
                            sys.exit(1)
                        elif scenario_type == "arcaflow_scenarios":
                            failed_post_scenarios, scenario_telemetries = arcaflow_plugin.run(
-                                scenarios_list, kubeconfig_path, telemetry
+                                scenarios_list, kubeconfig_path, telemetry_k8s
                            )
                            chaos_telemetry.scenarios.extend(scenario_telemetries)

@@ -230,10 +270,10 @@ def main(cfg):
                                kraken_config,
                                failed_post_scenarios,
                                wait_duration,
-                                telemetry
+                                telemetry_k8s
                            )
                            chaos_telemetry.scenarios.extend(scenario_telemetries)
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "container_scenarios":
                            logging.info("Running container scenarios")
                            failed_post_scenarios, scenario_telemetries = pod_scenarios.container_run(
@@ -243,18 +283,18 @@ def main(cfg):
                                failed_post_scenarios,
                                wait_duration,
                                kubecli,
-                                telemetry
+                                telemetry_k8s
                            )
                            chaos_telemetry.scenarios.extend(scenario_telemetries)

                        # Inject node chaos scenarios specified in the config
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "node_scenarios":
                            logging.info("Running node scenarios")
-                            failed_post_scenarios, scenario_telemetries = nodeaction.run(scenarios_list, config, wait_duration, kubecli, telemetry)
+                            failed_post_scenarios, scenario_telemetries = nodeaction.run(scenarios_list, config, wait_duration, kubecli, telemetry_k8s)
                            chaos_telemetry.scenarios.extend(scenario_telemetries)
                        # Inject managedcluster chaos scenarios specified in the config
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "managedcluster_scenarios":
                            logging.info("Running managedcluster scenarios")
                            managedcluster_scenarios.run(
@@ -263,11 +303,11 @@ def main(cfg):

                        # Inject time skew chaos scenarios specified
                        # in the config
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "time_scenarios":
                            if distribution == "openshift":
                                logging.info("Running time skew scenarios")
-                                failed_post_scenarios, scenario_telemetries = time_actions.run(scenarios_list, config, wait_duration, kubecli, telemetry)
+                                failed_post_scenarios, scenario_telemetries = time_actions.run(scenarios_list, config, wait_duration, kubecli, telemetry_k8s)
                                chaos_telemetry.scenarios.extend(scenario_telemetries)
                            else:
                                logging.error(
@@ -316,50 +356,50 @@ def main(cfg):
                                sys.exit(1)

                        # Inject cluster shutdown scenarios
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "cluster_shut_down_scenarios":
-                            failed_post_scenarios, scenario_telemetries = shut_down.run(scenarios_list, config, wait_duration, kubecli, telemetry)
+                            failed_post_scenarios, scenario_telemetries = shut_down.run(scenarios_list, config, wait_duration, kubecli, telemetry_k8s)
                            chaos_telemetry.scenarios.extend(scenario_telemetries)

                        # Inject namespace chaos scenarios
-                        # krkn_lib_kubernetes
-                        elif scenario_type == "namespace_scenarios":
-                            logging.info("Running namespace scenarios")
-                            failed_post_scenarios, scenario_telemetries = namespace_actions.run(
+                        # krkn_lib
+                        elif scenario_type == "service_disruption_scenarios":
+                            logging.info("Running service disruption scenarios")
+                            failed_post_scenarios, scenario_telemetries = service_disruption.run(
                                scenarios_list,
                                config,
                                wait_duration,
                                failed_post_scenarios,
                                kubeconfig_path,
                                kubecli,
-                                telemetry
+                                telemetry_k8s
                            )
                            chaos_telemetry.scenarios.extend(scenario_telemetries)

                        # Inject zone failures
                        elif scenario_type == "zone_outages":
                            logging.info("Inject zone outages")
-                            failed_post_scenarios, scenario_telemetries = zone_outages.run(scenarios_list, config, wait_duration, telemetry)
+                            failed_post_scenarios, scenario_telemetries = zone_outages.run(scenarios_list, config, wait_duration, telemetry_k8s)
                            chaos_telemetry.scenarios.extend(scenario_telemetries)
                        # Application outages
                        elif scenario_type == "application_outages":
                            logging.info("Injecting application outage")
                            failed_post_scenarios, scenario_telemetries = application_outage.run(
-                                scenarios_list, config, wait_duration, telemetry)
+                                scenarios_list, config, wait_duration, telemetry_k8s)
                            chaos_telemetry.scenarios.extend(scenario_telemetries)

                        # PVC scenarios
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "pvc_scenarios":
                            logging.info("Running PVC scenario")
-                            failed_post_scenarios, scenario_telemetries = pvc_scenario.run(scenarios_list, config, kubecli, telemetry)
+                            failed_post_scenarios, scenario_telemetries = pvc_scenario.run(scenarios_list, config, kubecli, telemetry_k8s)
                            chaos_telemetry.scenarios.extend(scenario_telemetries)

                        # Network scenarios
-                        # krkn_lib_kubernetes
+                        # krkn_lib
                        elif scenario_type == "network_chaos":
                            logging.info("Running Network Chaos")
-                            failed_post_scenarios, scenario_telemetries = network_chaos.run(scenarios_list, config, wait_duration, kubecli, telemetry)
+                            failed_post_scenarios, scenario_telemetries = network_chaos.run(scenarios_list, config, wait_duration, kubecli, telemetry_k8s)

                        # Check for critical alerts when enabled
                        if check_critical_alerts:
@@ -379,22 +419,43 @@ def main(cfg):
            logging.info("")

        # telemetry
+        # in order to print decoded telemetry data even if telemetry collection
+        # is disabled, it's necessary to serialize the ChaosRunTelemetry object
+        # to json, and recreate a new object from it.
+        end_time = int(time.time())
+
+        # if platform is openshift will be collected
+        # Cloud platform and network plugins metadata
+        # through OCP specific APIs
+        if config["kraken"]["distribution"] == "openshift":
+            telemetry_ocp.collect_cluster_metadata(chaos_telemetry)
+        else:
+            telemetry_k8s.collect_cluster_metadata(chaos_telemetry)
+
+        decoded_chaos_run_telemetry = ChaosRunTelemetry(json.loads(chaos_telemetry.to_json()))
+        logging.info(f"Telemetry data:\n{decoded_chaos_run_telemetry.to_json()}")
+
        if config["telemetry"]["enabled"]:
            logging.info(f"telemetry data will be stored on s3 bucket folder: {telemetry_request_id}")
            logging.info(f"telemetry upload log: {safe_logger.log_file_name}")
            try:
-                telemetry.send_telemetry(config["telemetry"], telemetry_request_id, chaos_telemetry)
-                safe_logger.info("archives download started:")
-                prometheus_archive_files = telemetry.get_ocp_prometheus_data(config["telemetry"], telemetry_request_id)
-                safe_logger.info("archives upload started:")
-                telemetry.put_ocp_prometheus_data(config["telemetry"], prometheus_archive_files, telemetry_request_id)
+                telemetry_k8s.send_telemetry(config["telemetry"], telemetry_request_id, chaos_telemetry)
+                telemetry_k8s.put_cluster_events(telemetry_request_id, config["telemetry"], start_time, end_time)
+                # prometheus data collection is available only on Openshift
+                if config["telemetry"]["prometheus_backup"] and config["kraken"]["distribution"] == "openshift":
+                    safe_logger.info("archives download started:")
+                    prometheus_archive_files = telemetry_ocp.get_ocp_prometheus_data(config["telemetry"], telemetry_request_id)
+                    safe_logger.info("archives upload started:")
+                    telemetry_k8s.put_prometheus_data(config["telemetry"], prometheus_archive_files, telemetry_request_id)
+                if config["telemetry"]["logs_backup"]:
+                    telemetry_ocp.put_ocp_logs(telemetry_request_id, config["telemetry"], start_time, end_time)
            except Exception as e:
                logging.error(f"failed to send telemetry data: {str(e)}")
        else:
            logging.info("telemetry collection disabled, skipping.")

        # Capture the end time
-        end_time = int(time.time())
+

        # Capture metrics for the run
        if capture_metrics:
@@ -427,7 +488,7 @@ def main(cfg):
            else:
                logging.error("Alert profile is not defined")
                sys.exit(1)
-  
+
        if litmus_uninstall and litmus_installed:
            common_litmus.delete_chaos(litmus_namespace, kubecli)
            common_litmus.delete_chaos_experiments(litmus_namespace, kubecli)
--- a/scenarios/arcaflow/cpu-hog/sub-workflow.yaml
+++ b/scenarios/arcaflow/cpu-hog/sub-workflow.yaml
@@ -1,8 +1,9 @@
+version: v0.1.0
 input:
  root: RootObject
  objects:
    RootObject:
-      id: RootObject
+      id: input_item
      properties:
        kubeconfig:
          display:
@@ -60,17 +61,17 @@ input:

 steps:
  kubeconfig:
-    plugin: quay.io/arcalot/arcaflow-plugin-kubeconfig:latest
+    plugin: quay.io/arcalot/arcaflow-plugin-kubeconfig:0.2.0
    input:
      kubeconfig: !expr $.input.kubeconfig
  stressng:
-    plugin: quay.io/arcalot/arcaflow-plugin-stressng:0.2.0
+    plugin: quay.io/arcalot/arcaflow-plugin-stressng:0.5.0
    step: workload
    input:
+      cleanup: "true"
      StressNGParams:
        timeout: !expr $.input.duration
-        cleanup: "true"
-        items:
+        stressors:
          - stressor: cpu
            cpu_count: !expr $.input.cpu_count
            cpu_method: !expr $.input.cpu_method
--- a/scenarios/arcaflow/cpu-hog/workflow.yaml
+++ b/scenarios/arcaflow/cpu-hog/workflow.yaml
@@ -1,3 +1,4 @@
+version: v0.1.0
 input:
  root: RootObject
  objects:
--- a/scenarios/arcaflow/io-hog/config.yaml
+++ b/scenarios/arcaflow/io-hog/config.yaml
@@ -0,0 +1,10 @@
+deployer:
+  connection: {}
+  type: kubernetes
+log:
+  level: debug
+logged_outputs:
+  error:
+    level: error
+  success:
+    level: debug
--- a/scenarios/arcaflow/io-hog/input.yaml
+++ b/scenarios/arcaflow/io-hog/input.yaml
@@ -0,0 +1,13 @@
+input_list:
+- duration: 30s
+  io_block_size: 1m
+  io_workers: 1
+  io_write_bytes: 10m
+  kubeconfig: ''
+  namespace: default
+  node_selector: {}
+  target_pod_folder: /hog-data
+  target_pod_volume:
+    hostPath:
+      path: /tmp
+    name: node-volume
--- a/scenarios/arcaflow/io-hog/sub-workflow.yaml
+++ b/scenarios/arcaflow/io-hog/sub-workflow.yaml
@@ -0,0 +1,139 @@
+version: v0.1.0
+input:
+  root: RootObject
+  objects:
+    RootObject:
+      id: input_item
+      properties:
+        kubeconfig:
+          display:
+            description: The complete kubeconfig file as a string
+            name: Kubeconfig file contents
+          type:
+            type_id: string
+          required: true
+        namespace:
+          display:
+            description: The namespace where the container will be deployed
+            name: Namespace
+          type:
+            type_id: string
+          required: true
+        node_selector:
+            display:
+              description: kubernetes node name where the plugin must be deployed
+            type:
+              type_id: map
+              values:
+                type_id: string
+              keys:
+                type_id: string
+            required: true
+        duration:
+          display:
+            name: duration the scenario expressed in seconds
+            description: stop  stress  test  after  T  seconds.  One  can  also specify the units of time in
+              seconds, minutes, hours, days or years with the suffix s, m, h, d or  y
+          type:
+            type_id: string
+          required: true
+        io_workers:
+          display:
+            description: number of workers
+            name: start N workers continually writing, reading  and  removing  temporary  files
+          type:
+            type_id: integer
+          required: true
+        io_block_size:
+            display:
+              description: single write size
+              name: specify size of each write in bytes. Size can be from 1 byte to 4MB.
+            type:
+              type_id: string
+            required: true
+        io_write_bytes:
+          display:
+            description: Total number of bytes written
+            name: write  N  bytes for each hdd process, the default is 1 GB. One can specify the size
+              as % of free space on the file system or in units  of  Bytes,  KBytes,  MBytes  and
+              GBytes using the suffix b, k, m or g
+          type:
+            type_id: string
+          required: true
+        target_pod_folder:
+          display:
+            description: Target Folder
+            name: Folder in the pod where the test will be executed and the test files will be written
+          type:
+            type_id: string
+          required: true
+        target_pod_volume:
+          display:
+            name: kubernetes volume definition
+            description: the volume that will be attached to the pod. In order to stress
+                         the node storage only hosPath mode is currently supported
+          type:
+            type_id: object
+            id: k8s_volume
+            properties:
+              name:
+                display:
+                  description: name of the volume (must match the name in pod definition)
+                type:
+                  type_id: string
+                required: true
+              hostPath:
+                  display:
+                    description: hostPath options expressed as string map (key-value)
+                  type:
+                    type_id: map
+                    values:
+                      type_id: string
+                    keys:
+                      type_id: string
+                  required: true
+          required: true
+
+steps:
+  kubeconfig:
+    plugin: quay.io/arcalot/arcaflow-plugin-kubeconfig:0.2.0
+    input:
+      kubeconfig: !expr $.input.kubeconfig
+  stressng:
+    plugin: quay.io/arcalot/arcaflow-plugin-stressng:0.5.0
+    step: workload
+    input:
+      cleanup: "true"
+      StressNGParams:
+        timeout: !expr $.input.duration
+        workdir: !expr $.input.target_pod_folder
+        stressors:
+          - stressor: hdd
+            hdd: !expr $.input.io_workers
+            hdd_bytes: !expr $.input.io_write_bytes
+            hdd_write_size: !expr $.input.io_block_size
+
+    deploy:
+      type: kubernetes
+      connection: !expr $.steps.kubeconfig.outputs.success.connection
+      pod:
+        metadata:
+          namespace: !expr $.input.namespace
+          labels:
+            arcaflow: stressng
+        spec:
+          nodeSelector: !expr $.input.node_selector
+          pluginContainer:
+            imagePullPolicy: Always
+            securityContext:
+              privileged: true
+            volumeMounts:
+              - mountPath: /hog-data
+                name: node-volume
+          volumes:
+            - !expr $.input.target_pod_volume
+
+outputs:
+  success:
+    stressng: !expr $.steps.stressng.outputs.success
+
--- a/scenarios/arcaflow/io-hog/workflow.yaml
+++ b/scenarios/arcaflow/io-hog/workflow.yaml
@@ -0,0 +1,114 @@
+version: v0.1.0
+input:
+  root: RootObject
+  objects:
+    RootObject:
+      id: RootObject
+      properties:
+        input_list:
+          type:
+            type_id: list
+            items:
+              id: input_item
+              type_id: object
+              properties:
+                kubeconfig:
+                  display:
+                    description: The complete kubeconfig file as a string
+                    name: Kubeconfig file contents
+                  type:
+                    type_id: string
+                  required: true
+                namespace:
+                  display:
+                    description: The namespace where the container will be deployed
+                    name: Namespace
+                  type:
+                    type_id: string
+                  required: true
+                node_selector:
+                  display:
+                    description: kubernetes node name where the plugin must be deployed
+                  type:
+                    type_id: map
+                    values:
+                      type_id: string
+                    keys:
+                      type_id: string
+                  required: true
+                duration:
+                  display:
+                    name: duration the scenario expressed in seconds
+                    description: stop  stress  test  after  T  seconds.  One  can  also specify the units of time in
+                      seconds, minutes, hours, days or years with the suffix s, m, h, d or  y
+                  type:
+                    type_id: string
+                  required: true
+                io_workers:
+                  display:
+                    description: number of workers
+                    name: start N workers continually writing, reading  and  removing  temporary  files
+                  type:
+                    type_id: integer
+                  required: true
+                io_block_size:
+                  display:
+                    description: single write size
+                    name: specify size of each write in bytes. Size can be from 1 byte to 4MB.
+                  type:
+                    type_id: string
+                  required: true
+                io_write_bytes:
+                  display:
+                    description: Total number of bytes written
+                    name: write  N  bytes for each hdd process, the default is 1 GB. One can specify the size
+                      as % of free space on the file system or in units  of  Bytes,  KBytes,  MBytes  and
+                      GBytes using the suffix b, k, m or g
+                  type:
+                    type_id: string
+                  required: true
+                target_pod_folder:
+                  display:
+                    description: Target Folder
+                    name: Folder in the pod where the test will be executed and the test files will be written
+                  type:
+                    type_id: string
+                  required: true
+                target_pod_volume:
+                  display:
+                    name: kubernetes volume definition
+                    description: the volume that will be attached to the pod. In order to stress
+                      the node storage only hosPath mode is currently supported
+                  type:
+                    type_id: object
+                    id: k8s_volume
+                    properties:
+                      name:
+                        display:
+                          description: name of the volume (must match the name in pod definition)
+                        type:
+                          type_id: string
+                        required: true
+                      hostPath:
+                        display:
+                          description: hostPath options expressed as string map (key-value)
+                        type:
+                          type_id: map
+                          values:
+                            type_id: string
+                          keys:
+                            type_id: string
+                        required: true
+                  required: true
+steps:
+  workload_loop:
+    kind: foreach
+    items: !expr $.input.input_list
+    workflow: sub-workflow.yaml
+    parallelism: 1000
+outputs:
+  success:
+    workloads: !expr $.steps.workload_loop.outputs.success.data
+
+
+
--- a/scenarios/arcaflow/memory-hog/sub-workflow.yaml
+++ b/scenarios/arcaflow/memory-hog/sub-workflow.yaml
@@ -1,8 +1,9 @@
+version: v0.1.0
 input:
  root: RootObject
  objects:
    RootObject:
-      id: RootObject
+      id: input_item
      properties:
        kubeconfig:
          display:
@@ -52,17 +53,17 @@ input:

 steps:
  kubeconfig:
-    plugin: quay.io/arcalot/arcaflow-plugin-kubeconfig:latest
+    plugin: quay.io/arcalot/arcaflow-plugin-kubeconfig:0.2.0
    input:
      kubeconfig: !expr $.input.kubeconfig
  stressng:
-    plugin: quay.io/arcalot/arcaflow-plugin-stressng:0.2.0
+    plugin: quay.io/arcalot/arcaflow-plugin-stressng:0.5.0
    step: workload
    input:
+      cleanup: "true"
      StressNGParams:
        timeout: !expr $.input.duration
-        cleanup: "true"
-        items:
+        stressors:
          - stressor: vm
            vm: !expr $.input.vm_workers
            vm_bytes: !expr $.input.vm_bytes
--- a/scenarios/arcaflow/memory-hog/workflow.yaml
+++ b/scenarios/arcaflow/memory-hog/workflow.yaml
@@ -1,3 +1,4 @@
+version: v0.1.0
 input:
  root: RootObject
  objects:
--- a/scenarios/openshift/container_etcd.yml
+++ b/scenarios/openshift/container_etcd.yml
@@ -3,6 +3,6 @@ scenarios:
  namespace: "openshift-etcd"
  label_selector: "k8s-app=etcd"
  container_name: "etcd"
-  action: "kill 1"
+  action: 1
  count: 1
  expected_recovery_time: 60
--- a/scenarios/openshift/prom_kill.yml
+++ b/scenarios/openshift/prom_kill.yml
@@ -0,0 +1,9 @@
+- id: kill-pods
+  config:
+    namespace_pattern: ^openshift-monitoring$
+    label_selector: statefulset.kubernetes.io/pod-name=prometheus-k8s-0
+- id: wait-for-pods
+  config:
+    namespace_pattern: ^openshift-monitoring$
+    label_selector: statefulset.kubernetes.io/pod-name=prometheus-k8s-0
+    count: 1
--- a/utils/chaos_recommender/README.md
+++ b/utils/chaos_recommender/README.md
@@ -0,0 +1,106 @@
+# Chaos Recommendation Tool
+
+This tool, designed for Redhat Kraken, operates through the command line and offers recommendations for chaos testing. It suggests probable chaos test cases that can disrupt application services by analyzing their behavior and assessing their susceptibility to specific fault types.
+
+This tool profiles an application and gathers telemetry data such as CPU, Memory, and Network usage, analyzing it to suggest probable chaos scenarios. For optimal results, it is recommended to activate the utility while the application is under load.
+
+## Pre-requisites
+
+- Openshift Or Kubernetes Environment where the application is hosted
+- Access to the telemetry data via the exposed Prometheus endpoint
+- Python3
+
+## Usage
+
+1. To run
+
+    ```
+    $ python3 -m venv chaos
+    $ source chaos/bin/activate
+    $ git clone https://github.com/redhat-chaos/krkn.git 
+    $ cd krkn
+    $ pip3 install -r requirements.txt
+    $ python3 utils/chaos_recommender/chaos_recommender.py
+    ```
+
+2. Follow the prompts to provide the required information.
+
+## Configuration
+To run the recommender with a config file specify the config file path with the `-c` argument.
+You can customize the default values by editing the `krkn/config/recommender_config.yaml` file. The configuration file contains the following options:
+
+  - `application`: Specify the application name.
+  - `namespace`: Specify the namespace name. If you want to profile
+  - `labels`: Specify the labels (not used).
+  - `kubeconfig`: Specify the location of the kubeconfig file (not used).
+  - `prometheus_endpoint`: Specify the prometheus endpoint (must).
+  - `auth_token`: Auth token to connect to prometheus endpoint (must).
+  - `scrape_duration`: For how long data should be fetched, e.g., '1m' (must).
+  - `chaos_library`: "kraken" (currently it only supports kraken).
+  - `chaos_tests`: (for output purpose only do not change if not needed)
+    - `GENERAL`: list of general purpose tests available in Krkn
+    - `MEM`: list of memory related tests available in Krkn
+    - `NETWORK`: list of network related tests available in Krkn
+    - `CPU`: list of memory related tests available in Krkn
+
+*TIP:* to collect prometheus endpoint and token from your OpenShift cluster you can run the following commands:
+        ```
+         prometheus_url=$(kubectl get routes -n openshift-monitoring prometheus-k8s --no-headers | awk '{print $2}')
+         #TO USE YOUR CURRENT SESSION TOKEN
+         token=$(oc whoami -t)
+         #TO CREATE A NEW TOKEN
+         token=$(kubectl create token -n openshift-monitoring prometheus-k8s --duration=6h || oc sa new-token -n openshift-monitoring prometheus-k8s)
+        ```
+
+You can also provide the input values through command-line arguments launching the recommender with `-o` option:
+
+```
+  -o, --options         Evaluate command line options
+  -a APPLICATION, --application APPLICATION
+                        Kubernetes application name
+  -n NAMESPACE, --namespace NAMESPACE
+                        Kubernetes application namespace
+  -l LABELS, --labels LABELS
+                        Kubernetes application labels
+  -p PROMETHEUS_ENDPOINT, --prometheus-endpoint PROMETHEUS_ENDPOINT
+                        Prometheus endpoint URI
+  -k KUBECONFIG, --kubeconfig KUBECONFIG
+                        Kubeconfig path
+  -t TOKEN, --token TOKEN
+                        Kubernetes authentication token
+  -s SCRAPE_DURATION, --scrape-duration SCRAPE_DURATION
+                        Prometheus scrape duration
+  -i LIBRARY, --library LIBRARY
+                        Chaos library
+  -L LOG_LEVEL, --log-level LOG_LEVEL
+                        log level (DEBUG, INFO, WARNING, ERROR, CRITICAL
+  -M MEM [MEM ...], --MEM MEM [MEM ...]
+                        Memory related chaos tests (space separated list)
+  -C CPU [CPU ...], --CPU CPU [CPU ...]
+                        CPU related chaos tests (space separated list)
+  -N NETWORK [NETWORK ...], --NETWORK NETWORK [NETWORK ...]
+                        Network related chaos tests (space separated list)
+  -G GENERIC [GENERIC ...], --GENERIC GENERIC [GENERIC ...]
+                        Memory related chaos tests (space separated list)
+
+```
+
+If you provide the input values through command-line arguments, the corresponding config file inputs would be ignored.
+
+## Podman & Docker image
+
+To run the recommender image please visit the [krkn-hub](https://github.com/redhat-chaos/krkn-hub for further infos.
+
+## How it works
+
+After obtaining telemetry data, sourced either locally or from Prometheus, the tool conducts a comprehensive data analysis to detect anomalies. Employing the Z-score method and heatmaps, it identifies outliers by evaluating CPU, memory, and network usage against established limits. Services with Z-scores surpassing a specified threshold are categorized as outliers. This categorization classifies services as network, CPU, or memory-sensitive, consequently leading to the recommendation of relevant test cases.
+
+## Customizing Thresholds and Options
+
+You can customize the thresholds and options used for data analysis by modifying the `krkn/kraken/chaos_recommender/analysis.py` file. For example, you can adjust the threshold for identifying outliers by changing the value of the `threshold` variable in the `identify_outliers` function.
+
+## Additional Files
+
+- `config/recommender_config.yaml`: The configuration file containing default values for application, namespace, labels, and kubeconfig.
+
+Happy Chaos!
--- a/utils/chaos_recommender/chaos_recommender.py
+++ b/utils/chaos_recommender/chaos_recommender.py
@@ -0,0 +1,117 @@
+import argparse
+import logging
+import os.path
+import sys
+import yaml
+# kraken module import for running the recommender
+# both from the root directory and the recommender
+# folder
+sys.path.insert(0, './')
+sys.path.insert(0, '../../')
+
+import kraken.chaos_recommender.analysis as analysis
+import kraken.chaos_recommender.prometheus as prometheus
+from kubernetes import config as kube_config
+
+
+
+def parse_arguments(parser):
+
+    # command line options
+    parser.add_argument("-c", "--config-file", action="store", help="Config file path")
+    parser.add_argument("-o", "--options", action="store_true", help="Evaluate command line options")
+    parser.add_argument("-n", "--namespace", action="store", default="", help="Kubernetes application namespace")
+    parser.add_argument("-p", "--prometheus-endpoint", action="store", default="", help="Prometheus endpoint URI")
+    parser.add_argument("-k", "--kubeconfig", action="store", default=kube_config.KUBE_CONFIG_DEFAULT_LOCATION, help="Kubeconfig path")
+    parser.add_argument("-t", "--token", action="store", default="", help="Kubernetes authentication token")
+    parser.add_argument("-s", "--scrape-duration", action="store", default="10m", help="Prometheus scrape duration")
+    parser.add_argument("-L", "--log-level", action="store", default="INFO", help="log level (DEBUG, INFO, WARNING, ERROR, CRITICAL")
+
+    parser.add_argument("-M", "--MEM", nargs='+', action="store", default=[],
+                        help="Memory related chaos tests (space separated list)")
+    parser.add_argument("-C", "--CPU", nargs='+', action="store", default=[],
+                        help="CPU related chaos tests (space separated list)")
+    parser.add_argument("-N", "--NETWORK", nargs='+', action="store", default=[],
+                        help="Network related chaos tests (space separated list)")
+    parser.add_argument("-G", "--GENERIC", nargs='+', action="store", default=[],
+                        help="Memory related chaos tests (space separated list)")
+
+
+    return parser.parse_args()
+
+def read_configuration(config_file_path):
+    if not os.path.exists(config_file_path):
+        logging.error(f"Config file not found: {config_file_path}")
+        sys.exit(1)
+
+    with open(config_file_path, mode="r") as config_file:
+        config = yaml.safe_load(config_file)
+
+    log_level = config.get("log level", "INFO")
+    namespace = config.get("namespace", "")
+    kubeconfig = config.get("kubeconfig", kube_config.KUBE_CONFIG_DEFAULT_LOCATION)
+
+    prometheus_endpoint = config.get("prometheus_endpoint", "")
+    auth_token = config.get("auth_token", "")
+    scrape_duration = config.get("scrape_duration", "10m")
+    chaos_tests = config.get("chaos_tests" , {})
+    return (namespace, kubeconfig, prometheus_endpoint, auth_token, scrape_duration,
+            chaos_tests, log_level)
+
+def prompt_input(prompt, default_value):
+    user_input = input(f"{prompt} [{default_value}]: ")
+    if user_input.strip():
+        return user_input
+    return default_value
+
+def main():
+    parser = argparse.ArgumentParser(description="Krkn Chaos Recommender Command-Line tool")
+    args = parse_arguments(parser)
+
+    if args.config_file is None and not args.options:
+        logging.error("You have to either specify a config file path or pass recommender options as command line arguments")
+        parser.print_help()
+        sys.exit(1)
+
+    if args.config_file is not None:
+        (
+         namespace,
+         kubeconfig,
+         prometheus_endpoint,
+         auth_token,
+         scrape_duration,
+         chaos_tests,
+         log_level
+         ) = read_configuration(args.config_file)
+
+    if args.options:
+        namespace = args.namespace
+        kubeconfig = args.kubeconfig
+        auth_token = args.token
+        scrape_duration = args.scrape_duration
+        log_level = args.log_level
+        prometheus_endpoint = args.prometheus_endpoint
+        chaos_tests = {"MEM": args.MEM, "GENERIC": args.GENERIC, "CPU": args.CPU, "NETWORK": args.NETWORK}
+
+    if log_level not in ["DEBUG","INFO", "WARNING", "ERROR","CRITICAL"]:
+        logging.error(f"{log_level} not a valid log level")
+        sys.exit(1)
+
+    logging.basicConfig(level=log_level)
+
+    logging.info("============================INPUTS===================================")
+    logging.info(f"Namespace: {namespace}")
+    logging.info(f"Kubeconfig: {kubeconfig}")
+    logging.info(f"Prometheus endpoint: {prometheus_endpoint}")
+    logging.info(f"Scrape duration: {scrape_duration}")
+    for test in chaos_tests.keys():
+        logging.info(f"Chaos tests {test}: {chaos_tests[test]}")
+    logging.info("=====================================================================")
+    logging.info("Starting Analysis ...")
+    logging.info("Fetching the Telemetry data")
+
+    file_path = prometheus.fetch_utilization_from_prometheus(prometheus_endpoint, auth_token, namespace, scrape_duration)
+    analysis(file_path, chaos_tests)
+
+if __name__ == "__main__":
+    main()
Author	SHA1	Message	Date
Tullio Sebastiani	7a966a71d0	krkn integration of telemetry events collection (#523 ) * function package refactoring in krkn-lib * cluster events collection flag * krkn-lib version bump requirements * dockerfile bump	2023-10-31 14:31:33 -04:00
Naga Ravi Chaitanya Elluri	43d891afd3	Bump telemetry archive default size to 500MB This commit also removes litmus configs as they are not maintained.	2023-10-30 12:50:04 -04:00
Tullio Sebastiani	27fabfd4af	OCP/K8S functionalities and packages splitting in `krkn-lib` (#507 ) * krkn-lib ocp/k8s split adaptation * library reference updated * requirements update * rebase with main + fix	2023-10-30 17:31:48 +01:00
Tullio Sebastiani	724068a978	Chaos recommender refactoring (#516 ) * basic structure working * config and options refactoring nits and changes * removed unused function with typo + fixed duration * removed unused arguments * minor fixes	2023-10-30 15:51:09 +01:00
Tullio Sebastiani	c9778474f1	arcaflow version bump (#520 ) arcaflow version bump stressng version typo	2023-10-27 18:09:46 +02:00
dependabot[bot]	6efdb2eb84	Bump werkzeug from 2.2.3 to 3.0.1 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.3 to 3.0.1. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/2.2.3...3.0.1) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>	2023-10-26 11:09:29 -04:00
Naga Ravi Chaitanya Elluri	0e852da7d4	Deprecate kubernetes method of deploying Krkn This will ensure users will use the recommended methods ( standlone or containerized ) of installing and running Krkn.	2023-10-25 12:32:46 -04:00
jtydlack	86d1fda325	Fix container scenario to accept only signal number (#350 ) (#485 )	2023-10-24 16:51:48 -04:00
Naga Ravi Chaitanya Elluri	fc6344176b	Add pointer to the CNCF sandbox discussion (#517 ) Signed-off-by: Naga Ravi Chaitanya Elluri <nelluri@redhat.com>	2023-10-24 16:07:40 -04:00
jtydlack	ff469579e9	Use function get_yaml_item_value Enables using default even though the value was loaded as None.	2023-10-24 14:55:49 -04:00
Naga Ravi Chaitanya Elluri	8cbd1c5e7f	Add docs for installing chaos-recommender dependencies This commit also updates roadmap around chaos-recommender.	2023-10-18 08:56:33 -04:00
Mudit Verma	5953e53b46	chaos recommendation entry in README (#510 )	2023-10-16 11:26:32 -04:00
Mudit Verma	23f1fc044b	Chaos Recommendation Utility (#508 ) * application profiling based chaos recommendation * deleted unused dir * Update requirements.txt Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update config.ini Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update Makefile Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update Dockerfile Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> * Update README.md Signed-off-by: Mudit Verma <mudiverm@in.ibm.com> --------- Signed-off-by: Mudit Verma <mudiverm@in.ibm.com>	2023-10-16 10:06:02 -04:00
Naga Ravi Chaitanya Elluri	69e386db53	Update roadmap with upcoming integrations and enhancements	2023-10-11 09:24:34 -04:00
Tullio Sebastiani	fef77cfc0e	dockerfiles version update	2023-10-09 09:26:57 -04:00
Naga Ravi Chaitanya Elluri	eb2eabe029	Update community slack channel	2023-10-06 17:47:25 -04:00
Paige Rubendall	f7f1b2dfb0	Service disruption (#494 ) * adding service disruption * fixing kil services * service log changes * remvoing extra logging * adding daemon set * adding service disruption name changes * cerberus config back * bad string	2023-10-06 12:51:10 -04:00
Tullio Sebastiani	61356fd70b	Added log telemetry piece to Krkn (#500 ) * config * log collection and upload dictionary key fix * escape regex in config.yaml * bump krkn-lib version * updated funtest github cli command * update krkn-lib version to 1.3.2 * fixed requirements.txt	2023-10-06 10:08:46 -04:00
Tullio Sebastiani	067969a81a	docker version update (#502 )	2023-10-03 11:49:20 -04:00
Naga Ravi Chaitanya Elluri	972ac12921	Bump krkn-lib version (#499 )	2023-10-03 17:00:07 +02:00
Tullio Sebastiani	ea813748ae	added OCP/K8S functionalities split in the roadmap (#498 )	2023-09-26 00:59:20 -04:00
Tullio Sebastiani	782d04c1b1	Prints the telemetry json after sending it to the webservice (#479 ) * prints telemetry json after sending it to the service deserialized base64 parameters * json output even if telemetry collection is disabled.	2023-09-25 12:00:08 -04:00
Naga Ravi Chaitanya Elluri	2fb58f9897	Update roadmap with completed and new items	2023-09-21 15:33:44 -04:00
Tullio Sebastiani	5712721410	bumped docker version (#493 ) Co-authored-by: Naga Ravi Chaitanya Elluri <nelluri@redhat.com>	2023-09-19 17:38:44 +02:00
Tullio Sebastiani	5567c06cd0	reinstated io-hog documentation (#492 )	2023-09-19 17:27:59 +02:00
Sahil Shah	0ad4c11356	Fix for time scenario (#490 )	2023-09-14 12:36:08 -04:00
Tullio Sebastiani	f6f686e8fe	fixed io-hog scenario	2023-09-13 09:57:00 -04:00
Pratyusha Thammineni	3a66f8a5a3	Added Docker image build workflow status badge This Allows the users to track the docker-build action in README.md without navigationg to Actions tab on Github	2023-09-11 15:16:28 -04:00
Sahil Shah	585d519687	Adding Prometheus Disruption Scenario (#484 )	2023-09-11 11:18:29 -04:00
yogananth-subramanian	e40fedcd44	Update etcd metrics	2023-09-08 11:11:42 -04:00
Paige Rubendall	1bb5b8ad04	adding comment	2023-08-29 21:54:17 -04:00
Paige Rubendall	725d58c8ce	adding docs update again	2023-08-25 14:37:07 -04:00
Paige Rubendall	c6058da7a7	adding comment	2023-08-25 12:19:03 -04:00
Naga Ravi Chaitanya Elluri	06a8ed220c	Bump release version to v1.4.4	2023-08-24 13:28:39 -04:00
Dustin Black	2c6b50bcdc	bump arcaflow stressng plugin to 0.3.1 for bug fix	2023-08-24 12:50:28 -04:00
Naga Ravi Chaitanya Elluri	ed97c8df2b	Bump release version to v1.4.3	2023-08-23 11:56:39 -04:00
Tullio Sebastiani	1baa68bcee	engine bump to v0.6.1	2023-08-23 11:38:23 -04:00
Naga Ravi Chaitanya Elluri	ab84f09448	Use release tags vs latest for kubeconfig arca plugins (#473 )	2023-08-23 09:59:33 -04:00
Dustin Black	6ace3c952b	update to plugin release stressng:0.3.0 (#472 )	2023-08-23 09:15:30 -04:00
Tullio Sebastiani	cee5259fd3	arcaflow scenarios removed from config.yaml	2023-08-23 08:50:19 -04:00
Tullio Sebastiani	f868000ebd	Switched from krkn_lib_kubernetes to krkn_lib v1.0.0 (#469 ) * changed all the references to krkn_lib_kubernetes to the new krkn_lib changed all the references * added krkn-lib pointer in documentation	2023-08-22 12:41:40 -04:00
pratyusha	d2d80be241	Updated config.yaml file with more scenarios (#468 )	2023-08-21 11:26:33 -04:00
Naga Ravi Chaitanya Elluri	da464859c4	Bump release version to v1.4.2	2023-08-21 09:06:28 -04:00