* feat: parallelize SLO PromQL queries with ThreadPoolExecutor (#1560)
Use concurrent.futures.ThreadPoolExecutor to run independent SLO range
queries in parallel, reducing per-pass evaluation from ~11-12s to ~2-3s.
Includes configurable max_workers (default 10, capped to SLO count),
per-future error isolation for malformed SLO dicts, and comprehensive
unit tests for parallelism, mixed results, and exception isolation.
Signed-off-by: Darshan Jain <ddjain@redhat.com>
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: prevent malformed SLOs from silently inflating resiliency score
- Move slo["expr"]/slo["name"] access inside try block so KeyError is
caught and re-raised as ValueError with context
- Write failure result in the outer future.result() handler so malformed
SLOs appear in the results mapping instead of being dropped
- Treat SLOs missing from prometheus_results as failed (score.py) rather
than excluding them, which previously inflated the score
- Clamp max_workers to minimum of 1 to prevent ValueError from
ThreadPoolExecutor when max_workers <= 0
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: Darshan Jain <ddjain@redhat.com>
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
the managed cluster scenarios have been loading manifestwork.j2 at
runtime since the plugin was moved into scenario_plugins, but the file
itself never made it into the new layout. every scenario call crashed
with a TemplateNotFound error before the manifestwork was created.
the template was found in the git history of the old kraken layout and
restored, with the invalid namespace dropped from the ClusterRole
(ClusterRoles are cluster-scoped, a namespace on one fails to apply and
breaks the rbac setup for the job). added a test that checks the file
is present and that it renders into a valid ManifestWork document with
the variables the scenarios pass in.
Signed-off-by: Sahil Lenka <sahillenka44@gmail.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
* fix: skip unnecessary wait_duration sleep after the last scenario
The wait_duration sleep was running unconditionally after every scenario,
including the final one. This wasted up to 60s (default) on single-scenario
runs where there is no next scenario to wait for.
Closes#1556
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: use wait_duration as soak window for SLO evaluation
Move end_timestamp capture to after the wait_duration sleep so the
resiliency score / SLO evaluation window includes the settling period.
This ensures delayed impacts during cooldown are captured in metrics.
The sleep now runs for every scenario (including the last) since it
serves as a measurement soak window, not just an inter-scenario delay.
Closes krkn-chaos#1556
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add force pod deletion option to pod disruption scenario
Add a `force` boolean config option (default: false) that controls
whether pods are killed gracefully or forcefully. When force is true,
grace_period_seconds=0 is passed to delete_pod(), causing immediate
termination without waiting for the pod's terminationGracePeriodSeconds.
Works with both serial and parallel execution modes.
Depends on: krkn-lib feat/force-pod-delete branch
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address PR review - validate force config, bump krkn-lib, add tests
- Add type validation for 'force' config to reject non-boolean values
- Bump krkn-lib to 6.1.3 (includes grace_period_seconds in delete_pod)
- Add unit tests for force deletion in serial and parallel modes
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(pod-disruption): support parallel pod deletion
Introduce 'kill_mode' field to pod disruption scenario to support parallel
deletion of pods. By default 'kill_mode' is 'sequential' preserving
existing behavior. When set to 'parallel', it concurrently deletes pods
using threading and queue, enabling effective testing of disruption scenarios
like etcd quorum loss where simultaneous disruption is necessary.
Resolves: #1516
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* fix(pod-disruption): validate kill_mode and address PR feedback
- Validate 'kill_mode' at parse time: fallback to 'sequential' and log a
warning if an unknown string is provided, preventing silent failures.
- Update test_parallel_mode_calls_delete_concurrently to use a
threading.Barrier to definitively prove concurrent thread execution.
- Revert 'scenarios/openshift/etcd.yml' to default non-destructive behavior
and extract parallel quorum loss example into a new explicitly named file
'etcd_quorum_loss.yml'.
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* refactor(pod-disruption): optimize imports and cap threads
- Reorganize imports in pod_disruption_scenario_plugin.py to comply with
standard PEP-8 grouping (stdlib, third-party, local).
- Replace unbounded OS thread spawning in _delete_pods_parallel with
concurrent.futures.ThreadPoolExecutor. Capped max_workers to 10 to
prevent excessive concurrent API calls that could overload the Kubernetes
API server under aggressive scenario configurations.
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* refactor(pod-disruption): align execution config with network_chaos_ng
- Renamed config field from 'kill_mode' to 'execution' for consistency
with the network_chaos_ng plugin.
- Changed valid values from 'sequential|parallel' to 'serial|parallel'.
- Replaced fallback warning with strict validation, raising ValueError on
invalid execution values, aligning with the strict validation pattern
used elsewhere in the codebase.
- Updated relevant tests and scenario YAML templates.
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* fix(pod-disruption): remove unused logging import in models.py
Following the switch from a warning log to raising a ValueError for
invalid execution types, the logging import was no longer used.
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: swagatobauri <swagato731123@gmail.com>
---------
Signed-off-by: swagatobauri <swagato731123@gmail.com>
Co-authored-by: Darshan Jain <darjain@redhat.com>
* feat: add k8s trigger type for event-driven chaos
Closes#1496
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
* fix: address review comments on k8s trigger
- Validate namespace against resource_api.namespaced to fail fast
when a namespaced resource is missing the namespace config
- Use numeric equality first in _compare for == and != operators
so that 1.0 == 1 works consistently with >= and <=
- Fix krknctl-input.json description for apiVersion field
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
* feat: add optional context field to k8s trigger
Allows selecting a specific kubeconfig context for cross-cluster
triggers. If omitted, uses the default context as before.
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
* fix: use kubecli.dyn_client instead of raw DynamicClient and catch ValueError
- K8sTrigger now receives kubecli from TriggerManager and uses
kubecli.dyn_client for centralized config and proxy support
- TriggerManager passes kubecli through _build_trigger to K8sTrigger
- run_kraken.py passes kubecli to TriggerManager
- Added ValueError to the except clause in evaluate() to handle
non-numeric comparisons and namespace validation errors gracefully
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
* docs: add k8s trigger example config
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
* fix: make kubecli required and remove unused context field from k8s trigger
Address remaining review comments on PR #1515:
- Make kubecli a required argument with early validation instead of
silently failing when None
- Remove _context field since _get_client() delegates to kubecli.dyn_client
which is already initialized with the correct context
- Remove TRIGGER_K8S_CONTEXT from krknctl-input.json
- Update tests to pass mock kubecli and cover missing-kubecli validation
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
---------
Signed-off-by: Saurabh Wattamwar <swattamw@redhat.com>
* protect config.telemetry access with fallback to empty dict
If the YAML config does not include a telemetry section, three
spots in main() crash with KeyError because they access
config.telemetry directly. Everything else in the file uses
get_yaml_item_value which handles missing keys gracefully.
Added config.telemetry = get_yaml_item_value(config, telemetry, {})
before the first usage. This ensures the key always exists as at
least an empty dict, protecting all downstream .get() calls and
assignments (archive_path, run_tag, telemetry_group, etc.).
Wrote a test that passes a config without a telemetry section and
asserts main() returns -1 (no kubeconfig) instead of crashing
with KeyError.
Signed-off-by: Sahil Lenka <sahillenka44@gmail.com>
* clean up test imports and prevent filesystem side effects
Remove unused imports (Mock, get_yaml_item_value). Add
rollback_versions_directory to mock config so the test
does not write ~/.krkn/rollback to disk.
Signed-off-by: Sahil Lenka <sahillenka44@gmail.com>
---------
Signed-off-by: Sahil Lenka <sahillenka44@gmail.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
ThreadPool.close() was placed after starmap/map but outside a
finally block. If the pool operation throws, close() never runs
and worker threads hang around until process exit.
Moved pool.close() and added pool.join() into a finally block.
Changed logging.info to logging.error in the exception handler
so pool failures are visible at the default log level.
Wrote two tests before touching code. The first verifies that
close() and join() are called even when map raises. The second
verifies that logging.error is used instead of logging.info.
Signed-off-by: Sahil Lenka <sahillenka44@gmail.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
* feat: add http trigger type for event-driven chaos
Adds an http trigger type so krkn can wait for an HTTP endpoint to return
an expected status before chaos injection. Completes the trigger types
on top of the Phase 1 framework (#1484).
- krkn/scenario_plugins/triggers/http_trigger.py: implements HttpTrigger
- krkn/scenario_plugins/triggers/trigger_manager.py: registers "http" type
- krkn/scenario_plugins/triggers/__init__.py: exports HttpTrigger
- tests/test_triggers/test_http_trigger.py: 30 tests for HttpTrigger
- scenarios/kube/trigger_http_example.yaml: example configuration
Closeskrkn-chaos/krkn#1498
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* fix(triggers): address code review feedback for http trigger
- Add type validation for body_contains in http_trigger.py
- Use mock_info.assert_not_called() for strict logging assertions in tests
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* feat(krknctl): add http trigger arguments to krknctl input fields
Wire HTTP triggers into krknctl-input.json so they are exposed as CLI flags
by krknctl.
Related to krkn-chaos/krkn#1498
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* test(triggers): add factory wiring and invalid body_contains tests
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* fix(triggers): move trigger_http_example.yaml to config directory per review
Signed-off-by: swagatobauri <swagato731123@gmail.com>
* fix(triggers): move HttpTrigger import to top of file per review
Signed-off-by: swagatobauri <swagato731123@gmail.com>
---------
Signed-off-by: swagatobauri <swagato731123@gmail.com>
Co-authored-by: Darshan Jain <darjain@redhat.com>
* fix: build tc netem tree with actual loss/delay values instead of changing after
The egress path in node-network-chaos built the tc htb+netem tree with
hardcoded loss 0% and delay 0ms, then tried to update via tc qdisc change.
On some hardware (bare-metal with Mellanox NIC), the change command fails
silently — no packets are ever dropped, producing false-negative tests.
Build the tree with actual rate/delay/loss values in a single pass,
matching how the ingress path already works. This eliminates the
tc qdisc change step entirely. Also log stderr warnings from tc commands
that were previously swallowed silently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: ddjain <darjain@redhat.com>
* refactor: remove unused get_egress_shaping_comand after single-pass tc tree build
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add event-driven chaos triggers with command trigger (Phase 1)
Adds a pluggable trigger system that gates chaos injection on
user-defined preconditions. A top-level `triggers` block in
config.yaml is evaluated before the chaos loop, with configurable
polling mode (all_of/any_of), timeout, interval, and on_timeout
behavior (skip/fail/run_anyway).
Phase 1 implements the command trigger type which runs a shell
command and checks its exit code.
- AbstractTrigger base class for pluggable trigger types
- CommandTrigger with input validation (expected_rc int coercion)
- TriggerManager with positive-number validation for timeout/interval
- Trigger evaluation runs before health check plugins start
- Broad exception handling in evaluate() prevents UnboundLocalError
- Debug logging of command returncode/stderr for observability
- 33 unit tests covering all functionality and edge cases
Closes#1483
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: ddjain <darjain@redhat.com>
* fix: address review feedback on trigger validation
- Validate conditions is a list, not a string
- Expose on_timeout as a property, use it in run_kraken.py
instead of re-reading raw config
- Validate expected_rc is in 0-255 Unix range
- Add 4 new tests (37 trigger tests total)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: ddjain <darjain@redhat.com>
---------
Signed-off-by: ddjain <darjain@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: validate container_name in container scenario plugin
The container scenario plugin removed pods and incremented killed_count
even when the requested container_name was never found, causing scenarios
to silently report success with no disruption (issue #1409).
Track whether a container was actually found and killed; only increment
killed_count on a real kill, skip pods that lack the target container,
and raise a clear RuntimeError once all pods are exhausted without a kill.
Adds unit tests covering invalid, valid, empty, heterogeneous, and
count-exceeds-target scenarios.
Closes#1409
* fix: report actual kill count in container-not-found error
When the candidate pod list is exhausted without finding the target
container, the RuntimeError now reports how many containers were
actually killed ("N of M requested container(s) were killed") instead
of always claiming "No containers were killed", which was inaccurate in
partial-success cases.
* fix: only raise container-not-found error when nothing was killed
Address review feedback: the "not found in any matching pod" error was
raised even after one or more containers had already been killed (when
count exceeds the number of pods containing the target), making the
message contradictory.
Now that error only fires when killed_count == 0. When some kills
already happened but the candidate list is exhausted, the loop falls
through to the existing "Trying to kill more containers than were found"
error, which accurately describes that case.
---------
Co-authored-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com>
Co-authored-by: Darshan Jain <darjain@redhat.com>
- Add exec_with_shell_fallback method with retry logic and shell fallback
- Add unit tests for the new method with proper mocking
- All tests now pass as expected
Signed-off-by: NITESH SINGH <niteshkumar121411@gmail.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
Allows selecting VMIs by label selector as an alternative to vm_name regex,
making vm_name optional when label_selector is provided.
Signed-off-by: Paige Patton <prubenda@redhat.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(cerberus): fix config key typo and shadowed global in get_status
Fixes#1302
- Fix typo in set_url(): read 'check_application_routes' instead of
'check_applicaton_routes' so the correct config key is used
- Remove local 'check_application_routes = False' in get_status() and
declare global instead, so the value set by set_url() is actually read
- Update test config key and remove the workaround comment that
acknowledged the shadowed global
Signed-off-by: netram75 <netram.24bcs10329@sst.scaler.com>
* fix(cerberus): remove extra cerberus_url arg from application_status call
application_status(start_time, end_time) takes two parameters but was
being called with three (cerberus_url, start_time, end_time), which
would raise a TypeError whenever check_application_routes is enabled.
Remove the stale argument since the function already reads cerberus_url
from the module global.
Signed-off-by: netram75 <netram.24bcs10329@sst.scaler.com>
---------
Signed-off-by: netram75 <netram.24bcs10329@sst.scaler.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
Add rollback functionality for GCP zone outage scenarios following the
established rollback pattern (Service Hijacking, PVC, Syn Flood).
- Add @set_rollback_context_decorator to run()
- Set rollback callable before stopping nodes with base64/JSON encoded data
- Add rollback_gcp_zone_outage() static method with per-node error handling
- Fix missing poll_interval argument in starmap calls
- Add unit tests for rollback and run methods
Closes#915
Signed-off-by: YASHASVIYADAV30 <yashasviydv30@gmail.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
- Fix typo 'wating' -> 'waiting' in scenario wait log message
- Replace print() with logging.debug() for pod metrics in prometheus client
- Replace star import with explicit imports in utils/__init__.py
- Remove unnecessary global declaration in main()
- Log VM status exceptions at ERROR level with exception details
Include unit tests in tests/test_logging_and_code_quality.py covering all fixes.
Signed-off-by: 1PoPTRoN <vrxn.arp1traj@gmail.com>
Co-authored-by: Paige Patton <64206430+paigerube14@users.noreply.github.com>
Darshan JainandCursor
Sahil LenkaandPaige Patton
reeeandqsxDree
SWAGATO BAURIandDarshan Jain
Teju Gangisetty
Saurabh Wattamwar
varun-ai69andPaige Patton
Shivansh Katiyar
Nitesh SinghandPaige Patton
Netram FaranandPaige Patton
Arpit RajandPaige Patton
Netram FaranandPaige Patton
Semih702andsemih702
Sahil Shah
Paige Patton
Tullio Sebastiani
Yashasvi YadavandPaige Patton