augmentcode[bot]andlnx01
5a532190aa
* test: attach krkn execution logs to HTML report and assert scenario executed CI/tests_v2 enhancements for observability and false-positive guarding: - run_kraken fixture stashes stdout/stderr/returncode on request.node so the report hook can attach them. - conftest pytest_runtest_makereport renders the krkn log as a timestamp/level/ message table via pytest_html.extras for every test (pass or fail). - conftest pytest_terminal_summary prints an execution-evidence table and writes a markdown summary to $GITHUB_STEP_SUMMARY when running in GitHub Actions. - utils adds SCENARIO_EXECUTION_MARKERS, an EXECUTION_EVIDENCE registry, and assert_scenario_executed() that fails happy-path tests when krkn exits 0 but no scenario-specific marker is present. Skipped under KRKN_TEST_DRY_RUN=1. - pod_disruption, storage_throttle, and application_outage happy-path tests now call assert_scenario_executed. Negative tests remain exempt. Closes #1394 * test: stash krkn logs on timeout and add log-path hint to evidence assert Address Deep Code Review feedback on PR #1395: - run_kraken now catches subprocess.TimeoutExpired, stashes partial stdout/stderr with a synthetic rc=124 so timed-out runs still attach logs to the HTML report. - assert_scenario_executed failure message now includes the 'Full logs:' tmp_path hint, matching assert_kraken_success/failure. --------- Co-authored-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com>
CI Tests
First steps
Edit functional_tests with tests you want to run
How to run
./CI/run.sh
This will run kraken using python, make sure python3 is set up and configured properly with all requirements
Adding a test case
-
Add in simple scenario yaml file to execute under ../CI/scenarios/
-
Copy test_application_outages.sh for example on how to get started
-
Lines to change for bash script
a. 11: Set scenario type to be your new scenario name
b. 12: Add pointer to scenario file for the test
c. 13: If a post action file is needed; add in pointer
d. 14: Set filled in config yaml file name specific to your scenario
e. 15: Make sure name of config in line 14 matches what you pass on this line
-
Add test name to functional_tests file
a. This will be the name of the file without ".sh"
-
If any changes to the main config (other than the scenario list), please be sure to add them into the common_config