mirror of
https://github.com/krkn-chaos/krkn.git
synced 2026-09-01 09:37:16 +00:00
* test: migrate test_namespace.sh to CI/tests_v2 namespace_deletion Migrate the legacy CI/tests/test_namespace.sh functional test to the pytest-based v2 framework under CI/tests_v2/scenarios/namespace_deletion/. Covers the service_disruption (namespace deletion) scenario with happy-path cases (single-namespace object deletion, multi-namespace delete_count, multiple runs, wait_time handling, label-selector targeting) and negative cases (no-match regex, namespace/label mutual exclusion, delete_count exceeding available namespaces). - Add scenario assets (resource.yaml with Deployment+Service, scenario_base.yaml) - Register namespace_deletion marker in pytest.ini - Add namespace_deletion execution-evidence marker in lib/utils.py Closes #1405 * test: move namespace_deletion helpers into reusable lib modules Address review feedback: extract the per-test helpers into shared lib modules so other scenarios can reuse them. - lib/namespace.py: add POD_SECURITY_PRIVILEGED_LABELS, create_labeled_namespace, delete_namespace_quietly, and a make_namespace factory fixture (auto-cleanup) - lib/deploy.py: add deployment_exists, wait_for_no_deployment, wait_for_present_deployment_count, deploy_manifest_to_namespace - conftest.py: re-export make_namespace fixture - test_namespace_deletion.py: drop local helpers, use the lib functions/fixture * test: honor --keep-ns-on-fail in make_namespace factory The make_namespace finalizer always deleted ad-hoc namespaces, ignoring --keep-ns-on-fail on failure unlike test_namespace. Extract the keep-on-fail decision into a shared helper and use it in both fixtures so the documented debugging workflow works for scenarios that create extra namespaces. * test: surface FailToCreateError in deploy_manifest_to_namespace Mirror deploy_workload's error handling so manifest-apply failures in multi-namespace tests raise a formatted RuntimeError listing the underlying API exceptions instead of an opaque FailToCreateError. * test: clarify why label-selector test bypasses run_scenario The inline comment claimed run_scenario was bypassed because **overrides would collide with the positional namespace arg. The real reason is that NAMESPACE_IS_REGEX=True wraps an empty namespace as '^$', whereas label-selector mode needs a literal empty string. * test: use actual scenario inputs in negative-test failure context The no-match and mutual-exclusion tests reported context=namespace=self.ns (the ephemeral namespace), not the inputs actually under test. Reference the real namespace (and label_selector) so unexpected-success diagnostics are clear. * test: use non-default wait_time so override patching is exercised wait_time=30 matched the scenario_base.yaml default, so the test passed even if override patching regressed. Use wait_time=5 (non-default) so the override path is actually validated. * test: guard cluster post-checks under KRKN_TEST_DRY_RUN Three namespace_deletion tests asserted cluster side effects (workload deletion) after the Kraken run. Under KRKN_TEST_DRY_RUN=1 Kraken is skipped, so the seeded workload is never deleted and wait_for_no_deployment / wait_for_present_deployment_count would time out and fail. Guard those post-checks, and in test_label_selector_targeting (which bypasses run_scenario and calls run_kraken directly) honor dry-run explicitly by skipping the invocation and post-check. * test: tighten no-match assertion, clarify runs-loop test, dry-run-safe negatives Addresses Deep Code Review feedback on the namespace_deletion suite: - test_no_match_namespace_fails: drop the dead 'no namespaces matching' OR branch; the service_disruption plugin only ever logs 'not enough namespaces matching ...', so the assertion now checks that string directly. - test_multiple_runs_repeat_deletion -> test_multiple_runs_repeat_disruption_loop: rename + docstring make explicit that it verifies the outer runs loop iterates twice, not that object deletion recurs (Krkn does not redeploy between runs, so run 2 re-selects an already-empty namespace). Object removal is asserted in test_single_namespace_object_deletion. - Negative tests now return early under KRKN_TEST_DRY_RUN=1, since run_scenario returns a fake rc=0 and the failure path cannot be exercised; this makes the whole class consistent under make test-dry-run. * test: use UUID-based namespace in no-match test to avoid accidental matches * test: assert correct zero-match error in no-match namespace test A regex matching zero namespaces makes krkn_lib's check_namespaces raise 'there exists no namespaces matching' before the plugin's delete loop, so the 'not enough namespaces matching' branch is never reached for this case. Assert the actual zero-match message instead. * test: poll for async Service deletion in namespace_deletion test Kubernetes deletions are asynchronous, so checking the Service immediately after the scenario run could be flaky. Add wait_for_no_service (mirroring wait_for_no_deployment) and poll until the Service is actually gone before asserting. * test: assert Service deletion in label-selector namespace_deletion test test_label_selector_targeting deploys both a Deployment and a Service but only asserted the Deployment was removed. Add wait_for_no_service so the test fully validates that label-selector mode deletes all objects (including Services), matching test_single_namespace_object_deletion. Signed-off-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com> * test: snapshot namespaces in wait_for_present_deployment_count list(namespaces) consumed the iterable before the poll loop re-iterated over the original, so a one-shot iterable (e.g. generator) would be empty on every poll. Snapshot to a list once and iterate over that snapshot. Signed-off-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com> * ci: free up runner disk space before tests_v2 KinD run The Tests v2 (pytest functional) job intermittently fails with 'System.IO.IOException: No space left on device' on ubuntu-latest runners, which ship with only ~14GB free. Creating the KinD cluster plus pulling and kind-loading nginx:alpine and krkn:tools exhausts the disk, failing even the runner's own diagnostic logging. Reclaim ~20-30GB by removing the bundled .NET/Android/GHC SDKs and pruning docker images before the cluster is created. Signed-off-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com> --------- Signed-off-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com> Co-authored-by: augmentcode[bot] <185243770+augmentcode[bot]@users.noreply.github.com> Co-authored-by: Darshan Jain <darjain@redhat.com>
170 lines
5.9 KiB
Python
170 lines
5.9 KiB
Python
"""
|
|
Namespace lifecycle fixtures for CI/tests_v2: create, delete, stale cleanup.
|
|
"""
|
|
|
|
import logging
|
|
import os
|
|
import time
|
|
import uuid
|
|
from datetime import datetime
|
|
|
|
import pytest
|
|
from kubernetes import client
|
|
from kubernetes.client.rest import ApiException
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
STALE_NS_AGE_MINUTES = 30
|
|
|
|
# Privileged pod-security labels applied to ephemeral test namespaces so the same
|
|
# workloads are admitted on both Kubernetes and OpenShift. Shared by the test_namespace
|
|
# fixture and the make_namespace factory.
|
|
POD_SECURITY_PRIVILEGED_LABELS = {
|
|
"pod-security.kubernetes.io/audit": "privileged",
|
|
"pod-security.kubernetes.io/enforce": "privileged",
|
|
"pod-security.kubernetes.io/enforce-version": "v1.24",
|
|
"pod-security.kubernetes.io/warn": "privileged",
|
|
"security.openshift.io/scc.podSecurityLabelSync": "false",
|
|
}
|
|
|
|
|
|
def _namespace_age_minutes(metadata) -> float:
|
|
"""Return age of namespace in minutes from its creation_timestamp."""
|
|
if not metadata or not metadata.creation_timestamp:
|
|
return 0.0
|
|
created = metadata.creation_timestamp
|
|
if hasattr(created, "timestamp"):
|
|
created_ts = created.timestamp()
|
|
else:
|
|
try:
|
|
dt = datetime.fromisoformat(created.replace("Z", "+00:00"))
|
|
created_ts = dt.timestamp()
|
|
except Exception:
|
|
return 0.0
|
|
return (time.time() - created_ts) / 60.0
|
|
|
|
|
|
def _wait_for_namespace_gone(k8s_core, name: str, timeout: int = 60):
|
|
"""Poll until the namespace no longer exists."""
|
|
deadline = time.monotonic() + timeout
|
|
while time.monotonic() < deadline:
|
|
try:
|
|
k8s_core.read_namespace(name=name)
|
|
except ApiException as e:
|
|
if e.status == 404:
|
|
return
|
|
raise
|
|
time.sleep(1)
|
|
raise TimeoutError(f"Namespace {name} did not disappear within {timeout}s")
|
|
|
|
|
|
def create_labeled_namespace(k8s_core, name: str, extra_labels: dict = None) -> str:
|
|
"""Create a namespace with privileged pod-security labels plus any extra_labels.
|
|
|
|
Reusable across scenarios that need ad-hoc namespaces (e.g. multi-namespace
|
|
selection or label-selector targeting). Returns the namespace name.
|
|
"""
|
|
labels = dict(POD_SECURITY_PRIVILEGED_LABELS)
|
|
if extra_labels:
|
|
labels.update(extra_labels)
|
|
body = client.V1Namespace(metadata=client.V1ObjectMeta(name=name, labels=labels))
|
|
k8s_core.create_namespace(body=body)
|
|
logger.info("Created test namespace: %s", name)
|
|
return name
|
|
|
|
|
|
def delete_namespace_quietly(k8s_core, name: str) -> None:
|
|
"""Background-delete a namespace, logging (never raising) on failure. Safe in finalizers."""
|
|
try:
|
|
k8s_core.delete_namespace(
|
|
name=name,
|
|
body=client.V1DeleteOptions(propagation_policy="Background"),
|
|
)
|
|
except Exception as e: # noqa: BLE001 - cleanup must never raise
|
|
logger.warning("Failed to delete namespace %s: %s", name, e)
|
|
|
|
|
|
def _keep_namespace_on_fail(request) -> bool:
|
|
"""True when --keep-ns-on-fail is set and the test's call phase failed."""
|
|
keep_on_fail = request.config.getoption("--keep-ns-on-fail", False)
|
|
rep_call = getattr(request.node, "rep_call", None)
|
|
failed = rep_call is not None and rep_call.failed
|
|
return bool(keep_on_fail and failed)
|
|
|
|
|
|
@pytest.fixture(scope="function")
|
|
def test_namespace(request, k8s_core):
|
|
"""
|
|
Create an ephemeral namespace for the test. Deleted after the test unless
|
|
--keep-ns-on-fail is set and the test failed.
|
|
"""
|
|
name = f"krkn-test-{uuid.uuid4().hex[:8]}"
|
|
ns = client.V1Namespace(
|
|
metadata=client.V1ObjectMeta(
|
|
name=name,
|
|
labels=dict(POD_SECURITY_PRIVILEGED_LABELS),
|
|
)
|
|
)
|
|
k8s_core.create_namespace(body=ns)
|
|
logger.info("Created test namespace: %s", name)
|
|
|
|
yield name
|
|
|
|
if _keep_namespace_on_fail(request):
|
|
logger.info("[keep-ns-on-fail] Keeping namespace %s for debugging", name)
|
|
return
|
|
|
|
delete_namespace_quietly(k8s_core, name)
|
|
|
|
|
|
@pytest.fixture(scope="function")
|
|
def make_namespace(request, k8s_core):
|
|
"""
|
|
Factory fixture to create ad-hoc privileged test namespaces during a test.
|
|
|
|
Returns a callable make(name, extra_labels=None) -> name. Each created namespace
|
|
is registered for teardown deletion, honouring --keep-ns-on-fail the same way the
|
|
test_namespace fixture does. Useful for scenarios that need several namespaces
|
|
(multi-namespace selection) or a uniquely labelled namespace (label-selector targeting).
|
|
"""
|
|
|
|
def _make(name: str, extra_labels: dict = None) -> str:
|
|
create_labeled_namespace(k8s_core, name, extra_labels=extra_labels)
|
|
|
|
def _finalize(ns_name=name):
|
|
if _keep_namespace_on_fail(request):
|
|
logger.info("[keep-ns-on-fail] Keeping namespace %s for debugging", ns_name)
|
|
return
|
|
delete_namespace_quietly(k8s_core, ns_name)
|
|
|
|
request.addfinalizer(_finalize)
|
|
return name
|
|
|
|
return _make
|
|
|
|
|
|
@pytest.fixture(scope="session", autouse=True)
|
|
def _cleanup_stale_namespaces(k8s_core):
|
|
"""Delete krkn-test-* namespaces older than STALE_NS_AGE_MINUTES at session start."""
|
|
if os.environ.get("PYTEST_XDIST_WORKER"):
|
|
return
|
|
try:
|
|
namespaces = k8s_core.list_namespace()
|
|
except Exception as e:
|
|
logger.warning("Could not list namespaces for stale cleanup: %s", e)
|
|
return
|
|
for ns in namespaces.items or []:
|
|
name = ns.metadata.name if ns.metadata else ""
|
|
if not name.startswith("krkn-test-"):
|
|
continue
|
|
if _namespace_age_minutes(ns.metadata) <= STALE_NS_AGE_MINUTES:
|
|
continue
|
|
try:
|
|
logger.warning("Deleting stale namespace: %s", name)
|
|
k8s_core.delete_namespace(
|
|
name=name,
|
|
body=client.V1DeleteOptions(propagation_policy="Background"),
|
|
)
|
|
except Exception as e:
|
|
logger.warning("Failed to delete stale namespace %s: %s", name, e)
|