- Isolate TestLiveMigrationInBackground into dedicated e2e-livemigration namespace with t.Cleanup teardown.
- Switch VMI containerdisk image from fedora (1.2GB, 512MB RAM) to cirros-container-disk-demo (12MB, 128MB RAM), reducing Kind memory pressure by 75%.
- Filter pod list API calls with LabelSelector=kubevirt.io=virt-launcher to reduce API server serialization load.
- Override --descheduling-interval to 3s in startDeschedulerServer (e2e_leaderelection_test.go) so leader descheduler re-evaluates pod lifetime periodically.
- Exclude context cancellation/deadline from isClientRateLimiterError in e2e_test.go so context deadlines are not swallowed.
- Add t.Cleanup and IsAlreadyExists handling to prevent leftover resources from causing cascading failures.
Signed-off-by: Amir Alavi <amiralavi7@gmail.com>
Pods evicted in background may stay in Succeeded/Failed phase forever
without being garbage-collected, so the metric was silently dropped.
Similarly, entries whose background eviction timed out in cleanCache
were expired without recording an outcome.
- UpdateFunc: emit "success"/"error" metric when a pod transitions to
PodSucceeded/PodFailed respectively, matching the label convention
used elsewhere in the file
- cleanCache: add onAssumedTimeout callback on evictionRequestsCache,
wired in NewPodEvictor to emit "error" for entries that exceed the
assumed-eviction timeout
- Add TestEvictionInBackgroundMetrics_PodCompleted and
TestEvictionInBackgroundMetrics_AssumedTimeout to cover both paths
Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
When KubeVirt sets EvictionInProgressAnnotationKey before returning
TooManyRequests, the informer's UpdateFunc can call addPod
(evictionAssumed=false) before evictPod's assumePod call arrives.
assumePod found the entry already present and returned early, leaving
evictionAssumed=false. DeleteFunc then skipped the "success" metric.
Fix: if the existing entry has evictionAssumed=false (added by addPod),
upgrade it in place without double-counting the pod in the counters.
Adds TestEvictionInBackgroundMetrics_InformerRace to reproduce the race
deterministically.
Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
Profile creation was moved outside the descheduling cycle in b214c147,
but reconcileInClusterSAToken() still runs only in runFnc(), after
newDescheduler() returns. This leaves the prometheus client nil when
LowNodeUtilization's New() runs, causing "prometheus client not
initialized" at startup.
Avoid failing at plugin creation time if the prometheus
client is not yet available. Instead, usageClientForMetrics() is now
called at the start of every extension point via a resetUsageClient()
helper, so each descheduling cycle picks up the latest client regardless
of when the SA token is reconciled or rotated.
Fixes: https://github.com/kubernetes-sigs/descheduler/issues/1840
Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
Background evictions were completely invisible in metrics: the ignore=true
path caused EvictPod to return before incrementing any counter, leaving
operators with no signal that a background eviction had been triggered or
completed.
Add a "background" result label emitted at eviction request time and a
"success" label emitted from the informer DeleteFunc when the pod is
actually gone. The two labels together give a complete picture:
"background" is recorded at eviction request time and may not have a
matching "success" if the descheduler restarts before the pod is deleted,
while "success" confirms the eviction completed within the same lifecycle.
Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
Move container waiting/terminated state checking from PodLifeTime and
RemovePodsHavingTooManyRestarts into podutil as separate exported helpers:
HasMatchingContainerWaitingState and HasMatchingContainerTerminatedState.
Each plugin composes only the helpers it needs.
When the descheduler is running in the dry mode the the kube client sandbox restoring may fail.
Which can be caused by timeouts when waiting for internal caches to sync.
The internal timeouts depend on the cluster size which changes in time.
No reason to cancel the context because of that.
Currently, there's a single prometheus client reconciler for both in
cluster and secret based strategies. The in cluster reconciling is run in
sync with each descheduling cycle. An in file token either changes or it
does not. If changed a new prometheus client is created. The secret
based reconciling is run async and watches for secret object changes. If a
secret changes a new client is created. The internal state of the
reconciler keeps previous connection data for clearing and checks.
The current reconciler implementation lacks mutually exclusive access.
So data races are possible. The prometheus configuration validation is
performed during every sync. The future refactorings is expected to move
the validation to the creation phase of the reconciler.
The extra unit testing is expected to cover the following scenarios:
- in cluster:
- in file token is unchanged: no-op
- in file token is changed: client is created or fails to be created
- secret:
- no secret is not found: no client creation, internal state cleared
- secret is found: if token changed a new client created, otherwise
no-op
- prometheus config validation
- prometheus client injection
Any error during new prom client creation should be followed by closing
the previous connection and reseting the internal state. Yet, the error
handling is not that strict currently. So the current extra unit testing
keeps the incomplete testing cases as they are.
Other use of the tests is to make sure every time a new prometheus
client is created a descheduling cycle injects a new profile with the
updated prometheus clients. So the future refactoring does not introduce
a regression.
The underlying implementation is the same. Only moving the code under a
separate controller that can be unit test independently of the
descheduler type implementation.
kubernetes-prow[bot]andGitHub
PKNaveen
Amir Alavi
Simone Tiraboschi
Jan Chaloupka
Linus FankhauserandGitHub
sammedsingalkar09
Kubernetes Prow RobotandGitHub
Danila Bobkov