diff --git a/README.md b/README.md index eab544da..aee900ed 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,12 @@ Istio progressive rollout gated by Prometheus HTTP success rate metric ### Usage +Deploy steerer in `istio-system` namespace: + +```bash +kubectl apply -f ./artifacts/steerer +``` + Create a test namespace: ```bash @@ -16,26 +22,65 @@ Create GA and canary deployments, services, hpa and Istio virtual service: kubectl apply -f ./artifacts/workloads/ ``` -Rollout: +Create rollout custom resources: -![rollout-cli](docs/screens/rollout-cli-output.png) +```bash +kubectl apply -f ./artifacts/rollouts/ +``` + +Rollout output: + +```bash +kubectl -n test describe rollout/podinfo + +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Synced 3m steerer Stating rollout for podinfo.test + Normal Synced 3m steerer Advance rollout podinfo.test weight 10 + Normal Synced 3m steerer Advance rollout podinfo.test weight 20 + Normal Synced 2m steerer Advance rollout podinfo.test weight 30 + Normal Synced 2m steerer Advance rollout podinfo.test weight 40 + Normal Synced 2m steerer Advance rollout podinfo.test weight 50 + Normal Synced 2m steerer Advance rollout podinfo.test weight 60 + Normal Synced 2m steerer Advance rollout podinfo.test weight 60 + Warning Synced 2m steerer Halt rollout podinfo.test success rate 88.89% < 99% + Warning Synced 2m steerer Halt rollout podinfo.test success rate 82.86% < 99% + Warning Synced 1m steerer Halt rollout podinfo.test success rate 80.49% < 99% + Warning Synced 1m steerer Halt rollout podinfo.test success rate 82.98% < 99% + Warning Synced 1m steerer Halt rollout podinfo.test success rate 83.33% < 99% + Warning Synced 1m steerer Halt rollout podinfo.test success rate 82.22% < 99% + Warning Synced 1m steerer Halt rollout podinfo.test success rate 94.74% < 99% + Normal Synced 1m steerer Advance rollout podinfo.test weight 70 + Normal Synced 55s steerer Advance rollout podinfo.test weight 80 + Normal Synced 45s steerer Advance rollout podinfo.test weight 90 + Normal Synced 35s steerer Advance rollout podinfo.test weight 100 + Normal Synced 25s steerer Copying podinfo-canary.test template spec to podinfo.test + Warning Synced 15s steerer Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available + Normal Synced 5s steerer Promotion complete! Scaling down podinfo-canary.test +``` Rollout flow: -* scan namespace for deployments marked for rollout -* scan namespace for a corresponding canary deployment (`-canary` prefix) -* check Istio virtual service routes are mapped to GA and canary ClusterIP services -* check GA and canary deployments status (halt rollout if a rolling update is underway or if pods are unhealthy) + +* scan for deployments marked for rollout +* check Istio virtual service routes are mapped to primary and canary ClusterIP services +* check primary and canary deployments status + * halt rollout if a rolling update is underway + * halt rollout if pods are unhealthy * increase canary traffic weight percentage by 10% -* check canary HTTP success rate (halt rollout if percentage is under the specified threshold) -* advance canary traffic wight by 10% till it reaches 100% +* check canary HTTP success rate + * halt rollout if percentage is under the specified threshold +* increase canary traffic wight by 10% till it reaches 100% * halt rollout while canary success rate is under the threshold - * halt rollout if the GA or canary deployment becomes unhealthy + * halt rollout if the primary or canary deployment becomes unhealthy * halt rollout while canary deployment is being scaled up/down by HPA -* promote canary to GA (copy canary deployment spec template over GA) -* wait for GA rolling update to finish -* route all traffic to GA +* promote canary to primary + * copy canary deployment spec template over primary +* wait for primary rolling update to finish + * halt rollout if pods are unhealthy +* route all traffic to primary * scale to zero the canary deployment -* mark rollout deployment as finished +* mark rollout as finished * wait for the canary deployment to be updated (revision bump) and start over HTTP success rate query: