diff --git a/README.md b/README.md index 20a1e115..110f64bd 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # steerer -[![travis](https://travis-ci.org/stefanprodan/steerer.svg?branch=master)](https://travis-ci.org/stefanprodan/steerer) -![release](https://img.shields.io/github/release/stefanprodan/steerer/all.svg) -![license](https://img.shields.io/github/license/stefanprodan/steerer.svg) +[![build](https://travis-ci.org/stefanprodan/steerer.svg?branch=master)](https://travis-ci.org/stefanprodan/steerer) +[![release](https://img.shields.io/github/release/stefanprodan/steerer/all.svg)](https://github.com/stefanprodan/steerer/releases) +[![license](https://img.shields.io/github/license/stefanprodan/steerer.svg)](https://github.com/stefanprodan/steerer/blob/master/LICENSE) Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis. @@ -12,7 +12,7 @@ using Istio routing for traffic shifting and Prometheus metrics for canary analy Before installing Steerer make sure you have Istio setup up with Prometheus enabled. If you are new to Istio you can follow my [GKE service mesh walk-through](https://github.com/stefanprodan/istio-gke). -Deploy Steerer in the `istio-system` using Helm: +Deploy Steerer in the `istio-system` namespace using Helm: ```bash # add the Helm repository @@ -41,8 +41,13 @@ Gated rollout stages: * halt rollout if a rolling update is underway * halt rollout if pods are unhealthy * increase canary traffic weight percentage from 0% to 10% -* check canary HTTP success rate - * halt rollout if percentage is under the specified threshold +* check canary HTTP request success rate and latency + * halt rollout if any metric is under the specified threshold + * increment the failed checks counter +* check if the number of failed checks reached the threshold + * route all traffic to primary + * scale to zero the canary deployment and mark it as failed + * wait for the canary deployment to be updated (revision bump) and start over * increase canary traffic wight by 10% (step wight) till it reaches 100% (max weight) * halt rollout while canary request success rate is under the threshold * halt rollout while canary request duration P99 is over the threshold @@ -57,7 +62,7 @@ Gated rollout stages: * mark rollout as finished * wait for the canary deployment to be updated (revision bump) and start over -You can change the canary analysis max weight and the step wight size in the rollout custom resource. +You can change the canary analysis _max weight_ and the _step wight_ percentage in the rollout custom resource. Assuming the primary deployment is named _podinfo_ and the canary one _podinfo-canary_, Steerer will require a virtual service configured with weight-based routing: @@ -120,6 +125,9 @@ spec: name: podinfo-canary host: podinfo-canary canaryAnalysis: + # max number of failed checks + # before rolling back the canary + threshold: 10 # max traffic percentage routed to canary # percentage (0-100) maxWeight: 100 @@ -136,7 +144,7 @@ spec: # maximum req duration P99 # milliseconds threshold: 500 - interval: 1m + interval: 30s ``` The canary analysis is using the following promql queries: @@ -233,7 +241,7 @@ Events: Normal Synced 3m steerer Starting rollout for podinfo.test Normal Synced 3m steerer Advance rollout podinfo.test weight 10 Normal Synced 3m steerer Advance rollout podinfo.test weight 20 - Normal Synced 2m steerer Advance rollout podinfo.test weight 30 + Normal Synced 3m steerer Advance rollout podinfo.test weight 30 Warning Synced 3m steerer Halt rollout podinfo.test request duration 2.525s > 500ms Warning Synced 3m steerer Halt rollout podinfo.test request duration 1.567s > 500ms Warning Synced 3m steerer Halt rollout podinfo.test request duration 823ms > 500ms @@ -249,7 +257,7 @@ Events: Normal Synced 35s steerer Advance rollout podinfo.test weight 100 Normal Synced 25s steerer Copying podinfo-canary.test template spec to podinfo.test Warning Synced 15s steerer Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available - Normal Synced 5s steerer Promotion complete! Scaling down podinfo-canary.test + Normal Synced 5s steerer Promotion completed! Scaling down podinfo-canary.test ``` During the rollout you can generate HTTP 500 errors and high latency to test if Steerer pauses the rollout. @@ -273,3 +281,28 @@ Generate latency: watch curl http://podinfo-canary:9898/delay/1 ``` +When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, +the canary is scaled to zero and the rollout is marked as failed. + +``` +kubectl -n test describe rollout/podinfo + +Status: + Canary Revision: 16695041 + Failed Checks: 10 + State: failed +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Synced 3m steerer Starting rollout for podinfo.test + Normal Synced 3m steerer Advance rollout podinfo.test weight 10 + Normal Synced 3m steerer Advance rollout podinfo.test weight 20 + Normal Synced 3m steerer Advance rollout podinfo.test weight 30 + Normal Synced 3m steerer Halt rollout podinfo.test success rate 69.17% < 99% + Normal Synced 2m steerer Halt rollout podinfo.test success rate 61.39% < 99% + Normal Synced 2m steerer Halt rollout podinfo.test success rate 55.06% < 99% + Normal Synced 2m steerer Halt rollout podinfo.test success rate 47.00% < 99% + Normal Synced 2m steerer (combined from similar events): Halt rollout podinfo.test success rate 38.08% < 99% + Warning Synced 1m steerer Rolling back podinfo-canary.test failed checks threshold reached 10 + Warning Synced 1m steerer Canary failed! Scaling down podinfo.test +``` diff --git a/artifacts/rollouts/podinfo.yaml b/artifacts/rollouts/podinfo.yaml index fd6473d2..aa8a7608 100644 --- a/artifacts/rollouts/podinfo.yaml +++ b/artifacts/rollouts/podinfo.yaml @@ -23,6 +23,9 @@ spec: name: podinfo-canary host: podinfo-canary canaryAnalysis: + # max number of failed metric checks + # before rolling back the canary + threshold: 10 # max traffic percentage routed to canary # percentage (0-100) maxWeight: 100 diff --git a/artifacts/steerer/crd.yaml b/artifacts/steerer/crd.yaml index f0126196..901add8c 100644 --- a/artifacts/steerer/crd.yaml +++ b/artifacts/steerer/crd.yaml @@ -47,6 +47,8 @@ spec: type: string canaryAnalysis: properties: + threshold: + type: number maxWeight: type: number stepWeight: diff --git a/artifacts/steerer/deployment.yaml b/artifacts/steerer/deployment.yaml index c3b76069..f03fcdff 100644 --- a/artifacts/steerer/deployment.yaml +++ b/artifacts/steerer/deployment.yaml @@ -22,7 +22,7 @@ spec: serviceAccountName: steerer containers: - name: steerer - image: stefanprodan/steerer:0.0.1-rc.1 + image: stefanprodan/steerer:0.0.1-rc.6 imagePullPolicy: Always ports: - name: http diff --git a/chart/steerer/Chart.yaml b/chart/steerer/Chart.yaml index cac84709..d8baa441 100644 --- a/chart/steerer/Chart.yaml +++ b/chart/steerer/Chart.yaml @@ -1,5 +1,5 @@ apiVersion: v1 name: steerer version: 0.0.1 -appVersion: 0.0.1-rc.1 +appVersion: 0.0.1-rc.6 description: Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis. diff --git a/chart/steerer/templates/crd.yaml b/chart/steerer/templates/crd.yaml index 153c2f83..eb30bd3d 100644 --- a/chart/steerer/templates/crd.yaml +++ b/chart/steerer/templates/crd.yaml @@ -48,6 +48,8 @@ spec: type: string canaryAnalysis: properties: + threshold: + type: number maxWeight: type: number stepWeight: diff --git a/chart/steerer/values.yaml b/chart/steerer/values.yaml index 0d9bb1b6..e4154672 100644 --- a/chart/steerer/values.yaml +++ b/chart/steerer/values.yaml @@ -2,7 +2,7 @@ image: repository: stefanprodan/steerer - tag: 0.0.1-rc.1 + tag: 0.0.1-rc.6 pullPolicy: IfNotPresent controlLoopInterval: "10s" diff --git a/docs/index.yaml b/docs/index.yaml index 5f71a325..5a584f88 100755 --- a/docs/index.yaml +++ b/docs/index.yaml @@ -2,14 +2,14 @@ apiVersion: v1 entries: steerer: - apiVersion: v1 - appVersion: 0.0.1-rc.1 - created: 2018-09-29T11:08:25.598356915+03:00 + appVersion: 0.0.1-rc.6 + created: 2018-09-30T09:47:15.355564357+03:00 description: Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis. - digest: af14826edae5afcda1b2afebf17e3b8007f1d2a35e65093ab32e786a6599b201 + digest: 02c30ba24d2fc0e00238fe12c5cf8525a4dbf5192af555a104f64e7bd2dbb35a name: steerer urls: - https://stefanprodan.github.io/steerer/steerer-0.0.1.tgz version: 0.0.1 -generated: 2018-09-29T11:08:25.597473362+03:00 +generated: 2018-09-30T09:47:15.354615937+03:00 diff --git a/docs/steerer-0.0.1.tgz b/docs/steerer-0.0.1.tgz index 09776f90..8195d067 100644 Binary files a/docs/steerer-0.0.1.tgz and b/docs/steerer-0.0.1.tgz differ diff --git a/pkg/version/version.go b/pkg/version/version.go index addeb583..0cb94f9e 100644 --- a/pkg/version/version.go +++ b/pkg/version/version.go @@ -1,4 +1,4 @@ package version -var VERSION = "0.0.1-rc.1" +var VERSION = "0.0.1-rc.6" var REVISION = "unknown"