github/flagger
1
0
Fork 0
mirror of https://github.com/fluxcd/flagger.git synced 2026-04-15 06:57:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Release 0.0.1-rc.6

Browse Source 
- canary rollback based on failed checks threshold
This commit is contained in:
Stefan Prodan
2018-09-30 09:48:39 +03:00
parent 887daeaffa
commit 217ad6ef19
10 changed files with 58 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+43 -10
View File
@@ -1,8 +1,8 @@
# steerer
[![travis](https://travis-ci.org/stefanprodan/steerer.svg?branch=master)](https://travis-ci.org/stefanprodan/steerer)
![release](https://img.shields.io/github/release/stefanprodan/steerer/all.svg)
![license](https://img.shields.io/github/license/stefanprodan/steerer.svg)
[![build](https://travis-ci.org/stefanprodan/steerer.svg?branch=master)](https://travis-ci.org/stefanprodan/steerer)
[![release](https://img.shields.io/github/release/stefanprodan/steerer/all.svg)](https://github.com/stefanprodan/steerer/releases)
[![license](https://img.shields.io/github/license/stefanprodan/steerer.svg)](https://github.com/stefanprodan/steerer/blob/master/LICENSE)
Steerer is a Kubernetes operator that automates the promotion of canary deployments
using Istio routing for traffic shifting and Prometheus metrics for canary analysis.
@@ -12,7 +12,7 @@ using Istio routing for traffic shifting and Prometheus metrics for canary analy
Before installing Steerer make sure you have Istio setup up with Prometheus enabled.
If you are new to Istio you can follow my [GKE service mesh walk-through](https://github.com/stefanprodan/istio-gke).
Deploy Steerer in the `istio-system` using Helm:
Deploy Steerer in the `istio-system` namespace using Helm:
```bash
# add the Helm repository
@@ -41,8 +41,13 @@ Gated rollout stages:
* halt rollout if a rolling update is underway
* halt rollout if pods are unhealthy
* increase canary traffic weight percentage from 0% to 10%
* check canary HTTP success rate
* halt rollout if percentage is under the specified threshold
* check canary HTTP request success rate and latency
* halt rollout if any metric is under the specified threshold
* increment the failed checks counter
* check if the number of failed checks reached the threshold
* route all traffic to primary
* scale to zero the canary deployment and mark it as failed
* wait for the canary deployment to be updated (revision bump) and start over
* increase canary traffic wight by 10% (step wight) till it reaches 100% (max weight)
* halt rollout while canary request success rate is under the threshold
* halt rollout while canary request duration P99 is over the threshold
@@ -57,7 +62,7 @@ Gated rollout stages:
* mark rollout as finished
* wait for the canary deployment to be updated (revision bump) and start over
You can change the canary analysis max weight and the step wight size in the rollout custom resource.
You can change the canary analysis _max weight_ and the _step wight_ percentage in the rollout custom resource.
Assuming the primary deployment is named _podinfo_ and the canary one _podinfo-canary_, Steerer will require
a virtual service configured with weight-based routing:
@@ -120,6 +125,9 @@ spec:
name: podinfo-canary
host: podinfo-canary
canaryAnalysis:
# max number of failed checks
# before rolling back the canary
threshold: 10
# max traffic percentage routed to canary
# percentage (0-100)
maxWeight: 100
@@ -136,7 +144,7 @@ spec:
# maximum req duration P99
# milliseconds
threshold: 500
interval: 1m
interval: 30s
```
The canary analysis is using the following promql queries:
@@ -233,7 +241,7 @@ Events:
Normal Synced 3m steerer Starting rollout for podinfo.test
Normal Synced 3m steerer Advance rollout podinfo.test weight 10
Normal Synced 3m steerer Advance rollout podinfo.test weight 20
Normal Synced 2m steerer Advance rollout podinfo.test weight 30
Normal Synced 3m steerer Advance rollout podinfo.test weight 30
Warning Synced 3m steerer Halt rollout podinfo.test request duration 2.525s > 500ms
Warning Synced 3m steerer Halt rollout podinfo.test request duration 1.567s > 500ms
Warning Synced 3m steerer Halt rollout podinfo.test request duration 823ms > 500ms
@@ -249,7 +257,7 @@ Events:
Normal Synced 35s steerer Advance rollout podinfo.test weight 100
Normal Synced 25s steerer Copying podinfo-canary.test template spec to podinfo.test
Warning Synced 15s steerer Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
Normal Synced 5s steerer Promotion complete! Scaling down podinfo-canary.test
Normal Synced 5s steerer Promotion completed! Scaling down podinfo-canary.test
```
During the rollout you can generate HTTP 500 errors and high latency to test if Steerer pauses the rollout.
@@ -273,3 +281,28 @@ Generate latency:
watch curl http://podinfo-canary:9898/delay/1
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```
kubectl -n test describe rollout/podinfo
Status:
Canary Revision: 16695041
Failed Checks: 10
State: failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m steerer Starting rollout for podinfo.test
Normal Synced 3m steerer Advance rollout podinfo.test weight 10
Normal Synced 3m steerer Advance rollout podinfo.test weight 20
Normal Synced 3m steerer Advance rollout podinfo.test weight 30
Normal Synced 3m steerer Halt rollout podinfo.test success rate 69.17% < 99%
Normal Synced 2m steerer Halt rollout podinfo.test success rate 61.39% < 99%
Normal Synced 2m steerer Halt rollout podinfo.test success rate 55.06% < 99%
Normal Synced 2m steerer Halt rollout podinfo.test success rate 47.00% < 99%
Normal Synced 2m steerer (combined from similar events): Halt rollout podinfo.test success rate 38.08% < 99%
Warning Synced 1m steerer Rolling back podinfo-canary.test failed checks threshold reached 10
Warning Synced 1m steerer Canary failed! Scaling down podinfo.test
```
artifacts/rollouts/podinfo.yaml
+3
View File
@@ -23,6 +23,9 @@ spec:
name: podinfo-canary
host: podinfo-canary
canaryAnalysis:
# max number of failed metric checks
# before rolling back the canary
threshold: 10
# max traffic percentage routed to canary
# percentage (0-100)
maxWeight: 100
artifacts/steerer/crd.yaml
+2
View File
@@ -47,6 +47,8 @@ spec:
type: string
canaryAnalysis:
properties:
threshold:
type: number
maxWeight:
type: number
stepWeight:
artifacts/steerer/deployment.yaml
+1 -1
View File
@@ -22,7 +22,7 @@ spec:
serviceAccountName: steerer
containers:
- name: steerer
image: stefanprodan/steerer:0.0.1-rc.1
image: stefanprodan/steerer:0.0.1-rc.6
imagePullPolicy: Always
ports:
- name: http
chart/steerer/Chart.yaml
+1 -1
View File
@@ -1,5 +1,5 @@
apiVersion: v1
name: steerer
version: 0.0.1
appVersion: 0.0.1-rc.1
appVersion: 0.0.1-rc.6
description: Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis.
chart/steerer/templates/crd.yaml
+2
View File
@@ -48,6 +48,8 @@ spec:
type: string
canaryAnalysis:
properties:
threshold:
type: number
maxWeight:
type: number
stepWeight:
chart/steerer/values.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
image:
repository: stefanprodan/steerer
tag: 0.0.1-rc.1
tag: 0.0.1-rc.6
pullPolicy: IfNotPresent
controlLoopInterval: "10s"
docs/index.yaml
+4 -4
View File
@@ -2,14 +2,14 @@ apiVersion: v1
entries:
steerer:
- apiVersion: v1
appVersion: 0.0.1-rc.1
created: 2018-09-29T11:08:25.598356915+03:00
appVersion: 0.0.1-rc.6
created: 2018-09-30T09:47:15.355564357+03:00
description: Steerer is a Kubernetes operator that automates the promotion of
canary deployments using Istio routing for traffic shifting and Prometheus metrics
for canary analysis.
digest: af14826edae5afcda1b2afebf17e3b8007f1d2a35e65093ab32e786a6599b201
digest: 02c30ba24d2fc0e00238fe12c5cf8525a4dbf5192af555a104f64e7bd2dbb35a
name: steerer
urls:
- https://stefanprodan.github.io/steerer/steerer-0.0.1.tgz
version: 0.0.1
generated: 2018-09-29T11:08:25.597473362+03:00
generated: 2018-09-30T09:47:15.354615937+03:00
docs/steerer-0.0.1.tgz
BIN
View File
Binary file not shown.
pkg/version/version.go
+1 -1
View File
@@ -1,4 +1,4 @@
package version
var VERSION = "0.0.1-rc.1"
var VERSION = "0.0.1-rc.6"
var REVISION = "unknown"