From 217ad6ef19e4f1f051d6f6b254f8b27717ddab3a Mon Sep 17 00:00:00 2001 From: Stefan Prodan Date: Sun, 30 Sep 2018 09:48:39 +0300 Subject: [PATCH] Release 0.0.1-rc.6 - canary rollback based on failed checks threshold --- README.md | 53 ++++++++++++++++++++++++------ artifacts/rollouts/podinfo.yaml | 3 ++ artifacts/steerer/crd.yaml | 2 ++ artifacts/steerer/deployment.yaml | 2 +- chart/steerer/Chart.yaml | 2 +- chart/steerer/templates/crd.yaml | 2 ++ chart/steerer/values.yaml | 2 +- docs/index.yaml | 8 ++--- docs/steerer-0.0.1.tgz | Bin 3127 -> 3132 bytes pkg/version/version.go | 2 +- 10 files changed, 58 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 20a1e115..110f64bd 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # steerer -[![travis](https://travis-ci.org/stefanprodan/steerer.svg?branch=master)](https://travis-ci.org/stefanprodan/steerer) -![release](https://img.shields.io/github/release/stefanprodan/steerer/all.svg) -![license](https://img.shields.io/github/license/stefanprodan/steerer.svg) +[![build](https://travis-ci.org/stefanprodan/steerer.svg?branch=master)](https://travis-ci.org/stefanprodan/steerer) +[![release](https://img.shields.io/github/release/stefanprodan/steerer/all.svg)](https://github.com/stefanprodan/steerer/releases) +[![license](https://img.shields.io/github/license/stefanprodan/steerer.svg)](https://github.com/stefanprodan/steerer/blob/master/LICENSE) Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis. @@ -12,7 +12,7 @@ using Istio routing for traffic shifting and Prometheus metrics for canary analy Before installing Steerer make sure you have Istio setup up with Prometheus enabled. If you are new to Istio you can follow my [GKE service mesh walk-through](https://github.com/stefanprodan/istio-gke). -Deploy Steerer in the `istio-system` using Helm: +Deploy Steerer in the `istio-system` namespace using Helm: ```bash # add the Helm repository @@ -41,8 +41,13 @@ Gated rollout stages: * halt rollout if a rolling update is underway * halt rollout if pods are unhealthy * increase canary traffic weight percentage from 0% to 10% -* check canary HTTP success rate - * halt rollout if percentage is under the specified threshold +* check canary HTTP request success rate and latency + * halt rollout if any metric is under the specified threshold + * increment the failed checks counter +* check if the number of failed checks reached the threshold + * route all traffic to primary + * scale to zero the canary deployment and mark it as failed + * wait for the canary deployment to be updated (revision bump) and start over * increase canary traffic wight by 10% (step wight) till it reaches 100% (max weight) * halt rollout while canary request success rate is under the threshold * halt rollout while canary request duration P99 is over the threshold @@ -57,7 +62,7 @@ Gated rollout stages: * mark rollout as finished * wait for the canary deployment to be updated (revision bump) and start over -You can change the canary analysis max weight and the step wight size in the rollout custom resource. +You can change the canary analysis _max weight_ and the _step wight_ percentage in the rollout custom resource. Assuming the primary deployment is named _podinfo_ and the canary one _podinfo-canary_, Steerer will require a virtual service configured with weight-based routing: @@ -120,6 +125,9 @@ spec: name: podinfo-canary host: podinfo-canary canaryAnalysis: + # max number of failed checks + # before rolling back the canary + threshold: 10 # max traffic percentage routed to canary # percentage (0-100) maxWeight: 100 @@ -136,7 +144,7 @@ spec: # maximum req duration P99 # milliseconds threshold: 500 - interval: 1m + interval: 30s ``` The canary analysis is using the following promql queries: @@ -233,7 +241,7 @@ Events: Normal Synced 3m steerer Starting rollout for podinfo.test Normal Synced 3m steerer Advance rollout podinfo.test weight 10 Normal Synced 3m steerer Advance rollout podinfo.test weight 20 - Normal Synced 2m steerer Advance rollout podinfo.test weight 30 + Normal Synced 3m steerer Advance rollout podinfo.test weight 30 Warning Synced 3m steerer Halt rollout podinfo.test request duration 2.525s > 500ms Warning Synced 3m steerer Halt rollout podinfo.test request duration 1.567s > 500ms Warning Synced 3m steerer Halt rollout podinfo.test request duration 823ms > 500ms @@ -249,7 +257,7 @@ Events: Normal Synced 35s steerer Advance rollout podinfo.test weight 100 Normal Synced 25s steerer Copying podinfo-canary.test template spec to podinfo.test Warning Synced 15s steerer Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available - Normal Synced 5s steerer Promotion complete! Scaling down podinfo-canary.test + Normal Synced 5s steerer Promotion completed! Scaling down podinfo-canary.test ``` During the rollout you can generate HTTP 500 errors and high latency to test if Steerer pauses the rollout. @@ -273,3 +281,28 @@ Generate latency: watch curl http://podinfo-canary:9898/delay/1 ``` +When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, +the canary is scaled to zero and the rollout is marked as failed. + +``` +kubectl -n test describe rollout/podinfo + +Status: + Canary Revision: 16695041 + Failed Checks: 10 + State: failed +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Synced 3m steerer Starting rollout for podinfo.test + Normal Synced 3m steerer Advance rollout podinfo.test weight 10 + Normal Synced 3m steerer Advance rollout podinfo.test weight 20 + Normal Synced 3m steerer Advance rollout podinfo.test weight 30 + Normal Synced 3m steerer Halt rollout podinfo.test success rate 69.17% < 99% + Normal Synced 2m steerer Halt rollout podinfo.test success rate 61.39% < 99% + Normal Synced 2m steerer Halt rollout podinfo.test success rate 55.06% < 99% + Normal Synced 2m steerer Halt rollout podinfo.test success rate 47.00% < 99% + Normal Synced 2m steerer (combined from similar events): Halt rollout podinfo.test success rate 38.08% < 99% + Warning Synced 1m steerer Rolling back podinfo-canary.test failed checks threshold reached 10 + Warning Synced 1m steerer Canary failed! Scaling down podinfo.test +``` diff --git a/artifacts/rollouts/podinfo.yaml b/artifacts/rollouts/podinfo.yaml index fd6473d2..aa8a7608 100644 --- a/artifacts/rollouts/podinfo.yaml +++ b/artifacts/rollouts/podinfo.yaml @@ -23,6 +23,9 @@ spec: name: podinfo-canary host: podinfo-canary canaryAnalysis: + # max number of failed metric checks + # before rolling back the canary + threshold: 10 # max traffic percentage routed to canary # percentage (0-100) maxWeight: 100 diff --git a/artifacts/steerer/crd.yaml b/artifacts/steerer/crd.yaml index f0126196..901add8c 100644 --- a/artifacts/steerer/crd.yaml +++ b/artifacts/steerer/crd.yaml @@ -47,6 +47,8 @@ spec: type: string canaryAnalysis: properties: + threshold: + type: number maxWeight: type: number stepWeight: diff --git a/artifacts/steerer/deployment.yaml b/artifacts/steerer/deployment.yaml index c3b76069..f03fcdff 100644 --- a/artifacts/steerer/deployment.yaml +++ b/artifacts/steerer/deployment.yaml @@ -22,7 +22,7 @@ spec: serviceAccountName: steerer containers: - name: steerer - image: stefanprodan/steerer:0.0.1-rc.1 + image: stefanprodan/steerer:0.0.1-rc.6 imagePullPolicy: Always ports: - name: http diff --git a/chart/steerer/Chart.yaml b/chart/steerer/Chart.yaml index cac84709..d8baa441 100644 --- a/chart/steerer/Chart.yaml +++ b/chart/steerer/Chart.yaml @@ -1,5 +1,5 @@ apiVersion: v1 name: steerer version: 0.0.1 -appVersion: 0.0.1-rc.1 +appVersion: 0.0.1-rc.6 description: Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis. diff --git a/chart/steerer/templates/crd.yaml b/chart/steerer/templates/crd.yaml index 153c2f83..eb30bd3d 100644 --- a/chart/steerer/templates/crd.yaml +++ b/chart/steerer/templates/crd.yaml @@ -48,6 +48,8 @@ spec: type: string canaryAnalysis: properties: + threshold: + type: number maxWeight: type: number stepWeight: diff --git a/chart/steerer/values.yaml b/chart/steerer/values.yaml index 0d9bb1b6..e4154672 100644 --- a/chart/steerer/values.yaml +++ b/chart/steerer/values.yaml @@ -2,7 +2,7 @@ image: repository: stefanprodan/steerer - tag: 0.0.1-rc.1 + tag: 0.0.1-rc.6 pullPolicy: IfNotPresent controlLoopInterval: "10s" diff --git a/docs/index.yaml b/docs/index.yaml index 5f71a325..5a584f88 100755 --- a/docs/index.yaml +++ b/docs/index.yaml @@ -2,14 +2,14 @@ apiVersion: v1 entries: steerer: - apiVersion: v1 - appVersion: 0.0.1-rc.1 - created: 2018-09-29T11:08:25.598356915+03:00 + appVersion: 0.0.1-rc.6 + created: 2018-09-30T09:47:15.355564357+03:00 description: Steerer is a Kubernetes operator that automates the promotion of canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis. - digest: af14826edae5afcda1b2afebf17e3b8007f1d2a35e65093ab32e786a6599b201 + digest: 02c30ba24d2fc0e00238fe12c5cf8525a4dbf5192af555a104f64e7bd2dbb35a name: steerer urls: - https://stefanprodan.github.io/steerer/steerer-0.0.1.tgz version: 0.0.1 -generated: 2018-09-29T11:08:25.597473362+03:00 +generated: 2018-09-30T09:47:15.354615937+03:00 diff --git a/docs/steerer-0.0.1.tgz b/docs/steerer-0.0.1.tgz index 09776f905bfe1f97464d941363c1bfed5d7b9acd..8195d067774a5ea8ed1d4d98b70d6a223535fa6f 100644 GIT binary patch delta 3102 zcmV+(4B_*)7`zyeJ%4i>H?H}cp8`*Ar-_?gt*qEiHP4;Nv(q=8CXUBW(&^-K6}1aV zY`7#ta%HQw@3Y^K`$N0>VLKPcX|KW`ww44y5(Gg2B#lMXsN4Hvs%DSvS15Jge6 zyS?rIM^QBYKkDvoKj?J3+fSn1XnUvoAnI)GM9~Ad`<^U;%zs#_A4JjT@|hppUj(RP zZ&4d2MGqz&LY1mMqc93PfsVr+lAwt-Q`VQC<|+fs0EM5k0cwF34M>HWTB*T~sRf!@ znNnA7#|TQxR625yLx`!MdP)EiR9sF|6xKjym>9vKu}p%NnRQ=7$4t}Vki}reZ0HNA zNZ?q@6zv!@1Ai%6&0^zl2^6e>3d*O3nUDyYVh;)}CzHn5$k!JBp8(4BKcPHB^JNUc zP3wO*nqU9jsM}qy|NDRqIKUyzxP@E;{F*D2A|!;Rbc8(upixP~tkhG#nub&;EfXr* z<E75Ub2=ii&f@F z6h&#v%YW}A}1AggB!pn4S9S&zS#m z{aZ{Gcjmf%^y>BV)6kyVJJYe7)_=6MHMjnsZ0+o>*Z+M$wV4ZJDd(81H^sjdEUy3G z#>f?F6IyjguD}iJe=FLW-~YEeJKO8^e;;sh(SP2A2}^rm(87>$v{Qw9sS`^)#vW|8 zuday0BG1p2^Szxr@AgLL$-CdNj~q#X7eInTCJ~#H4mY!zAAv=dwFk^-y@xSx_?fX3)Xw`<>#i#Hwqnr&HCP0<)JrBt* zn16Hk_|~=IMqyxpG0hBuktxGCDn0i?p9GyB`7p546 z@uQF&4nZSFY6_t_$A>auaYq4li^zBR2mZh5<( zqGzjqH}tPIv2M8(A*Y*EKe>gbx-ICW{~zl-1+fiXW5EqmnwA|63C6>3e>_BR=CBk>am-4y+9bFW|ofT zFp8nU+REjyxc+0E+;JN4hV|d=KG~g*|GUw8{of1BIgM`e^W2zQA$8+G6~l9j!hM42YJT%gmaPDrLemq!SF!r2b%t^P3&s zNXtyQX6NbwCNq1Lw_X($_9iGAnbkWms3({dDZiq~O6@H~QRc`B9*Sq0@>xfHp(%&S21DQW|S93ajsHnmgeU)_0sC_ zX0h;Daq3d5T-Iv`vmQ0e0bKam17kH4qefM;uk&Q98gk&ZVWISzrizh&(KZ~6rLmtx zl*{?g+OKl{Kc#bBrOgnR?|_f z>2yi$buw_kET$`}O4f4lJH~be%1!*?8eM2MvqC4Tj+VzYpUVh9QEO3)9<=`XE((5l z|NX=CQEMgswVKjwxtP2%q+R=Zt6|aqn;pvEl^b}I|F_dw$p7wk)_?imdx55lr%byF z0uL%ZSRMc}5%VlrnvJ|B23(+883Ha~-hBjE`cE1aXE8NBaL%H!npzwc+iY%8PB8b1 zU3Gado1f%T8;@Vq273FU-Z+ACO_pcIvq^Bk8;yQSv$qgk-Lo`}Y`{_DLQRfQq;gT$ zx>GvgP0XBMoM~pK`+rhcJfGWxHPYmj;L{T+?N5wje#+nQ+6%-?JTq?$>K=5WDC*QI z^GgfRa)T@48ebI7E-}E>Rd1o%(h*!;wdOe;*KRQ^(9Etl)a47Hq_&N%xRl4(TGjNO zO5q(#r}N?ED1z7nj zO?jmdRV7>k6MqQ`3!SY}F`Q#|#_V`@MpV1-GSi;MHJExEGoDZDEDK+6X$$3jnHl%$ z2F+)3u3gQUF|Xg(xjp}{u>UXme|kXUyG;Pz;Qw`YcAm`V|DHVAS^IzY0dx63Kl#fO zfWC@mb}Tjf!>jg+EMR^a??iIE?a9HdJjXxT>4DC;vwwU6P{saV%S@Rby!&qRyZ5AS z8LOfR>Ou7xfRLihlfF55u@Wx`K6eWKkBP)Z;s4DuAvvDl(DkvLi)%zh%7krQPd`N_ zCi8h1Klisi4HIR;cX61$7%t&sCT%h1rH61|XqzbQe>g^-vXPJ)?~=WTo6di?7vleJ z6m{12|9?JU1CH}7oB=B#R{%I0qkuuixHBD~X#9bW(1c_IUXPgplPMqGfEgp_FyeBM z$H7dD9=ioiEt?=HYR9#5DiX2*0!Mx|!b7ET$j&jzy{Z3s6oQNUKng4qgP>3YX97bK z9-RJmYNbZ90sAsdrGU5lr;sphNH}7)?f>T8Cx7AK58d{E%YyN!?f#UXW+K`%Mgtmu z$dtePX2@n}&J@`U2lNBk3~j2&=D)}Wyrr7S%)sHnb3;Ol z*Z;}$X9q8zhiUQ!+PEqIx4Sj(|7~^F??3JbHp=H*gxt})uKKGkatE)v7Ti~DR`#OxQE-|rF9@AqRVjPnG(0k;uCrbe132tiQt zE6^cqd#)B%L#D-b=yGyr?bfC?0gz7Kp&u20FaG9qlYtOUb(o28gfmXjHw{$e4vvlML z{`c+klheajM?Y3rv%KaE*_DCHlbx5)ufOQ(lYSNSgymbOcC0c-tz-iG$8X9vP>!|} zrOQe3mr*5E{#7UJZoOm;%KRPHe1B8EG;b)HYpNhv+E7L2A9D3czf2N2JRuWoJLIB} z`Q+A-6iE8@R8@&v#7^FZc`{?zNVuioTDZ9ClYT8tYSi7slxiE!XvQDiJ48_w z?eFgT|4|ez{*U_myN`PP{%(J--|O%99!0&K-TnPXaQ{770e`u%R6mNMZ{@Q%xZep- z#onVfOo{#ByG8EQ8ZkU+Bv9U~ombrCbW5-O>@tDP6Qa1L5 zR3vb!WrjAz+FFoA)UbBubi&gGN z6h&Fb%YW~4j%JCVzl5K42oW;DbL1FXsRwX%Lx`2!8CNES8Ni2+ggB!pn4S6RZ-(Q9JI}{sb_tze&J0E3e;55b^m{tcx(&CB&){=TIU6%ffGXvC9+H1z z!GGQ3Ti1pgg@F;qG&cxFX4njE{CLLVk^p+g$u?xj1Y3;`so5I*ZH!yYeieRDRYUiaV^b7AEw4x zi*P5Ab2I1kC*S?S!QE%I(a%~E1()!w5PwraT=%tA`qdp&nh86G&Yw;2XVY2WRulZc*7|EqtW%dV|em`DcTVs94Xi()0Xh<}_W z)KV9_7WuTRE2tLC;zl3B&5a9>onYwNT4u^Yfwv5SmTzvs>g2I^%5aP%`BZP zV3b3Hjg>24S^s05+;bZ67X9z<>@NEM&-&5E|9=oza2nm@7r8N~Ax+~z6~jx5!hMW*rfQhBT0r)CKV@zD%DQAgK?G?yUp?Gn6{C~cG9;cYm zLD^TOr9y34#nZP_%Blj5pL3=$X_mRxE!7jW|6n3nsF*TsbIMDjxKODy%ZhWFW@&wR zvt0P1JauVQuIjaed5_xV051F-fU%m1Nvo>e*F~~*4LR_uVX5>BO_d}6u5CC-rLkW{ zRLJ?a+OKN1;*qmqy}%SUFR> z0AwQOd9pGa`Gpv8iE3>KxP*EC5n$y%X;hrW)C|Bmi^ghdaZ+xxg)KV6!Yg*w)xm6W zl1puzzG@8g?nAwG1eH!!XU6kMaKKxQeoeD?5M9%=G_7pFN$WyQfl;P%Q8&6%IpJ;0 zf?u3#W@m>|SiD@=y?-^* zSH?BID4btnfSa4aQnjTMxVh;payo6?VpyV?UvX&47eGmETbp<#kJMV#_MIx>ol3Xz zpGVIl(yYxg+V+wI2)oNt4Itoh5^$U%KVV`k+jZrIi-1e1f`4lFKWGeR#f^;ihyn;o zzbWvclx|YDE^V~wIGdtCV@|akEjTWC*^Xb<47h0hVuFiqQvgBmIcF=&t!8KrdaG(! zhBCJYJ1gppV#$)j7(jUk#osUS&as=NZih{pGu0KCQj>+S75Tg?AGd$zmr{~iJs@_&BvS0?~{70qob zHT%^s?KN4z;xgWuK;opWdT0-%chr+=2YG6VSVm+ilNBu&d$7fn%* z>dyd#6jh$|-Px5Brbmc|2z|t;~9=!AFH|ef~ZK9ux+l>uaSw#eFn$R z{asJPM3wMe9;PpbYxtZ?TaJ04P`CMx?MrpPlk5mMuQviES?`S0#h{NH!g8~gtN zumz_@{(sJZl~61IxJXgJC}-T64p21yL?>uMvITEbX29gihc{qSr z$rc>SERzD>AD%TkxK0CUXPFM=uQtm6kq<8?ppxx4_f#U%$uvWTpRSFJBzJ zei>%TcWC3b{NMi0qW`zk+r0mH7}%RNu&MYgu!xVW$GNL@et z?SD5y-pcACd*hOww=eFqbrEw|uKeMEkl}C`OJSTR@F#E^A>?YJX@U?06~7W4!mj6P zsuTlG+(ATu7tdG-6*2`QZS}|KqrOoRzuX~Ll$5~O_SZ+hX_5>53v+uHpz=9 z^W)>K*7p-I82fqQ%-C|SsHPcO)TWrpn12cw`5UR84iiXn?F3p5x$v62wBHG~PTqM{ z)lHD2MPFSDx3L&j%DJQo0+s6t*2%&p(P_-&6b)1}+mxBilob9AAvH$EjM{2bnXiFD z{2Z%wTP}rSbcapJ7442C=kmgjmX0`@h8Sx(k9bR_<#p(4a_8;Z6q&BC$d(W72!A=P zNZ}fe=I<;0mq&H(KU_oomkbxPm_xWOQk0G7S#a&8zw|(BSdm*gnbTQ1aRmSV{^i;E z@tc#s)mZbq<{a6Tfhv-n*D!3p=o*q?9rT3NTc>WUa!0LX0*9yXsy9%MwijiqNs5-iCQHW7tGERd6j_-3-aFktT9fZjrflC8s_g&9TVO1*= n@}WWW@p0qhNx|Yr`xG14z!C<2AMn2c009609DL;U07L))YSSsi diff --git a/pkg/version/version.go b/pkg/version/version.go index addeb583..0cb94f9e 100644 --- a/pkg/version/version.go +++ b/pkg/version/version.go @@ -1,4 +1,4 @@ package version -var VERSION = "0.0.1-rc.1" +var VERSION = "0.0.1-rc.6" var REVISION = "unknown"