Compare commits

...

439 Commits

Author SHA1 Message Date
Stefan Prodan
e7fc72e6b5 Merge pull request #364 from weaveworks/release-0.20.2
Release v0.20.2
2019-11-07 12:08:18 +02:00
stefanprodan
4203232b05 Release v0.20.2 2019-11-07 11:34:25 +02:00
stefanprodan
a06aa05201 Add canary namespace to Linkerd webhooks example 2019-11-07 11:34:00 +02:00
Stefan Prodan
8e582e9b73 Merge pull request #363 from weaveworks/no-hpa
Use the specified replicas when scaling up the canary
2019-11-07 10:44:31 +02:00
stefanprodan
0e9fe8a446 Remove the traffic mention from the custom metrics error log
Fix: #361
2019-11-07 09:36:38 +02:00
stefanprodan
27b4bcc648 Use the specified replicas when scaling up the canary 2019-11-07 09:34:53 +02:00
Stefan Prodan
614b7c74c4 Merge pull request #358 from weaveworks/appmesh-gateway
Expose canaries on public domains with App Mesh Gateway
2019-11-06 13:21:20 +02:00
Stefan Prodan
5901129ec6 Merge pull request #359 from KeisukeYamashita/fix-typo-in-how-it-works
Fix typo in section "Webhook" of how-it-works.md
2019-11-06 13:20:53 +02:00
KeisukeYamashita
ded14345b4 doc(how-it-works): fix typo ca to can in how it works doc 2019-11-05 17:39:45 +09:00
stefanprodan
dd272c6870 Expose canaries on public domains with App Mesh Gateway
- map canary service hosts to domain gateway annotation
- map canary retries and timeout to gateway annotations
2019-11-04 18:26:28 +02:00
Stefan Prodan
b31c7c6230 Merge pull request #356 from weaveworks/docs-cleanup
Docs cleanup
2019-11-04 00:52:47 +02:00
stefanprodan
b0297213c3 Use kustomize in Istio docs 2019-11-04 00:35:28 +02:00
stefanprodan
d0fba2d111 Update Istio SMI tutorial 2019-11-04 00:13:19 +02:00
stefanprodan
9924cc2152 Update NGINX usage docs 2019-11-04 00:12:51 +02:00
Stefan Prodan
008a74f86c Merge pull request #354 from weaveworks/prep-0.20.1
Release v0.20.1
2019-11-03 12:29:14 +02:00
stefanprodan
4ca110292f Add v0.20.1 changelog 2019-11-03 11:57:58 +02:00
stefanprodan
55b4c19670 Release v0.20.1 2019-11-03 11:47:16 +02:00
stefanprodan
8349dd1cda Release load tester v0.11.0
- tools updates: Helm v2.15.1, Helm v3.0.0-rc.2, rimusz helm-tiller v0.9.3, gPRC probe v0.3.1
- add hey test during build
2019-11-03 11:46:18 +02:00
Stefan Prodan
402fb66b2a Merge pull request #353 from weaveworks/fix-promql
Fix Prometheus query escape
2019-11-03 11:04:43 +02:00
stefanprodan
f991274b97 Fix Prometheus query escape
Removing whitespace without trimming spaces
2019-11-03 00:01:32 +02:00
Stefan Prodan
0d94a49b6a Merge pull request #350 from laszlocph/update-hey-link
Updating hey release link
2019-10-30 09:01:56 +02:00
Laszlo Fogas
7c14225442 Updating hey release link 2019-10-30 06:40:57 +01:00
stefanprodan
2af0a050bc Fix Prometheus URL in EKS install docs 2019-10-29 18:32:15 +02:00
Stefan Prodan
582f8d6abd Merge pull request #346 from weaveworks/e2e-up
e2e testing: update providers
2019-10-28 16:26:06 +02:00
stefanprodan
eeea3123ac Update e2e NGINX ingress to v1.24.4 2019-10-28 16:08:00 +02:00
stefanprodan
51fe43e169 Update e2e Helm to v2.15.1 2019-10-28 15:32:02 +02:00
stefanprodan
6e6b127092 Update loadtester Helm to v3.0.0-beta.5 2019-10-28 15:31:17 +02:00
stefanprodan
c9bacdfe05 Update Istio to v1.3.3 2019-10-28 15:19:17 +02:00
stefanprodan
f56a69770c Update Linkerd to v2.6.0 2019-10-28 14:42:16 +02:00
Stefan Prodan
0196124c9f Merge pull request #343 from weaveworks/prep-0.20.0
Release v0.20.0
2019-10-22 19:11:59 +03:00
stefanprodan
63756d9d5f Add changelog for v0.20.0 2019-10-22 17:54:18 +03:00
stefanprodan
8e346960ac Add blue/green service mesh docs 2019-10-22 16:57:49 +03:00
stefanprodan
1b485b3459 Release v0.20.0 2019-10-22 09:39:14 +03:00
Stefan Prodan
ee05108279 Merge pull request #344 from weaveworks/gloo-refactoring
Gloo integration refactoring
2019-10-22 09:38:19 +03:00
stefanprodan
dfaa039c9c Update Goo docs 2019-10-22 00:48:15 +03:00
stefanprodan
46579d2ee6 Refactor Gloo integration
- build Gloo UpstreamGroup clientset
- drop solo-io, envoyproxy, hcl, consul, opencensus, apiextensions deps
- use the native routers with supergloo
2019-10-21 16:33:47 +03:00
Stefan Prodan
f372523fb8 Merge pull request #342 from weaveworks/prom-config
Implement metrics server override
2019-10-17 17:24:24 +03:00
stefanprodan
5e434df6ea Exclude high cardinality cAdvisor metrics 2019-10-17 13:02:18 +03:00
stefanprodan
d6c5bdd241 Implement metrics server override 2019-10-17 11:37:54 +03:00
stefanprodan
cdcd97244c Add the metrics server field to CRD 2019-10-17 11:36:25 +03:00
Stefan Prodan
60c4bba263 Merge pull request #340 from weaveworks/appmesh-ab-testing
Implement App Mesh A/B testing
2019-10-17 10:54:31 +03:00
stefanprodan
2b73bc5e38 Fix A/B testing examples 2019-10-17 09:12:39 +03:00
stefanprodan
03652dc631 Add App Mesh http match headers tests 2019-10-16 15:43:26 +03:00
stefanprodan
00155aff37 Add App Mesh A/B testing example to docs 2019-10-16 10:49:33 +03:00
stefanprodan
206c3e6d7a Implement App Mesh A/B testing 2019-10-15 16:39:54 +03:00
Stefan Prodan
8345fea812 Merge pull request #338 from weaveworks/appmesh-up
Implement App Mesh HTTP retry policy
2019-10-15 08:45:49 +03:00
stefanprodan
c11dba1e05 Add retry policy to docs and examples 2019-10-14 21:03:57 +03:00
stefanprodan
7d4c3c5814 Implement App Mesh HTTP retry policy 2019-10-14 20:27:48 +03:00
stefanprodan
9b36794c9d Update App Mesh CRD 2019-10-14 20:26:46 +03:00
Stefan Prodan
1f34c656e9 Merge pull request #336 from weaveworks/appmesh-router-fix
Generate unique names for App Mesh virtual routers and routes
2019-10-14 19:25:08 +03:00
stefanprodan
9982dc9c83 Generate unique names for App Mesh virtual routers and routes 2019-10-14 19:07:10 +03:00
Stefan Prodan
780f3d2ab9 Merge pull request #334 from weaveworks/env-vars
Allow setting Slack and Teams URLs with env vars
2019-10-10 09:05:04 +03:00
stefanprodan
1cb09890fb Add env to chart options to be used for Slack and Teams URLs 2019-10-09 16:53:34 +03:00
stefanprodan
faae6a7c3b Add env vars for Slack and Teams URLs 2019-10-09 16:03:30 +03:00
Stefan Prodan
d4250f3248 Merge pull request #333 from weaveworks/default-labels
Add the app/name label to services and primary deployment
2019-10-09 13:45:14 +03:00
stefanprodan
a8ee477b62 Add selector labels option to Helm chart 2019-10-09 13:22:10 +03:00
stefanprodan
673b6102a7 Add the name label to ClusterIP services and primary deployment 2019-10-09 13:01:15 +03:00
Stefan Prodan
316de42a2c Merge pull request #331 from weaveworks/prep-v0.19.0
Release v0.19.0
2019-10-08 13:22:16 +03:00
stefanprodan
dfb4b35e6c Release v0.19.0 2019-10-08 12:02:37 +03:00
Stefan Prodan
61ab596d1b Merge pull request #327 from weaveworks/target-port
Implement canary service target port
2019-10-08 11:10:04 +03:00
stefanprodan
3345692751 Add service target port to docs 2019-10-07 11:56:03 +03:00
stefanprodan
dff9287c75 Add target port to NGINX e2e tests 2019-10-07 10:01:28 +03:00
stefanprodan
b5fb7cdae5 Add target port number to Gloo e2e tests
Update Gloo to v0.20.2
Enable Gloo discovery Fix: #328
2019-10-07 09:34:23 +03:00
stefanprodan
2e79817437 Add target port number e2e test for Linkerd 2019-10-06 13:35:58 +03:00
stefanprodan
5f439adc36 Use kustomize in Linkerd e2e tests 2019-10-06 12:58:26 +03:00
stefanprodan
45df96ff3c Format imports 2019-10-06 12:54:01 +03:00
stefanprodan
98ee150364 Add target port and gPRC e2e tests for Linkerd 2019-10-06 12:26:03 +03:00
stefanprodan
d328a2146a Fix loadtester image tag 2019-10-06 11:43:25 +03:00
stefanprodan
4513f2e8be Use Docker Hub in e2e tests 2019-10-06 11:42:49 +03:00
stefanprodan
095fef1de6 Release loadtester v0.9.0 with gRPC health check 2019-10-06 11:26:42 +03:00
stefanprodan
754f02a30f Add gRPC acceptance test to Istio e2e tests 2019-10-06 11:03:00 +03:00
stefanprodan
01a4e7f6a8 Add service target port to Istio e2e tests 2019-10-06 11:02:05 +03:00
stefanprodan
6bba84422d Add service target port to Kubernetes e2e tests 2019-10-06 10:44:42 +03:00
stefanprodan
26190d0c6a Use podinfo v3.1.0 for e2e tests 2019-10-06 10:42:30 +03:00
stefanprodan
2d9098e43c Add target port number and name tests 2019-10-06 10:31:50 +03:00
stefanprodan
7581b396b2 Implement service target port 2019-10-06 10:21:34 +03:00
stefanprodan
67a6366906 Add service.targetPort field to Canary CRD 2019-10-06 10:04:21 +03:00
Stefan Prodan
5605fab740 Merge pull request #326 from weaveworks/force-bg
Enforce blue/green when using kubernetes networking
2019-10-05 18:55:13 +03:00
stefanprodan
b76d0001ed Move Istio routing docs to FAQ 2019-10-05 18:13:40 +03:00
stefanprodan
625eed0840 Enforce blue/green when using kubernetes networking
Use blue/green with ten iterations and warn that progressive traffic shifting and HTTP headers routing are not compatible with Kubernetes L4 networking.
2019-10-05 17:59:34 +03:00
stefanprodan
37f9151de3 Add traffic mirroring documentation 2019-10-05 16:23:43 +03:00
Stefan Prodan
20af98e4dc Merge pull request #325 from weaveworks/appmesh-grcp
Allow gPRC protocol for App Mesh
2019-10-05 12:49:07 +03:00
stefanprodan
76800d0ed0 Update canary spec in docs 2019-10-05 12:15:54 +03:00
stefanprodan
3103bde7f7 Use the App Mesh Prometheus chart in docs 2019-10-05 11:52:41 +03:00
stefanprodan
298d8c2d65 Allow gPRC protocol for App Mesh
Use the canary service port name to set http or grpc protocol on App Mesh virtual nodes and virtual routers
2019-10-05 11:21:43 +03:00
Stefan Prodan
5cdacf81e3 Merge pull request #324 from weaveworks/fix-ports-order
Fix port discovery diff
2019-10-05 11:03:35 +03:00
stefanprodan
2141d88ce1 Enable Prometheus scraping of Flagger metrics 2019-10-05 10:45:35 +03:00
stefanprodan
e8a2d4be2e Fix port discovery diff
Sort service ports by port number before comparing slices
2019-10-05 10:42:01 +03:00
Stefan Prodan
9a9baadf0e Merge pull request #311 from andrewjjenkins/mirror
Add traffic mirroring for Istio service mesh
2019-10-05 10:34:25 +03:00
Andrew Jenkins
a21e53fa31 Document traffic mirroring in the FAQ 2019-10-03 14:33:49 -06:00
Andrew Jenkins
61f8aea7d8 add Traffic Mirroring to Blue/Green deployments
Traffic mirroring for blue/green will mirror traffic for the entire
canary analysis phase of the blue/green deployment.
2019-10-03 14:33:49 -06:00
Andrew Jenkins
e384b03d49 Add Traffic Mirroring for Istio Service Mesh
Traffic mirroring is a pre-stage for canary deployments.  When mirroring
is enabled, at the beginning of a canary deployment traffic is mirrored
to the canary instead of shifted for one canary period.  The service
mesh should mirror by copying the request and sending one copy to the
primary and one copy to the canary; only the response from the primary
is sent to the user.  The response from the canary is only used for
collecting metrics.

Once the mirror period is over, the canary proceeds as usual, shifting
traffic from primary to canary until complete.

Added TestScheduler_Mirroring unit test.
2019-10-03 14:33:49 -06:00
Stefan Prodan
0c60cf39f8 Merge pull request #323 from weaveworks/prep-0.18.6
Release v0.18.6
2019-10-03 15:19:51 +03:00
stefanprodan
268fa9999f Release v0.18.6 2019-10-03 15:00:12 +03:00
stefanprodan
ff7d4e747c Update Linkerd to v2.5.0 2019-10-03 14:48:26 +03:00
stefanprodan
121fc57aa6 Update Prometheus to v2.12.0 2019-10-03 14:46:34 +03:00
Stefan Prodan
991fa1cfc8 Merge pull request #322 from weaveworks/appmesh-acceptance-testing
Add support for acceptance testing when using App Mesh
2019-10-03 14:31:51 +03:00
stefanprodan
fb2961715d Add App Mesh acceptance tests example to docs 2019-10-03 12:11:11 +03:00
stefanprodan
74c1c2f1ef Add App Mesh request duration metric check to docs
Fix: #143 depend on App Mesh Envoy >1.11
2019-10-03 11:52:56 +03:00
stefanprodan
4da6c1b6e4 Create canary virtual service during App Mesh reconciliation
Allows the canary pods to be accessed from inside the mesh during the canary analysis for conformance and load testing
2019-10-03 11:43:47 +03:00
Stefan Prodan
fff03b170f Merge pull request #320 from bvwells/json-tag
Fix JSON tag on virtual node condition
2019-10-03 11:07:05 +03:00
Stefan Prodan
434acbb71b Merge pull request #319 from weaveworks/appmesh-docs
Update App Mesh install docs
2019-10-03 10:55:45 +03:00
Ben Wells
01962c32cd Fix JSON tag on virtual node condition 2019-10-03 08:46:39 +01:00
stefanprodan
6b0856a054 Update App Mesh Envoy ingress to v1.11.1 2019-10-03 10:02:58 +03:00
stefanprodan
708dbd6bbc Use official App Mesh Helm charts in docs 2019-10-03 09:52:42 +03:00
Stefan Prodan
e3801cbff6 Merge pull request #318 from bvwells/notifier-fields
Fix slack/teams notification fields
2019-10-03 09:50:25 +03:00
Ben Wells
fc68635098 Fix slack/teams notification of fields 2019-10-02 22:35:16 +01:00
Stefan Prodan
6706ca5d65 Merge pull request #317 from weaveworks/appmesh-kustomize
Add Kustomize installer for App Mesh
2019-10-02 21:40:04 +03:00
stefanprodan
44c2fd57c5 Add App Mesh Kustomize installer to docs 2019-10-02 20:12:04 +03:00
stefanprodan
a9aab3e3ac Add Kustomize installer for App Mesh 2019-10-02 20:05:52 +03:00
Stefan Prodan
6478d0b6cf Merge pull request #316 from weaveworks/prep-0.18.5
Release v0.18.5
2019-10-02 18:10:01 +03:00
stefanprodan
958af18dc0 Add changelog for v0.18.5 2019-10-02 17:51:06 +03:00
stefanprodan
54b8257c60 Release v0.18.5 2019-10-02 16:51:08 +03:00
Stefan Prodan
e86f62744e Merge pull request #315 from nilscan/appmesh-init
Skip primary check for appmesh
2019-10-02 09:17:08 +03:00
nilscan
0734773993 Skip primary check for appmesh 2019-10-02 14:29:48 +13:00
Stefan Prodan
888cc667f1 Merge pull request #314 from weaveworks/podinfo-updates
Update podinfo to v3.1.0 and go to v1.13
2019-09-27 17:20:52 +03:00
stefanprodan
053d0da617 Remove thrift replace from go.mod 2019-09-27 16:59:15 +03:00
stefanprodan
7a4e0bc80c Update go mod to 1.13 2019-09-27 16:53:55 +03:00
stefanprodan
7b7306584f Update alpine to 3.10 2019-09-27 16:33:56 +03:00
stefanprodan
d6027af632 Update go to 1.13 in CI 2019-09-27 16:33:06 +03:00
stefanprodan
761746af21 Update podinfo to v3.1.0 2019-09-27 15:52:30 +03:00
stefanprodan
510a6eaaed Add JWT token issuing test to podinfo chart 2019-09-27 15:19:03 +03:00
Andrew Jenkins
655df36913 Extend test SetupMocks() to take arbitrary Canary resources
SetupMocks() currently takes a bool switch that tells it to configure
against either a shifting canary or an A-B canary.  I'll need a third
canary that has mirroring turned on so I changed this to an interface
that just takes the canary to configure (and configs the default
shifting canary if you pass nil).
2019-09-24 16:15:45 -06:00
Andrew Jenkins
2e079ba7a1 Add mirror to router interface and implement for istio router
The mirror option will be used to tell routers to configure traffic
mirroring.  Implement mirror for GetRoutes and SetRoutes for Istio.  For
other routers, GetRoutes always returns mirror == false, and SetRoutes
ignores mirror.

After this change there is no behavior change because no code sets
mirror true (yet).

Enhanced TestIstioRouter_SetRoutes and TestIstioRouter_GetRoutes.
2019-09-24 16:15:45 -06:00
Stefan Prodan
9df6bfbb5e Merge pull request #310 from weaveworks/canary-promotion
Canary promotion improvements
2019-09-24 14:19:43 +03:00
stefanprodan
2ff86fa56e Fix canary weight max value 2019-09-24 10:16:22 +03:00
stefanprodan
1b2e0481b9 Add promoting phase to status condition 2019-09-24 09:57:42 +03:00
stefanprodan
fe96af64e9 Add canary phases tests 2019-09-23 22:24:40 +03:00
stefanprodan
77d8e4e4d3 Use the promotion phase in A/B testing and Blue/Green 2019-09-23 22:14:44 +03:00
stefanprodan
800b0475ee Run the canary promotion on a separate stage
After the analysis finishes, Flagger will do the promotion and wait for the primary rollout to finish before routing all the traffic back to it. This ensures a smooth transition to the new version avoiding dropping in-flight requests.
2019-09-23 21:57:24 +03:00
stefanprodan
b58e13809c Add promoting phase to canary status conditions 2019-09-23 21:48:09 +03:00
Stefan Prodan
9845578cdd Merge pull request #307 from weaveworks/confirm-promotion
Implement confirm-promotion hook
2019-09-23 12:32:52 +03:00
stefanprodan
96ccfa54fb Add confirm-promotion hook example to docs 2019-09-22 14:14:35 +03:00
stefanprodan
b8a64c79be Add confirm-promotion webhook to e2e tests 2019-09-22 13:44:55 +03:00
stefanprodan
4a4c261a88 Add confirm-promotion webhook type to CRD 2019-09-22 13:36:07 +03:00
stefanprodan
8282f86d9c Implement confirm-promotion hook
The confirm promotion hooks are executed right before the promotion step. The canary promotion is paused until the hooks return HTTP 200. While the promotion is paused, Flagger will continue to run the metrics checks and load tests.
2019-09-22 13:23:19 +03:00
Stefan Prodan
2b6966d8e3 Merge pull request #306 from weaveworks/e2e-updates
Update end-to-end tests to Istio 1.3.0
2019-09-22 12:37:05 +03:00
stefanprodan
c667c947ad Istio e2e: update job names 2019-09-22 12:12:06 +03:00
stefanprodan
105b28bf42 Update e2e to Kind 0.5.1 and Istio to 1.3.0 2019-09-22 12:05:35 +03:00
Stefan Prodan
37a1ff5c99 Merge pull request #305 from weaveworks/service-mesh-blue-green
Implement B/G for service mesh providers
2019-09-22 12:01:10 +03:00
stefanprodan
d19a070faf Add canary status checks to Istio e2e tests 2019-09-22 11:45:07 +03:00
stefanprodan
d908355ab3 Add Blue/Green e2e tests 2019-09-22 09:32:25 +03:00
stefanprodan
a6d86f2e81 Skip mesh routers for B/G when provider is kubernetes 2019-09-22 00:48:42 +03:00
stefanprodan
9d856a4f96 Implement B/G for service mesh providers
Blue/Green steps:
- scale up green
- run conformance tests on green
- run load tests and metric checks on green
- route traffic to green
- promote green spec over blue
- wait for blue rollout
- route traffic to blue
2019-09-21 21:21:33 +03:00
Stefan Prodan
a7112fafb0 Merge pull request #304 from nilscan/pod-annotations
Add pod annotations on all deployments
2019-09-19 02:11:30 +01:00
nilscan
93f9e51280 Add pod annotations on all deployments 2019-09-19 12:42:22 +12:00
Stefan Prodan
65e9a402cf Merge pull request #297 from weaveworks/prep-0.18.4
Release v0.18.4
2019-09-08 11:37:47 +03:00
stefanprodan
f7513b33a6 Release v0.18.4 2019-09-08 11:21:16 +03:00
Stefan Prodan
0b3fa517d3 Merge pull request #296 from weaveworks/helmv3-tester
Implement Helm v3 tester
2019-09-08 09:49:52 +03:00
stefanprodan
507075920c Implement Helm v3 tester 2019-09-08 09:33:34 +03:00
Stefan Prodan
a212f032a6 Merge pull request #295 from weaveworks/grpc-hc
Add gPRC health check to load tester
2019-09-06 17:00:22 +03:00
stefanprodan
eb8755249f Update cert-manager to v0.10 2019-09-06 16:44:39 +03:00
stefanprodan
73bb2a9fa2 Release loadtester 0.7.1 2019-09-06 16:21:22 +03:00
stefanprodan
5d3ffa8c90 Add grpc_health_probe to load tester image 2019-09-06 16:19:23 +03:00
Stefan Prodan
87f143f5fd Merge pull request #293 from kislitsyn/nginx-annotations-prefix
Add annotations prefix for ingresses
2019-09-06 13:22:42 +03:00
Anton Kislitcyn
f56b6dd6a7 Add annotations prefix for ingresses 2019-09-06 11:36:06 +02:00
Stefan Prodan
5e40340f9c Merge pull request #289 from nilscan/owide
Add Wide columns in CRD
2019-09-04 14:59:17 +03:00
nilscan
2456737df7 Add Wide columns in CRD 2019-09-03 12:54:14 +12:00
stefanprodan
1191d708de Fix Prometheus GKE install docs 2019-08-30 13:13:36 +03:00
Stefan Prodan
4d26971fc7 Merge pull request #286 from jwenz723/patch-1
Enhanced error logging
2019-08-29 09:14:16 +03:00
Jeff Wenzbauer
0421b32834 Enhanced error logging
Updated the formatting of the `out` to be logged as a string rather than a bunch of bytes.
2019-08-28 12:43:08 -06:00
Stefan Prodan
360dd63e49 Merge pull request #282 from weaveworks/prep-0.18.3
Release 0.18.3
2019-08-22 18:53:15 +03:00
stefanprodan
f1670dbe6a Add 0.18.3 changelog 2019-08-22 18:39:47 +03:00
stefanprodan
e7ad5c0381 Release load tester v0.7.0 2019-08-22 18:31:05 +03:00
stefanprodan
2cfe2a105a Release Flagger v0.18.3 2019-08-22 18:30:46 +03:00
Stefan Prodan
bc83cee503 Merge pull request #278 from mjallday/patch-1
Embedding Health Check Protobuf
2019-08-22 18:19:58 +03:00
Stefan Prodan
5091d3573c Merge pull request #281 from weaveworks/fix-appmesh-crd
Fix App Mesh backends validation in CRD
2019-08-22 10:02:38 +03:00
Marshall Jones
ffe5dd91c5 Add an example and fix path to downloaded proto file 2019-08-21 15:15:01 -07:00
stefanprodan
d76b560967 Bump podinfo version in the App Mesh demo 2019-08-21 21:52:36 +03:00
stefanprodan
f062ef3a57 Fix App Mesh backends validation in CRD 2019-08-21 21:45:36 +03:00
Stefan Prodan
5fc1baf4df Merge pull request #280 from vbehar/loadtester-helm-tillerless
loadtester: add support for tillerless helm
2019-08-21 17:25:44 +03:00
Vincent Behar
777b77b69e loadtester: add support for tillerless helm
- upgrade helm to 2.14, and install the [helm-tiller](https://github.com/rimusz/helm-tiller) plugin to run in "tillerless" mode - with a local tiller instance
- also add support to create RBAC resources in the loadtester chart, because when running in tillerless mode, the pod service account will be used instead of the tiller one - so we need to give him specific permissions

this allow the use of the loadtester to run `helm test` in tillerless mode, with `helm tiller run -- helm test` for example
2019-08-21 15:54:49 +02:00
Marshall Jones
5d221e781a Propose Embedding Health Check Proto
Copy this file https://github.com/grpc/grpc-proto/blob/master/grpc/health/v1/health.proto into the ghz folder for use when promoting a canary running a grpc service. 

This repo describes the file:

> This repository contains common protocol definitions for peripheral services around gRPC such as health checking, load balancing etc..

Any app that chooses to implement this interface (which imo should be any grpc service exposing a health check) will then be able to use this without providing reflection. 

I'm not a grpc expert so I'm not sure what the best practices are around allowing reflection on the server but this seems like a simple solution for those who choose not to enable it.

Slack discussion on the weave users slack is here - https://weave-community.slack.com/archives/CGLQLLH9Q/p1566358441123400

You can utilize this file like so 

`/ghz --proto=/tmp/health.proto --call=grpc.health.v1.Health/Check ...`
2019-08-20 20:47:30 -07:00
Stefan Prodan
ddab72cd59 Merge pull request #276 from weaveworks/podinfo
Update podinfo to v2.0
2019-08-14 10:46:06 +03:00
stefanprodan
87d0b33327 Add provider field to nginx and gloo docs 2019-08-14 10:14:00 +03:00
stefanprodan
225a9015bb Update podinfo to v2.0 2019-08-14 09:28:36 +03:00
Stefan Prodan
c0b60b1497 Merge pull request #272 from weaveworks/appmesh
Set HTTP listeners for AppMesh virtual routers
2019-08-13 09:48:49 +03:00
Stefan Prodan
0463c19825 Merge pull request #275 from hiddeco/build/codegen
Support non `$GOPATH/src` location for codegen
2019-08-13 09:48:27 +03:00
Hidde Beydals
8e70aa90c1 Support non $GOPATH/src location for codegen
This commit fixes two things:

- it ensures the code generation works no matter the location of the
  project directory
- as a side effect; fixes the `hack/verify-codegen.sh` ran during CI

The previous script (or more specific: the `code-generator` library)
made the assumption during execution that the project was placed
inside `$GOPATH/src` and made the modifications there.

The idea of Go Modules is however that a project and/or package can
be placed anywhere, and this is also what the CI did, resulting in a
comparison of two identical `cp -r` copied directories. Giving a
false green light on every CI run.

To work around this limitation in `code-generator`: create a
temporary directory, use this as an output base and copy
everything back once generated.
2019-08-12 22:41:10 +02:00
stefanprodan
0a418eb88a Add notifier tests 2019-08-12 09:47:11 +03:00
stefanprodan
040dbb8d03 Add http listener to virtual router reconciliation 2019-08-10 11:04:15 +03:00
stefanprodan
64f2288bdd Add listeners to AppMesh virtual router 2019-08-10 10:58:20 +03:00
Stefan Prodan
8008562a33 Merge pull request #271 from weaveworks/crd
Add missing fields to CRD validation spec
2019-08-07 11:09:07 +03:00
stefanprodan
a39652724d Add confirm and pre rollout hooks to e2e tests 2019-08-07 10:55:15 +03:00
stefanprodan
691c3c4f36 Add missing fields to CRD validation spec 2019-08-07 10:22:07 +03:00
Stefan Prodan
f6fa5e3891 Merge pull request #270 from weaveworks/prep-0.18.2
Release v0.18.2
2019-08-05 18:57:54 +03:00
stefanprodan
a305a0b705 Release v0.18.2 2019-08-05 18:43:57 +03:00
Stefan Prodan
dfe619e2ea Merge pull request #269 from weaveworks/helm-circleci
Publish Helm chart from CircleCI
2019-08-05 17:57:21 +03:00
stefanprodan
2b3d425b70 Publish Helm chart from CircleCI 2019-08-05 17:08:33 +03:00
Stefan Prodan
6e55fea413 Merge pull request #268 from weaveworks/istio-1.2.3
Update e2e backends
2019-08-03 15:44:53 +03:00
stefanprodan
b6a08b6615 Fix AppMesh mesh name in docs 2019-08-03 15:24:31 +03:00
stefanprodan
eaa6906516 Update e2e NGINX ingress to v1.12.1 2019-08-03 13:42:27 +03:00
stefanprodan
62a7a92f2a Update e2e Gloo to v0.18.8 2019-08-03 13:01:57 +03:00
stefanprodan
3aeb0945c5 Update e2e Istio to v1.2.3 2019-08-03 12:05:21 +03:00
Stefan Prodan
e8c85efeae Merge pull request #267 from fcantournet/fix_virtualservices_multipleports
Fix Port discovery with multiple port services
2019-08-03 12:04:04 +03:00
Félix Cantournet
6651f6452b Multiple port canary: fix FAQ and add e2e tests 2019-08-02 14:23:58 +02:00
Félix Cantournet
0ca48d77be Fix Port discovery with multiple port services
This fixes issue https://github.com/weaveworks/flagger/issues/263

We actually don't need to specify any ports in the VirtualService
and DestinationRules.
Istio will create clusters/listerners for each named port we have declared in
the kubernetes services and the router can be shared as it operates only on L7 criterias

Also contains a tiny clean-up of imports
2019-08-02 10:07:00 +02:00
Stefan Prodan
a9e0e018e3 Merge pull request #266 from ExpediaInc/master
Parameterize image pull secrets for private docker repos
2019-08-01 11:07:53 +03:00
Sky Moon
122d11f445 Merge pull request #1 from ExpediaInc/parameterizeImagePullSecrets
parameterize image pull secrets for private docker repos.
2019-08-01 00:50:15 -07:00
cmoon
b03555858c parameterize image pull secrets for private docker repos. 2019-08-01 00:47:07 -07:00
Stefan Prodan
dcc5a40441 Merge pull request #262 from weaveworks/prep-0.18.1
Release v0.18.1
2019-07-30 13:52:25 +03:00
stefanprodan
8c949f59de Package helm charts locally 2019-07-30 13:35:09 +03:00
stefanprodan
e8d91a0375 Release v0.18.1 2019-07-30 13:22:51 +03:00
Stefan Prodan
fae9aa664d Merge pull request #261 from weaveworks/blue-green-e2e
Fix Blue/Green metrics provider and add e2e tests
2019-07-30 13:16:20 +03:00
stefanprodan
c31e9e5a96 Use Linkerd metrics for ingress and kubernetes routers 2019-07-30 13:00:28 +03:00
stefanprodan
99fff98274 Kustomize: set Flagger log level to info 2019-07-30 12:43:02 +03:00
stefanprodan
11d84bf35d Enable kubernetes metric provider 2019-07-30 12:27:53 +03:00
stefanprodan
e56ba480c7 Add Blue/Green e2e tests 2019-07-30 12:02:15 +03:00
Stefan Prodan
b9f0517c5d Merge pull request #255 from weaveworks/prep-0.18.0
Release v0.18.0
2019-07-29 16:06:23 +03:00
stefanprodan
6e66f02585 Update changelog 2019-07-29 15:52:50 +03:00
stefanprodan
5922e96044 Merge branch 'prep-0.18.0' of https://github.com/weaveworks/flagger into prep-0.18.0 2019-07-29 15:06:43 +03:00
stefanprodan
f36e7e414a Add manual gating link to readme 2019-07-29 15:06:31 +03:00
stefanprodan
606754d4a5 Disable supergloo e2e 2019-07-29 15:06:31 +03:00
stefanprodan
a3847e64df Add Kustomize download link to docs 2019-07-29 15:06:31 +03:00
stefanprodan
7a3f9f2e73 Use Kustomize for Istio e2e testing 2019-07-29 15:06:31 +03:00
stefanprodan
2e4e8b0bf9 Make installer work with Kustomize v3 2019-07-29 15:06:31 +03:00
stefanprodan
951fe80115 Use crd.create=false in docs 2019-07-29 15:06:30 +03:00
stefanprodan
c0a8149acb Add kubectl min version to Kustomize docs 2019-07-29 15:06:30 +03:00
stefanprodan
80b75b227d Add CRD install step to chart 2019-07-29 15:06:30 +03:00
stefanprodan
dff7de09f2 Use kubectl for CRD install 2019-07-29 15:06:30 +03:00
stefanprodan
b3bbadfccf Add v0.18.0 to changelog 2019-07-29 15:06:30 +03:00
stefanprodan
fc676e3cb7 Release v0.18.0 2019-07-29 15:06:30 +03:00
stefanprodan
860c82dff9 Remove test artifacts 2019-07-29 15:06:30 +03:00
Stefan Prodan
4829f5af7f Merge pull request #257 from weaveworks/promotion
Implement promotion finalising state
2019-07-29 15:03:05 +03:00
stefanprodan
c463b6b231 Add finalising state tests 2019-07-29 14:02:16 +03:00
stefanprodan
b2ca0c4c16 Implement finalising state
Set the canary status to finalising after routing the traffic back to the primary. Run one final loop before scaling the canary to zero so that the canary has a chance to process all inflight requests.
2019-07-29 13:52:11 +03:00
stefanprodan
69875cb3dc Add finalising status phase to CRD 2019-07-29 13:43:30 +03:00
stefanprodan
9e33a116d4 Add manual gating link to readme 2019-07-29 11:33:28 +03:00
stefanprodan
dab3d53b65 Disable supergloo e2e 2019-07-28 11:28:00 +03:00
stefanprodan
e3f8bff6fc Add Kustomize download link to docs 2019-07-27 15:51:22 +03:00
stefanprodan
0648d81d34 Use Kustomize for Istio e2e testing 2019-07-27 14:49:57 +03:00
stefanprodan
ece5c4401e Make installer work with Kustomize v3 2019-07-27 14:45:49 +03:00
stefanprodan
bfc64c7cf1 Use crd.create=false in docs 2019-07-27 13:20:55 +03:00
stefanprodan
0a2c134ece Add kubectl min version to Kustomize docs 2019-07-27 13:07:47 +03:00
stefanprodan
8bea9253c3 Add CRD install step to chart 2019-07-27 13:06:27 +03:00
stefanprodan
e1dacc3983 Use kubectl for CRD install 2019-07-26 15:52:00 +03:00
stefanprodan
0c6a7355e7 Add v0.18.0 to changelog 2019-07-26 14:05:19 +03:00
stefanprodan
83046282c3 Release v0.18.0 2019-07-26 13:53:40 +03:00
stefanprodan
65c9817295 Remove test artifacts 2019-07-26 13:51:15 +03:00
Stefan Prodan
e4905d3d35 Merge pull request #254 from weaveworks/podinfo
Use Kustomize installer in Linkerd docs
2019-07-26 13:44:51 +03:00
stefanprodan
6bc0670a7a Use Kustomize installer in Linkerd docs 2019-07-26 13:30:28 +03:00
stefanprodan
95ff6adc19 Use podinfo 1.7 in GitOps demo 2019-07-26 13:20:06 +03:00
stefanprodan
7ee51c7def Add podinfo to Kustomize installer 2019-07-26 13:19:36 +03:00
Stefan Prodan
dfa065b745 Merge pull request #251 from weaveworks/gates
Implement confirm rollout gate, hook and API
2019-07-26 01:40:35 +03:00
stefanprodan
e3b03debde Use podinfo v1.7 2019-07-26 01:25:44 +03:00
Stefan Prodan
ef759305cb Merge pull request #253 from grampelberg/master
Update Linkerd to use correct canaries directory.
2019-07-26 00:24:52 +03:00
grampelberg
ad65497d4e Update Linkerd to use correct canaries directory. 2019-07-25 11:10:52 -07:00
stefanprodan
163f5292b0 Push a notification when a canary is waiting for approval 2019-07-25 19:13:22 +03:00
stefanprodan
e07a82d024 Add manual gating to docs 2019-07-25 13:32:58 +03:00
stefanprodan
046245a8b5 Use Gloo 0.17.6 in e2e tests 2019-07-24 19:54:33 +03:00
stefanprodan
aa6a180bcc Remove Gloo NodePort from e2e tests 2019-07-24 19:44:06 +03:00
stefanprodan
c4d28e14fc Upgrade Gloo e2e to v0.17.5 2019-07-24 19:35:02 +03:00
stefanprodan
bc4bdcdc1c Upgrade Gloo e2e to v0.17.6 2019-07-24 19:21:41 +03:00
stefanprodan
be22ff9951 Bump load tester version 2019-07-24 16:28:46 +03:00
stefanprodan
f204fe53f4 Implement canary gating API with in-memory storage
POST /gate/[check|open|close]
2019-07-24 16:14:22 +03:00
stefanprodan
28e7e89047 Pause or resume analysis on confirmation gate toggle 2019-07-24 16:09:13 +03:00
stefanprodan
75d49304f3 Add confirm-rollout hook to docs 2019-07-24 12:17:11 +03:00
stefanprodan
04cbacb6e0 Implement confirm rollout gate and hook
The confirm-rollout hooks are executed before the pre-rollout hooks. Flagger will halt the canary rollout until the confirm webhook returns HTTP status 200.
2019-07-24 12:09:39 +03:00
stefanprodan
c46c7b9e21 Add canary status conditions to docs 2019-07-24 12:04:05 +03:00
stefanprodan
919dafa567 Add gate halt and approve endpoints 2019-07-24 12:02:44 +03:00
stefanprodan
dfdcfed26e Add Waiting canary status phase
means the canary rollout is paused (waiting for confirmation to proceed)
2019-07-24 12:00:04 +03:00
Stefan Prodan
a0a4d4cfc5 Merge pull request #248 from weaveworks/ghz
Add gRPC load testing tool
2019-07-23 12:44:04 +03:00
stefanprodan
970a589fd3 Add load tester to kustomize installer 2019-07-23 12:30:38 +03:00
stefanprodan
56d2c0952a Add gPRC load test example to docs 2019-07-22 15:16:13 +03:00
stefanprodan
4871be0345 Release loadtester v0.5.0 2019-07-22 14:57:14 +03:00
stefanprodan
e3e112e279 Add gRPC load testing tool
https://ghz.sh
2019-07-22 14:55:19 +03:00
Stefan Prodan
d2cbd40d89 Merge pull request #240 from weaveworks/refactor
Refactor canary change detection and status
2019-07-22 14:33:02 +03:00
stefanprodan
3786a49f00 Update Linkerd e2e to v2.4.0 2019-07-16 11:20:42 +02:00
stefanprodan
ff4aa62061 Retry canary status update on conflict 2019-07-10 11:31:20 +03:00
stefanprodan
9b6cfdeef7 Update Canary CRD helm chart and Kustomize 2019-07-10 09:55:46 +03:00
stefanprodan
9d89e0c83f Log status update error 2019-07-10 09:55:20 +03:00
stefanprodan
559cbd0d36 Pin NGINX helm chart to v1.8.2 2019-07-10 09:49:39 +03:00
stefanprodan
caea00e47f Pin NGINX helm chart to version 1.8.2 2019-07-10 09:42:49 +03:00
stefanprodan
b26542f38d Do not trigger a canary deployment on manual rollback
Save the primary spec hash and check if it matches the canary spec. If the canary hash is identical with the primary one skip promotion.
2019-07-10 09:08:33 +03:00
Stefan Prodan
bbab7ce855 Merge pull request #238 from weaveworks/prep-0.17.0
Release v0.17.0
2019-07-10 08:46:10 +03:00
stefanprodan
afa2d079f6 Add status conditions and descriptions to CRD 2019-07-09 17:11:13 +03:00
stefanprodan
108bf9ca65 Add initializing canary phase/status condition reason
Fix HPA reconciliation min replicas diff
2019-07-09 17:10:43 +03:00
stefanprodan
438f952128 Implement status conditions
Add Promoted status condition with the following reasons: Initialized, Progressing, Succeeded, Failed
Usage: `kubectl wait canary/app --for=condition=promoted`
Fix: #184
2019-07-09 15:22:56 +03:00
stefanprodan
3e84799644 Detect changes in pod template metadata
Use the pod template spec hash to track changes (breaking)
2019-07-09 08:52:31 +03:00
stefanprodan
d6e80bac7f Update webhook mTLS FAQ
Fix: #239
2019-07-08 17:21:59 +03:00
stefanprodan
9b3b24bddf Add v0.17.0 changelog 2019-07-08 15:02:09 +03:00
stefanprodan
5c831ae482 Add Linkerd to docs 2019-07-08 13:43:01 +03:00
stefanprodan
78233fafd3 Release v0.17.0 2019-07-08 13:35:36 +03:00
Stefan Prodan
73c3e07859 Merge pull request #236 from weaveworks/leader-election
Implement leader election
2019-07-08 11:32:06 +03:00
stefanprodan
10c61daee4 Exit when losing leadership 2019-07-07 13:09:05 +03:00
stefanprodan
b1bb9fa114 Enable leader election for e2e testing 2019-07-07 12:19:09 +03:00
stefanprodan
a7f4b6d2ae Add leader election and pod anti affinity to chart 2019-07-07 12:08:08 +03:00
stefanprodan
b937c4ea8d Implement leader election
Add enable-leader-election and leader-election-namespace flags
2019-07-07 11:41:27 +03:00
Stefan Prodan
e577311b64 Merge pull request #235 from weaveworks/msteams
Implement MS Teams notifications
2019-07-07 11:40:00 +03:00
stefanprodan
b847345308 Add 5 seconds timeout to notifier 2019-07-06 18:02:45 +03:00
stefanprodan
85e683446f Add MS Teams to docs 2019-07-06 17:22:39 +03:00
stefanprodan
4f49aa5760 Add MS Teams webhook field to chart 2019-07-06 17:14:50 +03:00
stefanprodan
8ca9cf24bb Implement MS Teams notifier 2019-07-06 17:14:21 +03:00
stefanprodan
61d0216c21 Add traffic routing to notifications 2019-07-06 17:13:09 +03:00
stefanprodan
ba4a2406ba Refactor notifier to allow more implementations 2019-07-06 15:47:12 +03:00
Stefan Prodan
c2974416b4 Merge pull request #234 from weaveworks/psp
Add pod security policy to Helm chart
2019-07-06 11:52:58 +03:00
stefanprodan
48fac4e876 Disable privilege escalation 2019-07-06 10:38:17 +03:00
stefanprodan
f0add9a67c Use a role binding for the PSP rbac 2019-07-05 17:10:05 +03:00
stefanprodan
20f9df01c2 Add pod security policy to Helm chart
- disable privileged, hostIPC, hostNetwork and hostPID
- add psp flag to chart readme
2019-07-05 16:47:41 +03:00
Stefan Prodan
514e850072 Merge pull request #232 from weaveworks/kustomize
Add Kustomize installer
2019-07-05 16:23:44 +03:00
stefanprodan
61fe78a982 Mention Prometheus data retention in docs 2019-07-04 15:07:05 +03:00
stefanprodan
c4b066c845 Add Kustomize installer to docs 2019-07-04 13:45:56 +03:00
stefanprodan
d24a23f3bd Kustomize installer: add installer readme 2019-07-04 13:29:02 +03:00
stefanprodan
22045982e2 Kustomize installer: add Linkerd overlay 2019-07-04 13:27:23 +03:00
stefanprodan
f496f1e18f Kustomize installer: add Istio overlay 2019-07-04 13:27:03 +03:00
stefanprodan
2e802432c4 Kustomize installer: add Kubernetes overlay 2019-07-04 13:26:52 +03:00
stefanprodan
a2f747e16f Kustomize installer: add Prometheus base manifests 2019-07-04 13:26:22 +03:00
stefanprodan
982338e162 Kustomize installer: add Flagger base manifests 2019-07-04 13:26:08 +03:00
Stefan Prodan
03fe4775dd Merge pull request #231 from weaveworks/gloo-0.14.2
Update Gloo and Prometheus
2019-07-02 09:24:14 +03:00
stefanprodan
def7d9bde0 Update Prometheus to v2.10.0 and set retention to 2h 2019-07-02 09:10:56 +03:00
stefanprodan
a58a7cbeeb Update Gloo to 0.14.2 2019-07-01 16:07:38 +03:00
Stefan Prodan
82ca66c23b Merge pull request #230 from weaveworks/linkerd
Add Linkerd support and e2e testing
2019-07-01 10:28:38 +03:00
stefanprodan
92c971c0d7 Add ingress and A/B testing example to Linkerd docs 2019-07-01 10:02:50 +03:00
stefanprodan
30c4faf72b Add Linkerd canary deployments docs 2019-07-01 09:23:38 +03:00
stefanprodan
85ee7d17cf Set min analysis interval to 10s 2019-07-01 09:23:05 +03:00
stefanprodan
a6d278ae91 Add Linkerd traffic split diagram 2019-06-30 13:52:59 +03:00
stefanprodan
ad8d02f701 Use Linkerd metrics when NGINX is the mesh ingress
Set the metrics provider to Linkerd Prometheus when using NGINX as Linkerd Ingress. This mitigates the lack of canary metrics in the NGINX controller exporter.
2019-06-30 13:03:27 +03:00
stefanprodan
00fa5542f7 Add linkerd as mesh provider 2019-06-30 12:46:23 +03:00
stefanprodan
9ed2719d19 Add canary rollback test to Linkerd e2e 2019-06-30 10:36:37 +03:00
stefanprodan
8a809baf35 Linkerd e2e testing: set canary max weight to 50% 2019-06-29 16:06:16 +03:00
stefanprodan
ff90c42fa7 Fix Linkerd CLI install 2019-06-29 15:53:36 +03:00
stefanprodan
d651e8fe48 Fix Linkerd metrics test 2019-06-29 15:23:40 +03:00
stefanprodan
bc613905e9 Add Linkerd edge-19.6.4 e2e testing 2019-06-29 15:20:35 +03:00
stefanprodan
e3321118e5 Fix linkerd success rate query 2019-06-29 15:03:36 +03:00
Stefan Prodan
31f526cbd6 Merge pull request #229 from weaveworks/istio-1.2.2
Update Istio e2e to v1.2.2
2019-06-29 10:53:31 +03:00
stefanprodan
493554178f Update Istio e2e to v1.2.2
Disable galley and MCP
2019-06-29 10:27:40 +03:00
Stefan Prodan
004b1cc7dd Merge pull request #228 from weaveworks/updates
Update Grafana and Kubernetes Kind
2019-06-27 10:31:20 +03:00
stefanprodan
767602592c Bump podinfo chart version 2019-06-27 10:19:34 +03:00
stefanprodan
34676acaf5 Add Istio TLS mode to podinfo chart 2019-06-27 10:14:06 +03:00
stefanprodan
491ab7affa Update Grafana to v6.2.5 2019-06-27 10:13:25 +03:00
stefanprodan
b522bbd903 Update Kubernetes Kind to v0.4.0 2019-06-27 09:58:56 +03:00
Stefan Prodan
dd3bc28806 Merge pull request #227 from dcherman/validate-k8s-version
Validate the minimum supported k8s version
2019-06-27 09:56:25 +03:00
Daniel Herman
764e7e275d Validate the minimum supported k8s version
Fixes #103
2019-06-27 00:18:08 -04:00
stefanprodan
931c051153 Fix tag composition in release script 2019-06-24 17:46:12 +03:00
Stefan Prodan
3da86fe118 Merge pull request #224 from weaveworks/hpa
Update primary HPA only when canary changed
2019-06-24 17:14:26 +03:00
stefanprodan
93f37a3022 Update primary HPA only when canary changed 2019-06-24 16:21:22 +03:00
stefanprodan
77b3d861e6 Add release workflow to CI 2019-06-24 16:08:56 +03:00
Stefan Prodan
ce0e16ffe8 Merge pull request #222 from weaveworks/release-v0.16.0
Release v0.16.0
2019-06-24 13:53:59 +03:00
stefanprodan
fb9709ae78 Add Blue/Green to FAQ 2019-06-23 15:48:13 +03:00
stefanprodan
191c3868ab Update changelog for v0.16.0 2019-06-23 13:52:01 +03:00
stefanprodan
d076f0859e Release v0.16.0 2019-06-23 13:48:39 +03:00
stefanprodan
df24ba86d0 Add Blue/Green tutorial to docs 2019-06-23 13:40:12 +03:00
stefanprodan
3996bcfa67 Add canary provider field to docs 2019-06-23 13:35:36 +03:00
Stefan Prodan
9e8a4ad384 Merge pull request #221 from weaveworks/gloo-v0.14.0
Update Gloo e2e testing to v0.14.0
2019-06-23 11:35:31 +03:00
stefanprodan
26ee668612 Use Kind 0.2.1 for Gloo e2e 2019-06-23 11:22:21 +03:00
stefanprodan
e3c102e7f8 Use test ns for Gloo virtual service in e2e 2019-06-22 17:49:13 +03:00
stefanprodan
ba60b127ea Use Kind 0.3.0 for Gloo e2e 2019-06-22 16:50:50 +03:00
stefanprodan
74c69dc07e Update Gloo e2e to v0.14.0 2019-06-22 16:48:38 +03:00
Stefan Prodan
0687d89178 Merge pull request #220 from weaveworks/update-e2e
Update e2e tests to Kubernetes 1.14
2019-06-22 16:12:20 +03:00
stefanprodan
7a454c005f Use Kind 0.2.1 for supergloo e2e 2019-06-22 15:55:48 +03:00
stefanprodan
2ce4f3a93e Revert supergloo upgrade (Istio 1.1 not supported in v0.3.23) 2019-06-22 15:41:17 +03:00
stefanprodan
7baaaebdd4 Use Kind 0.2.1 for Gloo e2e 2019-06-22 15:28:45 +03:00
stefanprodan
608c7f7a31 Use Istio 1.1.3 for supergloo e2e testing 2019-06-22 15:15:38 +03:00
stefanprodan
1a0daa8678 Use http probes with Kind 0.3.0 2019-06-22 14:58:01 +03:00
stefanprodan
ed0d25af97 Revert to Kind 0.2.1 2019-06-22 14:33:06 +03:00
stefanprodan
720d04aba1 Update Supergloo to v0.3.23 2019-06-22 14:16:30 +03:00
stefanprodan
901648393a Update Kubernetes Kind to v0.3.0 2019-06-22 14:02:33 +03:00
Stefan Prodan
b5acd817fc Merge pull request #219 from weaveworks/istio-1.2.0
Fix Istio 1.2.0 e2e testing
2019-06-22 13:58:48 +03:00
stefanprodan
2586fc6ef0 Update Kubernetes Kind to v0.3.0 2019-06-22 13:58:18 +03:00
stefanprodan
62e0eb6395 Update changelog 2019-06-22 13:44:47 +03:00
stefanprodan
768b0490e2 Show CircleCI build status 2019-06-22 13:43:39 +03:00
stefanprodan
852454fa2c Fix Istio v1.2.0 e2e testing by enabling galley 2019-06-22 13:42:57 +03:00
Stefan Prodan
970b67d6f6 Merge pull request #212 from marcoferrer/bump-e2e-istio-version
Upgrade e2e tests to Istio v1.2.0
2019-06-22 13:41:11 +03:00
Stefan Prodan
ea0eddff82 Merge pull request #218 from weaveworks/ci
Refactor CI and e2e testing
2019-06-22 13:11:29 +03:00
stefanprodan
0d4d2ac37b CircleCI: Build and push load tester 2019-06-22 09:51:37 +03:00
stefanprodan
d0591916a4 Update k8s packages 2019-06-21 23:42:59 +03:00
stefanprodan
6a8aef8675 CircleCI: workaround for code gen 2019-06-21 21:47:09 +03:00
stefanprodan
a894a7a0ce CircleCI: update code gen package 2019-06-21 20:36:18 +03:00
stefanprodan
0bbe724b8c CircleCI: chmod k8s code gen 2019-06-21 16:43:16 +03:00
stefanprodan
bea22c0259 CircleCI: run go mod download 2019-06-21 16:40:25 +03:00
stefanprodan
6363580120 Fix k8s code gen 2019-06-21 16:34:19 +03:00
stefanprodan
cbdc7ef2d3 Build and run k8s code gen with go modules 2019-06-21 16:31:51 +03:00
stefanprodan
0959406609 Remove vendor dir 2019-06-21 16:31:19 +03:00
stefanprodan
cf41f9a478 CircleCI - fix deprecated goreleaser config 2019-06-21 16:15:51 +03:00
stefanprodan
6fe6a41e3e CircleCI - cleanup branch filters 2019-06-21 15:58:23 +03:00
stefanprodan
91cd2648d9 CircleCI - run goreleaser for git tags 2019-06-21 15:39:56 +03:00
stefanprodan
240591a6b8 CircleCI - run goreleaser 2019-06-21 15:29:38 +03:00
stefanprodan
2973822113 CircleCI - test goreleaser job 2019-06-21 15:17:39 +03:00
stefanprodan
a6b2b1246c CircleCI - add goreleaser job 2019-06-21 15:07:00 +03:00
stefanprodan
c74456411d CircleCI - install Tiller after Kind create 2019-06-21 14:54:17 +03:00
stefanprodan
31b3fcf906 CircleCI - refactor e2e tests 2019-06-21 14:43:00 +03:00
stefanprodan
767be5b6a8 CircleCI - reset go mod cache 2019-06-21 14:28:29 +03:00
stefanprodan
48834cd8d1 CircleCI - refactor Istio e2e testing 2019-06-21 14:26:12 +03:00
stefanprodan
f4bb0ea9c2 CircleCI - add codecov 2019-06-21 13:58:36 +03:00
stefanprodan
cf25a9a8a5 CircleCI - fix config 2019-06-21 13:47:06 +03:00
stefanprodan
4f0ad7a067 CircleCI - run Istio e2e 2019-06-21 13:45:16 +03:00
stefanprodan
c0fe461a9f CircleCI - push to Docker Hub 2019-06-21 13:33:26 +03:00
stefanprodan
1911143514 CircleCI - run go test 2019-06-21 13:29:32 +03:00
stefanprodan
9b67b360d0 CircleCI - build and push container 2019-06-21 13:26:12 +03:00
stefanprodan
991e01efd2 CircleCI - fix container build 2019-06-21 13:10:27 +03:00
stefanprodan
83b8ae46c9 CircleCI - copy bin to workspace dir 2019-06-21 13:07:59 +03:00
stefanprodan
c3b7aee063 CircleCI - make workspace dir 2019-06-21 13:05:16 +03:00
stefanprodan
66d662c085 CircleCI - fix working_directory 2019-06-21 13:01:47 +03:00
stefanprodan
4d5876fb76 CircleCI - fix job name 2019-06-21 13:00:26 +03:00
stefanprodan
7ca2558a81 CircleCI - fix config 2019-06-21 12:59:26 +03:00
stefanprodan
8957994c1a CircleCI - set job deps 2019-06-21 12:58:17 +03:00
stefanprodan
0147aea69b Build binary and container in CircleCI
Cache go modules
2019-06-21 12:55:27 +03:00
stefanprodan
b5f73d66ec Add version command 2019-06-21 12:54:43 +03:00
Stefan Prodan
6800181594 Merge pull request #217 from weaveworks/provider
Add the service mesh provider to the canary spec
2019-06-21 11:13:17 +03:00
Stefan Prodan
6f5f80a085 Merge pull request #216 from weaveworks/hpa-promotion
Reconcile the primary HPA on canary promotion
2019-06-21 11:13:02 +03:00
stefanprodan
fd23a2f98f Add kubernetes provider type
Synonym to provider `none`, to be used for blue/green deployments
2019-06-20 15:18:48 +03:00
stefanprodan
63cb8a5ba5 Lookup the canary provider field during reconciliation
Override the global provider if one is specified in the canary spec
2019-06-20 14:52:43 +03:00
stefanprodan
4a9e3182c6 Add the mesh provider field to canary CRD 2019-06-20 14:50:21 +03:00
stefanprodan
5cbc3df7b5 Use internal load testers address in canary example 2019-06-20 13:32:06 +03:00
stefanprodan
dcadc2303f Add HPA promotion tests 2019-06-20 13:31:34 +03:00
stefanprodan
cf5f364ed2 Update the primary HPA on canary promotion 2019-06-20 13:30:55 +03:00
Stefan Prodan
e45ace5d9b Merge pull request #211 from weaveworks/noprouter
Allow blue/green deployments without a service mesh provider
2019-06-20 13:02:02 +03:00
Marco Ferrer
6e7421b0d8 Upgrade e2e tests to Istio v1.2.0 2019-06-19 13:26:22 -04:00
stefanprodan
647d02890f Add HTTP metrics when no mesh provider is specified
Implement request-success-rate and request-duration checks using http_request_duration_seconds histogram
2019-06-19 13:15:17 +03:00
stefanprodan
7e72d23b60 Bump load tester version to 0.4.0 2019-06-19 13:12:04 +03:00
stefanprodan
9fada306f0 Add a service mesh provider of type none
To be used for Kubernetes blue/green deployments with the no-operations router
2019-06-19 12:02:40 +03:00
stefanprodan
8d1cc83405 Add a no-operation router
To be used for Kubernetes blue/green deployments (no service mesh or ingress controller)
2019-06-19 12:01:02 +03:00
Stefan Prodan
1979bc59d0 Merge pull request #210 from weaveworks/nop-router
Kubernetes service reconciliation improvements
2019-06-19 11:49:10 +03:00
stefanprodan
bf7ebc9708 Skip readiness check on init for Istio SMI 2019-06-19 11:16:11 +03:00
stefanprodan
dc3cde88d2 Use Helm to install Flagger for Istio e2e tests 2019-06-19 11:03:44 +03:00
stefanprodan
98beb1011e Skip primary check on init when using Istio
The deployment will become ready after the ClusterIP are created
2019-06-19 10:50:55 +03:00
stefanprodan
8c59e9d2b4 Fix metrics URL getter 2019-06-19 10:30:19 +03:00
stefanprodan
9a87d47f45 Check primary readiness on initialisation
Wait for the primary to become ready before scaling down the canary in the init phase
2019-06-19 09:49:25 +03:00
stefanprodan
f25023ed1b Include selector in service reconciliation
- detect changes in the Kubernetes service selectors and ports
- preserve the immutable fields when updating the ClusterIP services
2019-06-18 17:57:00 +03:00
stefanprodan
806b233d58 Fix typo in ClusterIP FAQ 2019-06-18 13:35:14 +03:00
Stefan Prodan
677ee8d639 Merge pull request #207 from weaveworks/port-discovery
Implement port discovery
2019-06-18 13:09:35 +03:00
stefanprodan
61ac8d7a8c Add port discovery to canary example 2019-06-18 12:46:21 +03:00
stefanprodan
278680b248 Add port discovery to changelog 2019-06-18 12:43:50 +03:00
stefanprodan
5e4a58a1c1 Upgrade e2e tests to Istio v1.1.9 2019-06-18 11:35:21 +03:00
stefanprodan
757b5ca22e Add missing config params to chart readme 2019-06-18 11:35:07 +03:00
stefanprodan
6d1da5bb45 Use container name in port discovery
If the port name is missing, append the container name to the tcp port name
2019-06-17 20:50:21 +03:00
stefanprodan
9ca79d147d Add Istio virtual service merging to FAQ 2019-06-17 12:06:53 +03:00
stefanprodan
37fcfe15bb Merge feature comparison table 2019-06-16 11:48:21 +03:00
stefanprodan
a9c7466359 Add pod affinity and label selectors to FAQ 2019-06-16 11:18:51 +03:00
stefanprodan
91a3f2c9a7 Add NGINX A/B testing convention to FAQ 2019-06-16 10:52:33 +03:00
stefanprodan
9aa341d088 Add load tester mTLS to FAQ
Ref: #186
2019-06-16 10:38:07 +03:00
stefanprodan
c9e09fa8eb Add Istio mTLS to FAQ
Fix: #205
2019-06-16 10:36:25 +03:00
stefanprodan
e6257b7531 Add port discovery to FAQ 2019-06-16 10:33:03 +03:00
stefanprodan
aee027c91c Add Kubernetes services to FAQ 2019-06-16 10:32:25 +03:00
stefanprodan
c106796751 Add A/B testing to FAQ 2019-06-16 10:31:25 +03:00
stefanprodan
42bd600482 Update GKE Prometheus config 2019-06-15 16:55:27 +03:00
stefanprodan
47ad81be5b Remove unused go modules 2019-06-15 16:54:52 +03:00
stefanprodan
88c450e3bd Implement port discovery
If port discovery is enabled, Flagger scans the deployment pod template and extracts the container ports excluding the port specified in the canary service spec and Istio proxy ports. All the extra ports will be used when generation the ClusterIP services.
2019-06-15 16:34:32 +03:00
stefanprodan
2ebedd185c Add port discovery field to canary service spec 2019-06-15 16:18:54 +03:00
3553 changed files with 10326 additions and 1005285 deletions

View File

@@ -1,86 +1,247 @@
version: 2.1
jobs:
build-binary:
docker:
- image: circleci/golang:1.13
working_directory: ~/build
steps:
- checkout
- restore_cache:
keys:
- go-mod-v3-{{ checksum "go.sum" }}
- run:
name: Run go mod download
command: go mod download
- run:
name: Run go fmt
command: make test-fmt
- run:
name: Build Flagger
command: |
CGO_ENABLED=0 GOOS=linux go build \
-ldflags "-s -w -X github.com/weaveworks/flagger/pkg/version.REVISION=${CIRCLE_SHA1}" \
-a -installsuffix cgo -o bin/flagger ./cmd/flagger/*.go
- run:
name: Build Flagger load tester
command: |
CGO_ENABLED=0 GOOS=linux go build \
-a -installsuffix cgo -o bin/loadtester ./cmd/loadtester/*.go
- run:
name: Run unit tests
command: |
go test -race -coverprofile=coverage.txt -covermode=atomic $(go list ./pkg/...)
bash <(curl -s https://codecov.io/bash)
- run:
name: Verify code gen
command: make test-codegen
- save_cache:
key: go-mod-v3-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod/"
- persist_to_workspace:
root: bin
paths:
- flagger
- loadtester
push-container:
docker:
- image: circleci/golang:1.13
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/container-push.sh
push-binary:
docker:
- image: circleci/golang:1.13
working_directory: ~/build
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
- restore_cache:
keys:
- go-mod-v3-{{ checksum "go.sum" }}
- run: test/goreleaser.sh
e2e-istio-testing:
machine: true
steps:
- checkout
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/e2e-kind.sh
- run: test/e2e-istio.sh
- run: test/e2e-build.sh
- run: test/e2e-tests.sh
e2e-kubernetes-testing:
machine: true
steps:
- checkout
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/e2e-kind.sh
- run: test/e2e-kubernetes.sh
- run: test/e2e-kubernetes-tests.sh
e2e-smi-istio-testing:
machine: true
steps:
- checkout
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/e2e-kind.sh
- run: test/e2e-istio.sh
- run: test/e2e-smi-istio-build.sh
- run: test/e2e-tests.sh canary
e2e-supergloo-testing:
machine: true
steps:
- checkout
- run: test/e2e-kind.sh
- run: test/e2e-supergloo.sh
- run: test/e2e-build.sh supergloo:test.supergloo-system
- run: test/e2e-smi-istio.sh
- run: test/e2e-tests.sh canary
e2e-gloo-testing:
machine: true
steps:
- checkout
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/e2e-kind.sh
- run: test/e2e-gloo.sh
- run: test/e2e-gloo-build.sh
- run: test/e2e-gloo-tests.sh
e2e-nginx-testing:
machine: true
steps:
- checkout
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/e2e-kind.sh
- run: test/e2e-nginx.sh
- run: test/e2e-nginx-build.sh
- run: test/e2e-nginx-tests.sh
- run: test/e2e-nginx-cleanup.sh
- run: test/e2e-nginx-custom-annotations.sh
- run: test/e2e-nginx-tests.sh
e2e-linkerd-testing:
machine: true
steps:
- checkout
- attach_workspace:
at: /tmp/bin
- run: test/container-build.sh
- run: test/e2e-kind.sh
- run: test/e2e-linkerd.sh
- run: test/e2e-linkerd-tests.sh
push-helm-charts:
docker:
- image: circleci/golang:1.13
steps:
- checkout
- run:
name: Install kubectl
command: sudo curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && sudo chmod +x /usr/local/bin/kubectl
- run:
name: Install helm
command: sudo curl -L https://storage.googleapis.com/kubernetes-helm/helm-v2.14.2-linux-amd64.tar.gz | tar xz && sudo mv linux-amd64/helm /bin/helm && sudo rm -rf linux-amd64
- run:
name: Initialize helm
command: helm init --client-only --kubeconfig=$HOME/.kube/kubeconfig
- run:
name: Lint charts
command: |
helm lint ./charts/*
- run:
name: Package charts
command: |
mkdir $HOME/charts
helm package ./charts/* --destination $HOME/charts
- run:
name: Publish charts
command: |
if echo "${CIRCLE_TAG}" | grep -Eq "[0-9]+(\.[0-9]+)*(-[a-z]+)?$"; then
REPOSITORY="https://weaveworksbot:${GITHUB_TOKEN}@github.com/weaveworks/flagger.git"
git config user.email weaveworksbot@users.noreply.github.com
git config user.name weaveworksbot
git remote set-url origin ${REPOSITORY}
git checkout gh-pages
mv -f $HOME/charts/*.tgz .
helm repo index . --url https://flagger.app
git add .
git commit -m "Publish Helm charts v${CIRCLE_TAG}"
git push origin gh-pages
else
echo "Not a release! Skip charts publish"
fi
workflows:
version: 2
build-and-test:
build-test-push:
jobs:
- build-binary:
filters:
branches:
ignore:
- gh-pages
- e2e-istio-testing:
filters:
branches:
ignore:
- /gh-pages.*/
- /docs-.*/
- /release-.*/
- e2e-smi-istio-testing:
filters:
branches:
ignore:
- /gh-pages.*/
- /docs-.*/
- /release-.*/
- e2e-supergloo-testing:
filters:
branches:
ignore:
- /gh-pages.*/
- /docs-.*/
- /release-.*/
- e2e-nginx-testing:
filters:
branches:
ignore:
- /gh-pages.*/
- /docs-.*/
- /release-.*/
requires:
- build-binary
- e2e-kubernetes-testing:
requires:
- build-binary
- e2e-gloo-testing:
requires:
- build-binary
- e2e-nginx-testing:
requires:
- build-binary
- e2e-linkerd-testing:
requires:
- build-binary
- push-container:
requires:
- build-binary
- e2e-istio-testing
- e2e-kubernetes-testing
- e2e-gloo-testing
- e2e-nginx-testing
- e2e-linkerd-testing
release:
jobs:
- build-binary:
filters:
branches:
ignore:
- /gh-pages.*/
- /docs-.*/
- /release-.*/
ignore: /.*/
tags:
ignore: /^chart.*/
- push-container:
requires:
- build-binary
filters:
branches:
ignore: /.*/
tags:
ignore: /^chart.*/
- push-binary:
requires:
- push-container
filters:
branches:
ignore: /.*/
tags:
ignore: /^chart.*/
- push-helm-charts:
requires:
- push-container
filters:
branches:
ignore: /.*/
tags:
ignore: /^chart.*/

2
.gitignore vendored
View File

@@ -13,5 +13,7 @@
.DS_Store
bin/
_tmp/
artifacts/gcloud/
.idea

View File

@@ -8,7 +8,11 @@ builds:
- amd64
env:
- CGO_ENABLED=0
archive:
name_template: "{{ .Binary }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
files:
- none*
archives:
- name_template: "{{ .Binary }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
files:
- none*
changelog:
filters:
exclude:
- '^CircleCI'

View File

@@ -1,50 +0,0 @@
sudo: required
language: go
branches:
except:
- /gh-pages.*/
- /docs-.*/
go:
- 1.12.x
services:
- docker
addons:
apt:
packages:
- docker-ce
script:
- make test-fmt
- make test-codegen
- go test -race -coverprofile=coverage.txt -covermode=atomic $(go list ./pkg/...)
- make build
after_success:
- if [ -z "$DOCKER_USER" ]; then
echo "PR build, skipping image push";
else
BRANCH_COMMIT=${TRAVIS_BRANCH}-$(echo ${TRAVIS_COMMIT} | head -c7);
docker tag weaveworks/flagger:latest weaveworks/flagger:${BRANCH_COMMIT};
echo $DOCKER_PASS | docker login -u=$DOCKER_USER --password-stdin;
docker push weaveworks/flagger:${BRANCH_COMMIT};
fi
- if [ -z "$TRAVIS_TAG" ]; then
echo "Not a release, skipping image push";
else
docker tag weaveworks/flagger:latest weaveworks/flagger:${TRAVIS_TAG};
echo $DOCKER_PASS | docker login -u=$DOCKER_USER --password-stdin;
docker push weaveworks/flagger:$TRAVIS_TAG;
fi
- bash <(curl -s https://codecov.io/bash)
- rm coverage.txt
deploy:
- provider: script
skip_cleanup: true
script: curl -sL http://git.io/goreleaser | bash
on:
tags: true

View File

@@ -2,6 +2,210 @@
All notable changes to this project are documented in this file.
## 0.20.2 (2019-11-07)
Adds support for exposing canaries outside the cluster using App Mesh Gateway annotations
#### Improvements
- Expose canaries on public domains with App Mesh Gateway [#358](https://github.com/weaveworks/flagger/pull/358)
#### Fixes
- Use the specified replicas when scaling up the canary [#363](https://github.com/weaveworks/flagger/pull/363)
## 0.20.1 (2019-11-03)
Fixes promql execution and updates the load testing tools
#### Improvements
- Update load tester Helm tools [#8349dd1](https://github.com/weaveworks/flagger/commit/8349dd1cda59a741c7bed9a0f67c0fc0fbff4635)
- e2e testing: update providers [#346](https://github.com/weaveworks/flagger/pull/346)
#### Fixes
- Fix Prometheus query escape [#353](https://github.com/weaveworks/flagger/pull/353)
- Updating hey release link [#350](https://github.com/weaveworks/flagger/pull/350)
## 0.20.0 (2019-10-21)
Adds support for [A/B Testing](https://docs.flagger.app/usage/progressive-delivery#traffic-mirroring) and retry policies when using App Mesh
#### Features
- Implement App Mesh A/B testing based on HTTP headers match conditions [#340](https://github.com/weaveworks/flagger/pull/340)
- Implement App Mesh HTTP retry policy [#338](https://github.com/weaveworks/flagger/pull/338)
- Implement metrics server override [#342](https://github.com/weaveworks/flagger/pull/342)
#### Improvements
- Add the app/name label to services and primary deployment [#333](https://github.com/weaveworks/flagger/pull/333)
- Allow setting Slack and Teams URLs with env vars [#334](https://github.com/weaveworks/flagger/pull/334)
- Refactor Gloo integration [#344](https://github.com/weaveworks/flagger/pull/344)
#### Fixes
- Generate unique names for App Mesh virtual routers and routes [#336](https://github.com/weaveworks/flagger/pull/336)
## 0.19.0 (2019-10-08)
Adds support for canary and blue/green [traffic mirroring](https://docs.flagger.app/usage/progressive-delivery#traffic-mirroring)
#### Features
- Add traffic mirroring for Istio service mesh [#311](https://github.com/weaveworks/flagger/pull/311)
- Implement canary service target port [#327](https://github.com/weaveworks/flagger/pull/327)
#### Improvements
- Allow gPRC protocol for App Mesh [#325](https://github.com/weaveworks/flagger/pull/325)
- Enforce blue/green when using Kubernetes networking [#326](https://github.com/weaveworks/flagger/pull/326)
#### Fixes
- Fix port discovery diff [#324](https://github.com/weaveworks/flagger/pull/324)
- Helm chart: Enable Prometheus scraping of Flagger metrics [#2141d88](https://github.com/weaveworks/flagger/commit/2141d88ce1cc6be220dab34171c215a334ecde24)
## 0.18.6 (2019-10-03)
Adds support for App Mesh conformance tests and latency metric checks
#### Improvements
- Add support for acceptance testing when using App Mesh [#322](https://github.com/weaveworks/flagger/pull/322)
- Add Kustomize installer for App Mesh [#310](https://github.com/weaveworks/flagger/pull/310)
- Update Linkerd to v2.5.0 and Prometheus to v2.12.0 [#323](https://github.com/weaveworks/flagger/pull/323)
#### Fixes
- Fix slack/teams notification fields mapping [#318](https://github.com/weaveworks/flagger/pull/318)
## 0.18.5 (2019-10-02)
Adds support for [confirm-promotion](https://docs.flagger.app/how-it-works#webhooks) webhooks and blue/green deployments when using a service mesh
#### Features
- Implement confirm-promotion hook [#307](https://github.com/weaveworks/flagger/pull/307)
- Implement B/G for service mesh providers [#305](https://github.com/weaveworks/flagger/pull/305)
#### Improvements
- Canary promotion improvements to avoid dropping in-flight requests [#310](https://github.com/weaveworks/flagger/pull/310)
- Update end-to-end tests to Kubernetes v1.15.3 and Istio 1.3.0 [#306](https://github.com/weaveworks/flagger/pull/306)
#### Fixes
- Skip primary check for App Mesh [#315](https://github.com/weaveworks/flagger/pull/315)
## 0.18.4 (2019-09-08)
Adds support for NGINX custom annotations and Helm v3 acceptance testing
#### Features
- Add annotations prefix for NGINX ingresses [#293](https://github.com/weaveworks/flagger/pull/293)
- Add wide columns in CRD [#289](https://github.com/weaveworks/flagger/pull/289)
- loadtester: implement Helm v3 test command [#296](https://github.com/weaveworks/flagger/pull/296)
- loadtester: add gPRC health check to load tester image [#295](https://github.com/weaveworks/flagger/pull/295)
#### Fixes
- loadtester: fix tests error logging [#286](https://github.com/weaveworks/flagger/pull/286)
## 0.18.3 (2019-08-22)
Adds support for tillerless helm tests and protobuf health checking
#### Features
- loadtester: add support for tillerless helm [#280](https://github.com/weaveworks/flagger/pull/280)
- loadtester: add support for protobuf health checking [#280](https://github.com/weaveworks/flagger/pull/280)
#### Improvements
- Set HTTP listeners for AppMesh virtual routers [#272](https://github.com/weaveworks/flagger/pull/272)
#### Fixes
- Add missing fields to CRD validation spec [#271](https://github.com/weaveworks/flagger/pull/271)
- Fix App Mesh backends validation in CRD [#281](https://github.com/weaveworks/flagger/pull/281)
## 0.18.2 (2019-08-05)
Fixes multi-port support for Istio
#### Fixes
- Fix port discovery for multiple port services [#267](https://github.com/weaveworks/flagger/pull/267)
#### Improvements
- Update e2e testing to Istio v1.2.3, Gloo v0.18.8 and NGINX ingress chart v1.12.1 [#268](https://github.com/weaveworks/flagger/pull/268)
## 0.18.1 (2019-07-30)
Fixes Blue/Green style deployments for Kubernetes and Linkerd providers
#### Fixes
- Fix Blue/Green metrics provider and add e2e tests [#261](https://github.com/weaveworks/flagger/pull/261)
## 0.18.0 (2019-07-29)
Adds support for [manual gating](https://docs.flagger.app/how-it-works#manual-gating) and pausing/resuming an ongoing analysis
#### Features
- Implement confirm rollout gate, hook and API [#251](https://github.com/weaveworks/flagger/pull/251)
#### Improvements
- Refactor canary change detection and status [#240](https://github.com/weaveworks/flagger/pull/240)
- Implement finalising state [#257](https://github.com/weaveworks/flagger/pull/257)
- Add gRPC load testing tool [#248](https://github.com/weaveworks/flagger/pull/248)
#### Breaking changes
- Due to the status sub-resource changes in [#240](https://github.com/weaveworks/flagger/pull/240), when upgrading Flagger the canaries status phase will be reset to `Initialized`
- Upgrading Flagger with Helm will fail due to Helm poor support of CRDs, see [workaround](https://github.com/weaveworks/flagger/issues/223)
## 0.17.0 (2019-07-08)
Adds support for Linkerd (SMI Traffic Split API), MS Teams notifications and HA mode with leader election
#### Features
- Add Linkerd support [#230](https://github.com/weaveworks/flagger/pull/230)
- Implement MS Teams notifications [#235](https://github.com/weaveworks/flagger/pull/235)
- Implement leader election [#236](https://github.com/weaveworks/flagger/pull/236)
#### Improvements
- Add [Kustomize](https://docs.flagger.app/install/flagger-install-on-kubernetes#install-flagger-with-kustomize) installer [#232](https://github.com/weaveworks/flagger/pull/232)
- Add Pod Security Policy to Helm chart [#234](https://github.com/weaveworks/flagger/pull/234)
## 0.16.0 (2019-06-23)
Adds support for running [Blue/Green deployments](https://docs.flagger.app/usage/blue-green) without a service mesh or ingress controller
#### Features
- Allow blue/green deployments without a service mesh provider [#211](https://github.com/weaveworks/flagger/pull/211)
- Add the service mesh provider to the canary spec [#217](https://github.com/weaveworks/flagger/pull/217)
- Allow multi-port services and implement port discovery [#207](https://github.com/weaveworks/flagger/pull/207)
#### Improvements
- Add [FAQ page](https://docs.flagger.app/faq) to docs website
- Switch to go modules in CI [#218](https://github.com/weaveworks/flagger/pull/218)
- Update e2e testing to Kubernetes Kind 0.3.0 and Istio 1.2.0
#### Fixes
- Update the primary HPA on canary promotion [#216](https://github.com/weaveworks/flagger/pull/216)
## 0.15.0 (2019-06-12)
Adds support for customising the Istio [traffic policy](https://docs.flagger.app/how-it-works#istio-routing) in the canary service spec

View File

@@ -1,19 +1,4 @@
FROM golang:1.12
RUN mkdir -p /go/src/github.com/weaveworks/flagger/
WORKDIR /go/src/github.com/weaveworks/flagger
COPY . .
#RUN GO111MODULE=on go mod download
RUN GIT_COMMIT=$(git rev-list -1 HEAD) && \
GO111MODULE=on CGO_ENABLED=0 GOOS=linux go build -mod=vendor -ldflags "-s -w \
-X github.com/weaveworks/flagger/pkg/version.REVISION=${GIT_COMMIT}" \
-a -installsuffix cgo -o flagger ./cmd/flagger/*
FROM alpine:3.9
FROM alpine:3.10
RUN addgroup -S flagger \
&& adduser -S -g flagger flagger \
@@ -21,7 +6,7 @@ RUN addgroup -S flagger \
WORKDIR /home/flagger
COPY --from=0 /go/src/github.com/weaveworks/flagger/flagger .
COPY /bin/flagger .
RUN chown -R flagger:flagger ./

View File

@@ -1,15 +1,3 @@
FROM golang:1.12 AS builder
RUN mkdir -p /go/src/github.com/weaveworks/flagger/
WORKDIR /go/src/github.com/weaveworks/flagger
COPY . .
RUN go test -race ./pkg/loadtester/
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o loadtester ./cmd/loadtester/*
FROM bats/bats:v1.1.0
RUN addgroup -S app \
@@ -18,17 +6,39 @@ RUN addgroup -S app \
WORKDIR /home/app
RUN curl -sSLo hey "https://storage.googleapis.com/jblabs/dist/hey_linux_v0.1.2" && \
RUN curl -sSLo hey "https://storage.googleapis.com/hey-release/hey_linux_amd64" && \
chmod +x hey && mv hey /usr/local/bin/hey
RUN curl -sSL "https://get.helm.sh/helm-v2.12.3-linux-amd64.tar.gz" | tar xvz && \
# verify hey works
RUN hey -n 1 -c 1 https://flagger.app > /dev/null && echo $? | grep 0
RUN curl -sSL "https://get.helm.sh/helm-v2.15.1-linux-amd64.tar.gz" | tar xvz && \
chmod +x linux-amd64/helm && mv linux-amd64/helm /usr/local/bin/helm && \
chmod +x linux-amd64/tiller && mv linux-amd64/tiller /usr/local/bin/tiller && \
rm -rf linux-amd64
COPY --from=builder /go/src/github.com/weaveworks/flagger/loadtester .
RUN curl -sSL "https://get.helm.sh/helm-v3.0.0-rc.2-linux-amd64.tar.gz" | tar xvz && \
chmod +x linux-amd64/helm && mv linux-amd64/helm /usr/local/bin/helmv3 && \
rm -rf linux-amd64
RUN GRPC_HEALTH_PROBE_VERSION=v0.3.1 && \
wget -qO /usr/local/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \
chmod +x /usr/local/bin/grpc_health_probe
RUN curl -sSL "https://github.com/bojand/ghz/releases/download/v0.39.0/ghz_0.39.0_Linux_x86_64.tar.gz" | tar xz -C /tmp && \
mv /tmp/ghz /usr/local/bin && chmod +x /usr/local/bin/ghz && rm -rf /tmp/ghz-web
ADD https://raw.githubusercontent.com/grpc/grpc-proto/master/grpc/health/v1/health.proto /tmp/ghz/health.proto
RUN ls /tmp
COPY ./bin/loadtester .
RUN chown -R app:app ./
USER app
ENTRYPOINT ["./loadtester"]
RUN curl -sSL "https://github.com/rimusz/helm-tiller/archive/v0.9.3.tar.gz" | tar xvz && \
helm init --client-only && helm plugin install helm-tiller-0.9.3 && helm plugin list
ENTRYPOINT ["./loadtester"]

View File

@@ -2,41 +2,40 @@ TAG?=latest
VERSION?=$(shell grep 'VERSION' pkg/version/version.go | awk '{ print $$4 }' | tr -d '"')
VERSION_MINOR:=$(shell grep 'VERSION' pkg/version/version.go | awk '{ print $$4 }' | tr -d '"' | rev | cut -d'.' -f2- | rev)
PATCH:=$(shell grep 'VERSION' pkg/version/version.go | awk '{ print $$4 }' | tr -d '"' | awk -F. '{print $$NF}')
SOURCE_DIRS = cmd pkg/apis pkg/controller pkg/server pkg/logging pkg/version
SOURCE_DIRS = cmd pkg/apis pkg/controller pkg/server pkg/canary pkg/metrics pkg/router pkg/notifier
LT_VERSION?=$(shell grep 'VERSION' cmd/loadtester/main.go | awk '{ print $$4 }' | tr -d '"' | head -n1)
TS=$(shell date +%Y-%m-%d_%H-%M-%S)
run:
go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=istio -namespace=test \
-metrics-server=https://prometheus.istio.weavedx.com \
-slack-url=https://hooks.slack.com/services/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 \
-slack-channel="devops-alerts"
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=istio -namespace=test-istio \
-metrics-server=https://prometheus.istio.flagger.dev
run-appmesh:
go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=appmesh \
-metrics-server=http://acfc235624ca911e9a94c02c4171f346-1585187926.us-west-2.elb.amazonaws.com:9090 \
-slack-url=https://hooks.slack.com/services/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 \
-slack-channel="devops-alerts"
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=appmesh \
-metrics-server=http://acfc235624ca911e9a94c02c4171f346-1585187926.us-west-2.elb.amazonaws.com:9090
run-nginx:
go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=nginx -namespace=nginx \
-metrics-server=http://prometheus-weave.istio.weavedx.com \
-slack-url=https://hooks.slack.com/services/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 \
-slack-channel="devops-alerts"
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=nginx -namespace=nginx \
-metrics-server=http://prometheus-weave.istio.weavedx.com
run-smi:
go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=smi:istio -namespace=smi \
-metrics-server=https://prometheus.istio.weavedx.com \
-slack-url=https://hooks.slack.com/services/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 \
-slack-channel="devops-alerts"
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=smi:istio -namespace=smi \
-metrics-server=https://prometheus.istio.weavedx.com
run-gloo:
go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=gloo -namespace=gloo \
-metrics-server=https://prometheus.istio.weavedx.com \
-slack-url=https://hooks.slack.com/services/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 \
-slack-channel="devops-alerts"
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=gloo -namespace=gloo \
-metrics-server=https://prometheus.istio.weavedx.com
run-nop:
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=none -namespace=bg \
-metrics-server=https://prometheus.istio.weavedx.com
run-linkerd:
GO111MODULE=on go run cmd/flagger/* -kubeconfig=$$HOME/.kube/config -log-level=info -mesh-provider=linkerd -namespace=dev \
-metrics-server=https://prometheus.linkerd.flagger.dev
build:
GIT_COMMIT=$$(git rev-list -1 HEAD) && GO111MODULE=on CGO_ENABLED=0 GOOS=linux go build -ldflags "-s -w -X github.com/weaveworks/flagger/pkg/version.REVISION=$${GIT_COMMIT}" -a -installsuffix cgo -o ./bin/flagger ./cmd/flagger/*
docker build -t weaveworks/flagger:$(TAG) . -f Dockerfile
push:
@@ -57,8 +56,9 @@ test: test-fmt test-codegen
helm-package:
cd charts/ && helm package ./*
mv charts/*.tgz docs/
helm repo index docs --url https://weaveworks.github.io/flagger --merge ./docs/index.yaml
mv charts/*.tgz bin/
curl -s https://raw.githubusercontent.com/weaveworks/flagger/gh-pages/index.yaml > ./bin/index.yaml
helm repo index bin --url https://flagger.app --merge ./bin/index.yaml
helm-up:
helm upgrade --install flagger ./charts/flagger --namespace=istio-system --set crd.create=false
@@ -72,7 +72,8 @@ version-set:
sed -i '' "s/tag: $$current/tag: $$next/g" charts/flagger/values.yaml && \
sed -i '' "s/appVersion: $$current/appVersion: $$next/g" charts/flagger/Chart.yaml && \
sed -i '' "s/version: $$current/version: $$next/g" charts/flagger/Chart.yaml && \
echo "Version $$next set in code, deployment and charts"
sed -i '' "s/newTag: $$current/newTag: $$next/g" kustomize/base/flagger/kustomization.yaml && \
echo "Version $$next set in code, deployment, chart and kustomize"
version-up:
@next="$(VERSION_MINOR).$$(($(PATCH) + 1))" && \
@@ -106,11 +107,14 @@ reset-test:
kubectl apply -f ./artifacts/namespaces
kubectl apply -f ./artifacts/canaries
loadtester-run:
loadtester-run: loadtester-build
docker build -t weaveworks/flagger-loadtester:$(LT_VERSION) . -f Dockerfile.loadtester
docker rm -f tester || true
docker run -dp 8888:9090 --name tester weaveworks/flagger-loadtester:$(LT_VERSION)
loadtester-build:
GO111MODULE=on CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o ./bin/loadtester ./cmd/loadtester/*
loadtester-push:
docker build -t weaveworks/flagger-loadtester:$(LT_VERSION) . -f Dockerfile.loadtester
docker push weaveworks/flagger-loadtester:$(LT_VERSION)

View File

@@ -1,19 +1,19 @@
# flagger
[![build](https://travis-ci.org/weaveworks/flagger.svg?branch=master)](https://travis-ci.org/weaveworks/flagger)
[![build](https://img.shields.io/circleci/build/github/weaveworks/flagger/master.svg)](https://circleci.com/gh/weaveworks/flagger)
[![report](https://goreportcard.com/badge/github.com/weaveworks/flagger)](https://goreportcard.com/report/github.com/weaveworks/flagger)
[![codecov](https://codecov.io/gh/weaveworks/flagger/branch/master/graph/badge.svg)](https://codecov.io/gh/weaveworks/flagger)
[![license](https://img.shields.io/github/license/weaveworks/flagger.svg)](https://github.com/weaveworks/flagger/blob/master/LICENSE)
[![release](https://img.shields.io/github/release/weaveworks/flagger/all.svg)](https://github.com/weaveworks/flagger/releases)
Flagger is a Kubernetes operator that automates the promotion of canary deployments
using Istio, App Mesh, NGINX or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
using Istio, Linkerd, App Mesh, NGINX or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
The canary analysis can be extended with webhooks for running acceptance tests,
load tests or any other custom validation.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance
indicators like HTTP requests success rate, requests average duration and pods health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack or MS Teams.
![flagger-overview](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-overview.png)
@@ -35,12 +35,15 @@ Flagger documentation can be found at [docs.flagger.app](https://docs.flagger.ap
* [Custom metrics](https://docs.flagger.app/how-it-works#custom-metrics)
* [Webhooks](https://docs.flagger.app/how-it-works#webhooks)
* [Load testing](https://docs.flagger.app/how-it-works#load-testing)
* [Manual gating](https://docs.flagger.app/how-it-works#manual-gating)
* [FAQ](https://docs.flagger.app/faq)
* Usage
* [Istio canary deployments](https://docs.flagger.app/usage/progressive-delivery)
* [Istio A/B testing](https://docs.flagger.app/usage/ab-testing)
* [Linkerd canary deployments](https://docs.flagger.app/usage/linkerd-progressive-delivery)
* [App Mesh canary deployments](https://docs.flagger.app/usage/appmesh-progressive-delivery)
* [NGINX ingress controller canary deployments](https://docs.flagger.app/usage/nginx-progressive-delivery)
* [Gloo Canary Deployments](https://docs.flagger.app/usage/gloo-progressive-delivery)
* [Gloo ingress controller canary deployments](https://docs.flagger.app/usage/gloo-progressive-delivery)
* [Blue/Green deployments](https://docs.flagger.app/usage/blue-green)
* [Monitoring](https://docs.flagger.app/usage/monitoring)
* [Alerting](https://docs.flagger.app/usage/alerting)
* Tutorials
@@ -64,6 +67,9 @@ metadata:
name: podinfo
namespace: test
spec:
# service mesh provider (optional)
# can be: kubernetes, istio, linkerd, appmesh, nginx, gloo, supergloo
provider: istio
# deployment reference
targetRef:
apiVersion: apps/v1
@@ -78,14 +84,12 @@ spec:
kind: HorizontalPodAutoscaler
name: podinfo
service:
# container port
# ClusterIP port number
port: 9898
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
# Istio virtual service host names (optional)
hosts:
- podinfo.example.com
# container port name or number (optional)
targetPort: 9898
# port name can be http or grpc (default http)
portName: http
# HTTP match conditions (optional)
match:
- uri:
@@ -93,10 +97,6 @@ spec:
# HTTP rewrite (optional)
rewrite:
uri: /
# cross-origin resource sharing policy (optional)
corsPolicy:
allowOrigin:
- example.com
# request timeout (optional)
timeout: 5s
# promote the canary without analysing it (default false)
@@ -136,7 +136,7 @@ spec:
topic="podinfo"
}[1m]
)
# external checks (optional)
# testing (optional)
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
@@ -149,31 +149,21 @@ For more details on how the canary analysis and promotion works please [read the
## Features
| Service Mesh Feature | Istio | App Mesh | SuperGloo |
| -------------------------------------------- | ------------------ | ------------------ |------------------ |
| Canary deployments (weighted traffic) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| A/B testing (headers and cookies filters) | :heavy_check_mark: | :heavy_minus_sign: | :heavy_minus_sign: |
| Load testing | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Webhooks (acceptance testing) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Request success rate check (L7 metric) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Request duration check (L7 metric) | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark: |
| Custom promql checks | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Ingress gateway (CORS, retries and timeouts) | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark: |
| Ingress Controller Feature | NGINX | Gloo |
| -------------------------------------------- | ------------------ | ------------------ |
| Canary deployments (weighted traffic) | :heavy_check_mark: | :heavy_check_mark: |
| A/B testing (headers and cookies filters) | :heavy_check_mark: | :heavy_minus_sign: |
| Load testing | :heavy_check_mark: | :heavy_check_mark: |
| Webhooks (acceptance testing) | :heavy_check_mark: | :heavy_check_mark: |
| Request success rate check (L7 metric) | :heavy_minus_sign: | :heavy_check_mark: |
| Request duration check (L7 metric) | :heavy_minus_sign: | :heavy_check_mark: |
| Custom promql checks | :heavy_check_mark: | :heavy_check_mark: |
| Feature | Istio | Linkerd | App Mesh | NGINX | Gloo | Kubernetes CNI |
| -------------------------------------------- | ------------------ | ------------------ |------------------ |------------------ |------------------ |------------------ |
| Canary deployments (weighted traffic) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: |
| A/B testing (headers and cookies routing) | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: | :heavy_minus_sign: |
| Blue/Green deployments (traffic switch) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Webhooks (acceptance/load testing) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Manual gating (approve/pause/resume) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Request success rate check (L7 metric) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: |
| Request duration check (L7 metric) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: |
| Custom promql checks | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Traffic policy, CORS, retries and timeouts | :heavy_check_mark: | :heavy_minus_sign: | :heavy_minus_sign: | :heavy_minus_sign: | :heavy_minus_sign: | :heavy_minus_sign: |
## Roadmap
* Integrate with other service mesh technologies like Linkerd v2
* Integrate with other ingress controllers like Contour, HAProxy, ALB
* Add support for comparing the canary metrics to the primary ones and do the validation based on the derivation between the two
## Contributing

View File

@@ -1,67 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: abtest
namespace: test
labels:
app: abtest
spec:
minReadySeconds: 5
revisionHistoryLimit: 5
progressDeadlineSeconds: 60
strategy:
rollingUpdate:
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: abtest
template:
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
app: abtest
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.4.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898
name: http
protocol: TCP
command:
- ./podinfo
- --port=9898
- --level=info
- --random-delay=false
- --random-error=false
env:
- name: PODINFO_UI_COLOR
value: blue
livenessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/healthz
initialDelaySeconds: 5
timeoutSeconds: 5
readinessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/readyz
initialDelaySeconds: 5
timeoutSeconds: 5
resources:
limits:
cpu: 2000m
memory: 512Mi
requests:
cpu: 100m
memory: 64Mi

View File

@@ -1,19 +0,0 @@
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: abtest
namespace: test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: abtest
minReplicas: 2
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
# scale up if usage is above
# 99% of the requested CPU (100m)
targetAverageUtilization: 99

View File

@@ -20,8 +20,16 @@ spec:
service:
# container port
port: 9898
# container port name (optional)
# can be http or grpc
portName: http
# App Mesh reference
meshName: global
# App Mesh retry policy (optional)
retries:
attempts: 3
perTryTimeout: 1s
retryOn: "gateway-error,client-error,stream-error"
# define the canary analysis timing and KPIs
canaryAnalysis:
# schedule interval (default 60s)
@@ -41,8 +49,20 @@ spec:
# percentage (0-100)
threshold: 99
interval: 1m
# external checks (optional)
- name: request-duration
# maximum req duration P99
# milliseconds
threshold: 500
interval: 30s
# testing (optional)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s

View File

@@ -25,7 +25,7 @@ spec:
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.4.0
image: stefanprodan/podinfo:3.1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898

View File

@@ -13,7 +13,7 @@ data:
- address:
socket_address:
address: 0.0.0.0
port_value: 80
port_value: 8080
filter_chains:
- filters:
- name: envoy.http_connection_manager
@@ -48,11 +48,15 @@ data:
connect_timeout: 0.30s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
hosts:
- socket_address:
address: podinfo.test
port_value: 9898
load_assignment:
cluster_name: podinfo
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: podinfo.test
port_value: 9898
admin:
access_log_path: /dev/null
address:
@@ -91,7 +95,7 @@ spec:
terminationGracePeriodSeconds: 30
containers:
- name: ingress
image: "envoyproxy/envoy-alpine:d920944aed67425f91fc203774aebce9609e5d9a"
image: "envoyproxy/envoy-alpine:v1.11.1"
securityContext:
capabilities:
drop:
@@ -99,25 +103,20 @@ spec:
add:
- NET_BIND_SERVICE
command:
- /usr/bin/dumb-init
- --
args:
- /usr/local/bin/envoy
- --base-id 30
- --v2-config-only
args:
- -l
- $loglevel
- -c
- /config/envoy.yaml
- --base-id
- "1234"
ports:
- name: admin
containerPort: 9999
protocol: TCP
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
containerPort: 8080
protocol: TCP
livenessProbe:
initialDelaySeconds: 5
@@ -151,11 +150,7 @@ spec:
- protocol: TCP
name: http
port: 80
targetPort: 80
- protocol: TCP
name: https
port: 443
targetPort: 443
targetPort: http
type: LoadBalancer
---
apiVersion: appmesh.k8s.aws/v1beta1

View File

@@ -1,14 +1,14 @@
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: abtest
name: podinfo
namespace: test
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: abtest
name: podinfo
# the maximum time in seconds for the canary deployment
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
@@ -16,7 +16,7 @@ spec:
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: abtest
name: podinfo
service:
# container port
port: 9898
@@ -26,7 +26,12 @@ spec:
- mesh
# Istio virtual service host names (optional)
hosts:
- abtest.istio.weavedx.com
- app.example.com
# Istio traffic policy (optional)
trafficPolicy:
tls:
# use ISTIO_MUTUAL when mTLS is enabled
mode: DISABLE
canaryAnalysis:
# schedule interval (default 60s)
interval: 10s
@@ -36,12 +41,12 @@ spec:
iterations: 10
# canary match condition
match:
- headers:
user-agent:
regex: "^(?!.*Chrome)(?=.*\bSafari\b).*$"
- headers:
cookie:
regex: "^(.*?;)?(type=insider)(;.*)?$"
- headers:
user-agent:
regex: "(?=.*Safari)(?!.*Chrome).*$"
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)

View File

@@ -4,6 +4,10 @@ metadata:
name: podinfo
namespace: test
spec:
# service mesh provider (default istio)
# can be: kubernetes, istio, appmesh, smi, nginx, gloo, supergloo
# use the kubernetes provider for Blue/Green style deployments
provider: istio
# deployment reference
targetRef:
apiVersion: apps/v1
@@ -20,17 +24,25 @@ spec:
service:
# container port
port: 9898
# port name can be http or grpc (default http)
portName: http
# add all the other container ports
# when generating ClusterIP services (default false)
portDiscovery: false
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
# remove the mesh gateway if the public host is
# shared across multiple virtual services
- mesh
# Istio virtual service host names (optional)
hosts:
- app.istio.weavedx.com
- app.example.com
# Istio traffic policy (optional)
trafficPolicy:
loadBalancer:
simple: LEAST_CONN
tls:
# use ISTIO_MUTUAL when mTLS is enabled
mode: DISABLE
# HTTP match conditions (optional)
match:
- uri:

View File

@@ -20,12 +20,13 @@ spec:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9898"
labels:
app: podinfo
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.4.0
image: stefanprodan/podinfo:3.1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898

View File

@@ -5,17 +5,17 @@ metadata:
namespace: test
annotations:
flux.weave.works/automated: "true"
flux.weave.works/tag.chart-image: regexp:^1.4.*
flux.weave.works/tag.chart-image: regexp:^1.7.*
spec:
releaseName: backend
chart:
repository: https://flagger.app/
name: podinfo
version: 2.0.0
version: 2.2.0
values:
image:
repository: quay.io/stefanprodan/podinfo
tag: 1.4.0
tag: 1.7.0
httpServer:
timeout: 30s
canary:

View File

@@ -5,17 +5,17 @@ metadata:
namespace: test
annotations:
flux.weave.works/automated: "true"
flux.weave.works/tag.chart-image: semver:~1.4
flux.weave.works/tag.chart-image: semver:~1.7
spec:
releaseName: frontend
chart:
repository: https://flagger.app/
name: podinfo
version: 2.0.0
version: 2.2.0
values:
image:
repository: quay.io/stefanprodan/podinfo
tag: 1.4.0
tag: 1.7.0
backend: http://backend-podinfo:9898/echo
canary:
enabled: true

View File

@@ -11,8 +11,8 @@ spec:
chart:
repository: https://flagger.app/
name: loadtester
version: 0.1.0
version: 0.6.0
values:
image:
repository: quay.io/stefanprodan/flagger-loadtester
tag: 0.1.0
repository: weaveworks/flagger-loadtester
tag: 0.6.1

View File

@@ -1,59 +0,0 @@
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
# the maximum time in seconds for the canary deployment
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
# HPA reference (optional)
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: podinfo
service:
# container port
port: 9898
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
# Istio virtual service host names (optional)
hosts:
- app.iowa.weavedx.com
canaryAnalysis:
# schedule interval (default 60s)
interval: 10s
# max number of failed metric checks before rollback
threshold: 10
# max traffic percentage routed to canary
# percentage (0-100)
maxWeight: 50
# canary increment step
# percentage (0-100)
stepWeight: 5
# Istio Prometheus checks
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)
# percentage (0-100)
threshold: 99
interval: 1m
- name: request-duration
# maximum req duration P99
# milliseconds
threshold: 500
interval: 30s
# external checks (optional)
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

View File

@@ -1,16 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: podinfo-config-env
namespace: test
data:
color: blue
---
apiVersion: v1
kind: ConfigMap
metadata:
name: podinfo-config-vol
namespace: test
data:
output: console
textmode: "true"

View File

@@ -1,89 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
minReadySeconds: 5
revisionHistoryLimit: 5
progressDeadlineSeconds: 60
strategy:
rollingUpdate:
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: podinfo
template:
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
app: podinfo
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.3.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898
name: http
protocol: TCP
command:
- ./podinfo
- --port=9898
- --level=info
- --random-delay=false
- --random-error=false
env:
- name: PODINFO_UI_COLOR
valueFrom:
configMapKeyRef:
name: podinfo-config-env
key: color
- name: SECRET_USER
valueFrom:
secretKeyRef:
name: podinfo-secret-env
key: user
livenessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/healthz
initialDelaySeconds: 5
timeoutSeconds: 5
readinessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/readyz
initialDelaySeconds: 5
timeoutSeconds: 5
resources:
limits:
cpu: 2000m
memory: 512Mi
requests:
cpu: 100m
memory: 64Mi
volumeMounts:
- name: configs
mountPath: /etc/podinfo/configs
readOnly: true
- name: secrets
mountPath: /etc/podinfo/secrets
readOnly: true
volumes:
- name: configs
configMap:
name: podinfo-config-vol
- name: secrets
secret:
secretName: podinfo-secret-vol

View File

@@ -1,19 +0,0 @@
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
namespace: test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
# scale up if usage is above
# 99% of the requested CPU (100m)
targetAverageUtilization: 99

View File

@@ -1,16 +0,0 @@
apiVersion: v1
kind: Secret
metadata:
name: podinfo-secret-env
namespace: test
data:
password: cGFzc3dvcmQ=
user: YWRtaW4=
---
apiVersion: v1
kind: Secret
metadata:
name: podinfo-secret-vol
namespace: test
data:
key: cGFzc3dvcmQ=

View File

@@ -33,6 +33,26 @@ spec:
- name: Weight
type: string
JSONPath: .status.canaryWeight
- name: FailedChecks
type: string
JSONPath: .status.failedChecks
priority: 1
- name: Interval
type: string
JSONPath: .spec.canaryAnalysis.interval
priority: 1
- name: Mirror
type: boolean
JSONPath: .spec.canaryAnalysis.mirror
priority: 1
- name: StepWeight
type: string
JSONPath: .spec.canaryAnalysis.stepWeight
priority: 1
- name: MaxWeight
type: string
JSONPath: .spec.canaryAnalysis.maxWeight
priority: 1
- name: LastTransitionTime
type: string
JSONPath: .status.lastTransitionTime
@@ -45,11 +65,19 @@ spec:
- service
- canaryAnalysis
properties:
provider:
description: Traffic managent provider
type: string
metricsServer:
description: Prometheus URL
type: string
progressDeadlineSeconds:
description: Deployment progress deadline
type: number
targetRef:
description: Deployment selector
type: object
required: ['apiVersion', 'kind', 'name']
required: ["apiVersion", "kind", "name"]
properties:
apiVersion:
type: string
@@ -58,10 +86,11 @@ spec:
name:
type: string
autoscalerRef:
description: HPA selector
anyOf:
- type: string
- type: object
required: ['apiVersion', 'kind', 'name']
required: ["apiVersion", "kind", "name"]
properties:
apiVersion:
type: string
@@ -70,10 +99,11 @@ spec:
name:
type: string
ingressRef:
description: NGINX ingress selector
anyOf:
- type: string
- type: object
required: ['apiVersion', 'kind', 'name']
required: ["apiVersion", "kind", "name"]
properties:
apiVersion:
type: string
@@ -83,86 +113,207 @@ spec:
type: string
service:
type: object
required: ['port']
required: ["port"]
properties:
port:
description: Container port number
type: number
portName:
description: Container port name
type: string
targetPort:
description: Container target port name
anyOf:
- type: string
- type: number
portDiscovery:
description: Enable port dicovery
type: boolean
meshName:
description: AppMesh mesh name
type: string
backends:
description: AppMesh backend array
anyOf:
- type: string
- type: array
timeout:
description: Istio HTTP or gRPC request timeout
type: string
trafficPolicy:
description: Istio traffic policy
anyOf:
- type: string
- type: object
match:
description: Istio URL match conditions
anyOf:
- type: string
- type: array
rewrite:
description: Istio URL rewrite
anyOf:
- type: string
- type: object
headers:
description: Istio headers operations
anyOf:
- type: string
- type: object
corsPolicy:
description: Istio CORS policy
anyOf:
- type: string
- type: object
gateways:
description: Istio gateways list
anyOf:
- type: string
- type: array
hosts:
description: Istio hosts list
anyOf:
- type: string
- type: array
skipAnalysis:
type: boolean
canaryAnalysis:
properties:
interval:
description: Canary schedule interval
type: string
pattern: "^[0-9]+(m|s)"
iterations:
description: Number of checks to run for A/B Testing and Blue/Green
type: number
threshold:
description: Max number of failed checks before rollback
type: number
maxWeight:
description: Max traffic percentage routed to canary
type: number
stepWeight:
description: Canary incremental traffic percentage step
type: number
mirror:
description: Mirror traffic to canary before shifting
type: boolean
match:
description: A/B testing match conditions
anyOf:
- type: string
- type: array
metrics:
description: Prometheus query list for this canary
type: array
properties:
items:
type: object
required: ['name', 'threshold']
required: ["name", "threshold"]
properties:
name:
description: Name of the Prometheus metric
type: string
interval:
description: Interval of the promql query
type: string
pattern: "^[0-9]+(m|s)"
threshold:
description: Max scalar value accepted for this metric
type: number
query:
description: Prometheus query
type: string
webhooks:
description: Webhook list for this canary
type: array
properties:
items:
type: object
required: ['name', 'url', 'timeout']
required: ["name", "url"]
properties:
name:
description: Name of the webhook
type: string
type:
description: Type of the webhook pre, post or during rollout
type: string
enum:
- ""
- confirm-rollout
- pre-rollout
- rollout
- confirm-promotion
- post-rollout
url:
description: URL address of this webhook
type: string
format: url
timeout:
description: Request timeout for this webhook
type: string
pattern: "^[0-9]+(m|s)"
metadata:
description: Metadata (key-value pairs) for this webhook
anyOf:
- type: string
- type: object
status:
properties:
phase:
description: Analysis phase of this canary
type: string
enum:
- ""
- Initializing
- Initialized
- Waiting
- Progressing
- Promoting
- Finalising
- Succeeded
- Failed
canaryWeight:
description: Traffic weight percentage routed to canary
type: number
failedChecks:
description: Failed check count of the current canary analysis
type: number
iterations:
description: Iteration count of the current canary analysis
type: number
lastAppliedSpec:
description: LastAppliedSpec of this canary
type: string
lastTransitionTime:
description: LastTransitionTime of this canary
format: date-time
type: string
conditions:
description: Status conditions of this canary
type: array
properties:
items:
type: object
required: ["type", "status", "reason"]
properties:
lastTransitionTime:
description: LastTransitionTime of this condition
format: date-time
type: string
lastUpdateTime:
description: LastUpdateTime of this condition
format: date-time
type: string
message:
description: Message associated with this condition
type: string
reason:
description: Reason for the current status of this condition
type: string
status:
description: Status of this condition
type: string
type:
description: Type of this condition
type: string

View File

@@ -22,7 +22,7 @@ spec:
serviceAccountName: flagger
containers:
- name: flagger
image: weaveworks/flagger:0.15.0
image: weaveworks/flagger:0.20.2
imagePullPolicy: IfNotPresent
ports:
- name: http

View File

@@ -1,49 +1,4 @@
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
labels:
app: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
labels:
app: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: istio-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: istio-system
labels:
app: prometheus
---
# Source: istio/charts/prometheus/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
@@ -51,6 +6,9 @@ metadata:
namespace: istio-system
labels:
app: prometheus
chart: prometheus-1.0.6
heritage: Tiller
release: istio
data:
prometheus.yml: |-
global:
@@ -363,6 +321,70 @@ data:
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod_name
---
# Source: istio/charts/prometheus/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus-istio-system
labels:
app: prometheus
chart: prometheus-1.0.6
heritage: Tiller
release: istio
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
# Source: istio/charts/prometheus/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: istio-system
labels:
app: prometheus
chart: prometheus-1.0.6
heritage: Tiller
release: istio
---
# Source: istio/charts/prometheus/templates/clusterrolebindings.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus-istio-system
labels:
app: prometheus
chart: prometheus-1.0.6
heritage: Tiller
release: istio
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-istio-system
subjects:
- kind: ServiceAccount
name: prometheus
namespace: istio-system
---
# Source: istio/charts/prometheus/templates/service.yaml
@@ -384,13 +406,18 @@ spec:
port: 9090
---
apiVersion: apps/v1
# Source: istio/charts/prometheus/templates/deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: prometheus
namespace: istio-system
labels:
app: prometheus
chart: prometheus-1.0.6
heritage: Tiller
release: istio
spec:
replicas: 1
selector:
@@ -407,7 +434,7 @@ spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: "docker.io/prom/prometheus:v2.7.1"
image: "docker.io/prom/prometheus:v2.3.1"
imagePullPolicy: IfNotPresent
args:
- '--storage.tsdb.retention=6h'
@@ -441,3 +468,367 @@ spec:
defaultMode: 420
optional: true
secretName: istio.default
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 2
preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- weight: 2
preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
- weight: 2
preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: requestcount
namespace: istio-system
spec:
value: "1"
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
source_workload: source.workload.name | "unknown"
source_workload_namespace: source.workload.namespace | "unknown"
source_principal: source.principal | "unknown"
source_app: source.labels["app"] | "unknown"
source_version: source.labels["version"] | "unknown"
destination_workload: destination.workload.name | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_principal: destination.principal | "unknown"
destination_app: destination.labels["app"] | "unknown"
destination_version: destination.labels["version"] | "unknown"
destination_service: destination.service.host | "unknown"
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
request_protocol: api.protocol | context.protocol | "unknown"
response_code: response.code | 200
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: requestduration
namespace: istio-system
spec:
value: response.duration | "0ms"
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
source_workload: source.workload.name | "unknown"
source_workload_namespace: source.workload.namespace | "unknown"
source_principal: source.principal | "unknown"
source_app: source.labels["app"] | "unknown"
source_version: source.labels["version"] | "unknown"
destination_workload: destination.workload.name | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_principal: destination.principal | "unknown"
destination_app: destination.labels["app"] | "unknown"
destination_version: destination.labels["version"] | "unknown"
destination_service: destination.service.host | "unknown"
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
request_protocol: api.protocol | context.protocol | "unknown"
response_code: response.code | 200
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: requestsize
namespace: istio-system
spec:
value: request.size | 0
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
source_workload: source.workload.name | "unknown"
source_workload_namespace: source.workload.namespace | "unknown"
source_principal: source.principal | "unknown"
source_app: source.labels["app"] | "unknown"
source_version: source.labels["version"] | "unknown"
destination_workload: destination.workload.name | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_principal: destination.principal | "unknown"
destination_app: destination.labels["app"] | "unknown"
destination_version: destination.labels["version"] | "unknown"
destination_service: destination.service.host | "unknown"
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
request_protocol: api.protocol | context.protocol | "unknown"
response_code: response.code | 200
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: responsesize
namespace: istio-system
spec:
value: response.size | 0
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
source_workload: source.workload.name | "unknown"
source_workload_namespace: source.workload.namespace | "unknown"
source_principal: source.principal | "unknown"
source_app: source.labels["app"] | "unknown"
source_version: source.labels["version"] | "unknown"
destination_workload: destination.workload.name | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_principal: destination.principal | "unknown"
destination_app: destination.labels["app"] | "unknown"
destination_version: destination.labels["version"] | "unknown"
destination_service: destination.service.host | "unknown"
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
request_protocol: api.protocol | context.protocol | "unknown"
response_code: response.code | 200
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: tcpbytesent
namespace: istio-system
spec:
value: connection.sent.bytes | 0
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
source_workload: source.workload.name | "unknown"
source_workload_namespace: source.workload.namespace | "unknown"
source_principal: source.principal | "unknown"
source_app: source.labels["app"] | "unknown"
source_version: source.labels["version"] | "unknown"
destination_workload: destination.workload.name | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_principal: destination.principal | "unknown"
destination_app: destination.labels["app"] | "unknown"
destination_version: destination.labels["version"] | "unknown"
destination_service: destination.service.name | "unknown"
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
name: tcpbytereceived
namespace: istio-system
spec:
value: connection.received.bytes | 0
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
source_workload: source.workload.name | "unknown"
source_workload_namespace: source.workload.namespace | "unknown"
source_principal: source.principal | "unknown"
source_app: source.labels["app"] | "unknown"
source_version: source.labels["version"] | "unknown"
destination_workload: destination.workload.name | "unknown"
destination_workload_namespace: destination.workload.namespace | "unknown"
destination_principal: destination.principal | "unknown"
destination_app: destination.labels["app"] | "unknown"
destination_version: destination.labels["version"] | "unknown"
destination_service: destination.service.name | "unknown"
destination_service_name: destination.service.name | "unknown"
destination_service_namespace: destination.service.namespace | "unknown"
connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: prometheus
metadata:
name: handler
namespace: istio-system
spec:
metrics:
- name: requests_total
instance_name: requestcount.metric.istio-system
kind: COUNTER
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- request_protocol
- response_code
- connection_security_policy
- name: request_duration_seconds
instance_name: requestduration.metric.istio-system
kind: DISTRIBUTION
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- request_protocol
- response_code
- connection_security_policy
buckets:
explicit_buckets:
bounds: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
- name: request_bytes
instance_name: requestsize.metric.istio-system
kind: DISTRIBUTION
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- request_protocol
- response_code
- connection_security_policy
buckets:
exponentialBuckets:
numFiniteBuckets: 8
scale: 1
growthFactor: 10
- name: response_bytes
instance_name: responsesize.metric.istio-system
kind: DISTRIBUTION
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- request_protocol
- response_code
- connection_security_policy
buckets:
exponentialBuckets:
numFiniteBuckets: 8
scale: 1
growthFactor: 10
- name: tcp_sent_bytes_total
instance_name: tcpbytesent.metric.istio-system
kind: COUNTER
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- connection_security_policy
- name: tcp_received_bytes_total
instance_name: tcpbytereceived.metric.istio-system
kind: COUNTER
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- connection_security_policy
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: promhttp
namespace: istio-system
spec:
match: context.protocol == "http" || context.protocol == "grpc"
actions:
- handler: handler.prometheus
instances:
- requestcount.metric
- requestduration.metric
- requestsize.metric
- responsesize.metric
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: promtcp
namespace: istio-system
spec:
match: context.protocol == "tcp"
actions:
- handler: handler.prometheus
instances:
- tcpbytesent.metric
- tcpbytereceived.metric
---

View File

@@ -4,6 +4,7 @@ metadata:
name: podinfo
namespace: test
spec:
provider: gloo
targetRef:
apiVersion: apps/v1
kind: Deployment
@@ -28,9 +29,24 @@ spec:
threshold: 500
interval: 30s
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 10s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
- name: gloo-acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 10s
metadata:
type: bash
cmd: "curl -sd 'test' -H 'Host: app.example.com' http://gateway-proxy-v2.gloo-system/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://gloo.example.com/"
cmd: "hey -z 2m -q 5 -c 2 -host app.example.com http://gateway-proxy-v2.gloo-system"
logCmdOutput: "true"

View File

@@ -1,19 +0,0 @@
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
namespace: test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
# scale up if usage is above
# 99% of the requested CPU (100m)
targetAverageUtilization: 99

View File

@@ -7,11 +7,11 @@ spec:
virtualHost:
domains:
- '*'
name: podinfo.default
name: podinfo
routes:
- matcher:
prefix: /
routeAction:
upstreamGroup:
name: podinfo
namespace: gloo
namespace: test

View File

@@ -19,7 +19,7 @@ spec:
serviceAccountName: tiller
containers:
- name: helmtester
image: weaveworks/flagger-loadtester:0.4.0
image: weaveworks/flagger-loadtester:0.8.0
imagePullPolicy: IfNotPresent
ports:
- name: http

View File

@@ -17,7 +17,7 @@ spec:
spec:
containers:
- name: loadtester
image: weaveworks/flagger-loadtester:0.3.0
image: weaveworks/flagger-loadtester:0.11.0
imagePullPolicy: IfNotPresent
ports:
- name: http

View File

@@ -23,8 +23,10 @@ spec:
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
service:
# container port
port: 9898
# ClusterIP port number
port: 80
# container port number or name
targetPort: 9898
canaryAnalysis:
# schedule interval (default 60s)
interval: 10s

View File

@@ -1,69 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: podinfo
template:
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
app: podinfo
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.4.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898
name: http
protocol: TCP
command:
- ./podinfo
- --port=9898
- --level=info
- --random-delay=false
- --random-error=false
env:
- name: PODINFO_UI_COLOR
value: green
livenessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/healthz
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
readinessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/readyz
failureThreshold: 3
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 2
resources:
limits:
cpu: 1000m
memory: 256Mi
requests:
cpu: 100m
memory: 16Mi

View File

@@ -1,45 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: podinfo
namespace: test
spec:
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
hosts:
- podinfo.istio.weavedx.com
- podinfo
http:
- route:
- destination:
host: podinfo
subset: primary
weight: 50
- destination:
host: podinfo
subset: canary
weight: 50
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: podinfo-destination
namespace: test
spec:
host: podinfo
trafficPolicy:
loadBalancer:
consistentHash:
httpCookie:
name: istiouser
ttl: 30s
subsets:
- name: primary
labels:
app: podinfo
role: primary
- name: canary
labels:
app: podinfo
role: canary

View File

@@ -1,43 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: podinfo
namespace: test
spec:
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
hosts:
- app.istio.weavedx.com
- podinfo
http:
- match:
- headers:
user-agent:
regex: ^(?!.*Chrome)(?=.*\bSafari\b).*$
uri:
prefix: "/version/"
rewrite:
uri: /api/info
route:
- destination:
host: podinfo-primary
port:
number: 9898
weight: 0
- destination:
host: podinfo
port:
number: 9898
weight: 100
- match:
- uri:
prefix: "/version/"
rewrite:
uri: /api/info
route:
- destination:
host: podinfo-primary
port:
number: 9898
weight: 100

View File

@@ -1,25 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
hosts:
- podinfo.iowa.weavedx.com
- podinfo
http:
- route:
- destination:
host: podinfo-primary
port:
number: 9898
weight: 100
mirror:
host: podinfo
port:
number: 9898

View File

@@ -1,26 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
hosts:
- podinfo.iowa.weavedx.com
- podinfo
http:
- route:
- destination:
host: podinfo-primary
port:
number: 9898
weight: 100
- destination:
host: podinfo
port:
number: 9898
weight: 0

View File

@@ -1,131 +0,0 @@
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: trafficsplits.split.smi-spec.io
spec:
additionalPrinterColumns:
- JSONPath: .spec.service
description: The service
name: Service
type: string
group: split.smi-spec.io
names:
kind: TrafficSplit
listKind: TrafficSplitList
plural: trafficsplits
singular: trafficsplit
scope: Namespaced
subresources:
status: {}
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: smi-adapter-istio
namespace: istio-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: smi-adapter-istio
rules:
- apiGroups:
- ""
resources:
- pods
- services
- endpoints
- persistentvolumeclaims
- events
- configmaps
- secrets
verbs:
- '*'
- apiGroups:
- apps
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
verbs:
- '*'
- apiGroups:
- monitoring.coreos.com
resources:
- servicemonitors
verbs:
- get
- create
- apiGroups:
- apps
resourceNames:
- smi-adapter-istio
resources:
- deployments/finalizers
verbs:
- update
- apiGroups:
- split.smi-spec.io
resources:
- '*'
verbs:
- '*'
- apiGroups:
- networking.istio.io
resources:
- '*'
verbs:
- '*'
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: smi-adapter-istio
subjects:
- kind: ServiceAccount
name: smi-adapter-istio
namespace: istio-system
roleRef:
kind: ClusterRole
name: smi-adapter-istio
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: smi-adapter-istio
namespace: istio-system
spec:
replicas: 1
selector:
matchLabels:
name: smi-adapter-istio
template:
metadata:
labels:
name: smi-adapter-istio
annotations:
sidecar.istio.io/inject: "false"
spec:
serviceAccountName: smi-adapter-istio
containers:
- name: smi-adapter-istio
image: docker.io/stefanprodan/smi-adapter-istio:0.0.2-beta.1
command:
- smi-adapter-istio
imagePullPolicy: Always
env:
- name: WATCH_NAMESPACE
value: ""
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "smi-adapter-istio"

View File

@@ -1,69 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: podinfo
template:
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
app: podinfo
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.4.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898
name: http
protocol: TCP
command:
- ./podinfo
- --port=9898
- --level=info
- --random-delay=false
- --random-error=false
env:
- name: PODINFO_UI_COLOR
value: green
livenessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/healthz
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
readinessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/readyz
failureThreshold: 3
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 2
resources:
limits:
cpu: 1000m
memory: 256Mi
requests:
cpu: 100m
memory: 16Mi

View File

@@ -1,13 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: podinfo-canary
namespace: test
spec:
host: podinfo-canary
trafficPolicy:
loadBalancer:
consistentHash:
httpCookie:
name: user
ttl: 0s

View File

@@ -1,23 +0,0 @@
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
namespace: test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
minReplicas: 2
maxReplicas: 2
metrics:
- type: Resource
resource:
name: cpu
# scale up if usage is above
# 99% of the requested CPU (100m)
targetAverageUtilization: 99
- type: Resource
resource:
name: memory
targetAverageValue: 200Mi

View File

@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: podinfo-canary
namespace: test
spec:
type: ClusterIP
selector:
app: podinfo
ports:
- name: http
port: 9898
protocol: TCP
targetPort: http

View File

@@ -1,71 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo-primary
namespace: test
labels:
app: podinfo-primary
spec:
replicas: 1
strategy:
rollingUpdate:
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: podinfo-primary
template:
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
app: podinfo-primary
spec:
containers:
- name: podinfod
image: quay.io/stefanprodan/podinfo:1.4.1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9898
name: http
protocol: TCP
command:
- ./podinfo
- --port=9898
- --level=info
- --random-delay=false
- --random-error=false
env:
- name: PODINFO_UI_COLOR
value: blue
livenessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/healthz
initialDelaySeconds: 5
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
exec:
command:
- podcli
- check
- http
- localhost:9898/readyz
initialDelaySeconds: 5
failureThreshold: 3
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 2000m
memory: 512Mi
requests:
cpu: 10m
memory: 64Mi

View File

@@ -1,13 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: podinfo-primary
namespace: test
spec:
host: podinfo-primary
trafficPolicy:
loadBalancer:
consistentHash:
httpCookie:
name: user
ttl: 0s

View File

@@ -1,19 +0,0 @@
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo-primary
namespace: test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo-primary
minReplicas: 2
maxReplicas: 2
metrics:
- type: Resource
resource:
name: cpu
# scale up if usage is above
# 99% of the requested CPU (100m)
targetAverageUtilization: 99

View File

@@ -1,16 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: podinfo-primary
namespace: test
labels:
app: podinfo-primary
spec:
type: ClusterIP
selector:
app: podinfo-primary
ports:
- name: http
port: 9898
protocol: TCP
targetPort: http

View File

@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: podinfo
namespace: test
spec:
type: ClusterIP
selector:
app: podinfo-primary
ports:
- name: http
port: 9898
protocol: TCP
targetPort: http

View File

@@ -1,23 +0,0 @@
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
spec:
gateways:
- mesh
- public-gateway.istio-system.svc.cluster.local
hosts:
- podinfo
- app.istio.weavedx.com
http:
- route:
- destination:
host: podinfo-primary
weight: 50
- destination:
host: podinfo-canary
weight: 50
timeout: 5s

View File

@@ -1,10 +1,10 @@
apiVersion: v1
name: flagger
version: 0.15.0
appVersion: 0.15.0
version: 0.20.2
appVersion: 0.20.2
kubeVersion: ">=1.11.0-0"
engine: gotpl
description: Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, App Mesh or NGINX routing for traffic shifting and Prometheus metrics for canary analysis.
description: Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, Gloo or NGINX routing for traffic shifting and Prometheus metrics for canary analysis.
home: https://docs.flagger.app
icon: https://raw.githubusercontent.com/weaveworks/flagger/master/docs/logo/flagger-icon.png
sources:
@@ -17,4 +17,5 @@ keywords:
- canary
- istio
- appmesh
- linkerd
- gitops

View File

@@ -1,15 +1,14 @@
# Flagger
[Flagger](https://github.com/weaveworks/flagger) is a Kubernetes operator that automates the promotion of
canary deployments using Istio routing for traffic shifting and Prometheus metrics for canary analysis.
canary deployments using Istio, Linkerd, App Mesh, NGINX or Gloo routing for traffic shifting and Prometheus metrics for canary analysis.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators
like HTTP requests success rate, requests average duration and pods health.
Based on the KPIs analysis a canary is promoted or aborted and the analysis result is published to Slack.
Based on the KPIs analysis a canary is promoted or aborted and the analysis result is published to Slack or MS Teams.
## Prerequisites
* Kubernetes >= 1.11
* Istio >= 1.0
* Prometheus >= 2.6
## Installing the Chart
@@ -17,16 +16,35 @@ Based on the KPIs analysis a canary is promoted or aborted and the analysis resu
Add Flagger Helm repository:
```console
helm repo add flagger https://flagger.app
$ helm repo add flagger https://flagger.app
```
To install the chart with the release name `flagger`:
Install Flagger's custom resource definitions:
```console
$ helm install --name flagger --namespace istio-system flagger/flagger
$ kubectl apply -f https://raw.githubusercontent.com/weaveworks/flagger/master/artifacts/flagger/crd.yaml
```
To install the chart with the release name `flagger` for Istio:
```console
$ helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set meshProvider=istio \
--set metricsServer=http://prometheus:9090
```
To install the chart with the release name `flagger` for Linkerd:
```console
$ helm upgrade -i flagger flagger/flagger \
--namespace=linkerd \
--set crd.create=false \
--set meshProvider=linkerd \
--set metricsServer=http://linkerd-prometheus:9090
```
The command deploys Flagger on the Kubernetes cluster in the istio-system namespace.
The [configuration](#configuration) section lists the parameters that can be configured during installation.
## Uninstalling the Chart
@@ -48,11 +66,18 @@ Parameter | Description | Default
`image.repository` | image repository | `weaveworks/flagger`
`image.tag` | image tag | `<VERSION>`
`image.pullPolicy` | image pull policy | `IfNotPresent`
`metricsServer` | Prometheus URL | `http://prometheus.istio-system:9090`
`prometheus.install` | if `true`, installs Prometheus configured to scrape all pods in the custer including the App Mesh sidecar | `false`
`metricsServer` | Prometheus URL, used when `prometheus.install` is `false` | `http://prometheus.istio-system:9090`
`selectorLabels` | list of labels that Flagger uses to create pod selectors | `app,name,app.kubernetes.io/name`
`slack.url` | Slack incoming webhook | None
`slack.channel` | Slack channel | None
`slack.user` | Slack username | `flagger`
`msteams.url` | Microsoft Teams incoming webhook | None
`leaderElection.enabled` | leader election must be enabled when running more than one replica | `false`
`leaderElection.replicaCount` | number of replicas | `1`
`ingressAnnotationsPrefix` | annotations prefix for ingresses | `custom.ingress.kubernetes.io`
`rbac.create` | if `true`, create and use RBAC resources | `true`
`rbac.pspEnabled` | If `true`, create and use a restricted pod security policy | `false`
`crd.create` | if `true`, create Flagger's CRDs | `true`
`resources.requests/cpu` | pod CPU request | `10m`
`resources.requests/memory` | pod memory request | `32Mi`
@@ -67,6 +92,7 @@ Specify each parameter using the `--set key=value[,key=value]` argument to `helm
```console
$ helm upgrade -i flagger flagger/flagger \
--namespace istio-system \
--set crd.create=false \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general
```

View File

@@ -34,6 +34,26 @@ spec:
- name: Weight
type: string
JSONPath: .status.canaryWeight
- name: FailedChecks
type: string
JSONPath: .status.failedChecks
priority: 1
- name: Interval
type: string
JSONPath: .spec.canaryAnalysis.interval
priority: 1
- name: Mirror
type: boolean
JSONPath: .spec.canaryAnalysis.mirror
priority: 1
- name: StepWeight
type: string
JSONPath: .spec.canaryAnalysis.stepWeight
priority: 1
- name: MaxWeight
type: string
JSONPath: .spec.canaryAnalysis.maxWeight
priority: 1
- name: LastTransitionTime
type: string
JSONPath: .status.lastTransitionTime
@@ -46,9 +66,17 @@ spec:
- service
- canaryAnalysis
properties:
provider:
description: Traffic managent provider
type: string
metricsServer:
description: Prometheus URL
type: string
progressDeadlineSeconds:
description: Deployment progress deadline
type: number
targetRef:
description: Deployment selector
type: object
required: ['apiVersion', 'kind', 'name']
properties:
@@ -59,6 +87,7 @@ spec:
name:
type: string
autoscalerRef:
description: HPA selector
anyOf:
- type: string
- type: object
@@ -71,6 +100,7 @@ spec:
name:
type: string
ingressRef:
description: NGINX ingress selector
anyOf:
- type: string
- type: object
@@ -87,29 +117,95 @@ spec:
required: ['port']
properties:
port:
description: Container port number
type: number
portName:
description: Container port name
type: string
targetPort:
description: Container target port name
anyOf:
- type: string
- type: number
portDiscovery:
description: Enable port dicovery
type: boolean
meshName:
description: AppMesh mesh name
type: string
backends:
description: AppMesh backend array
anyOf:
- type: string
- type: array
timeout:
description: Istio HTTP or gRPC request timeout
type: string
trafficPolicy:
description: Istio traffic policy
anyOf:
- type: string
- type: object
match:
description: Istio URL match conditions
anyOf:
- type: string
- type: array
rewrite:
description: Istio URL rewrite
anyOf:
- type: string
- type: object
headers:
description: Istio headers operations
anyOf:
- type: string
- type: object
corsPolicy:
description: Istio CORS policy
anyOf:
- type: string
- type: object
gateways:
description: Istio gateways list
anyOf:
- type: string
- type: array
hosts:
description: Istio hosts list
anyOf:
- type: string
- type: array
skipAnalysis:
type: boolean
canaryAnalysis:
properties:
interval:
description: Canary schedule interval
type: string
pattern: "^[0-9]+(m|s)"
iterations:
description: Number of checks to run for A/B Testing and Blue/Green
type: number
threshold:
description: Max number of failed checks before rollback
type: number
maxWeight:
description: Max traffic percentage routed to canary
type: number
stepWeight:
description: Canary incremental traffic percentage step
type: number
mirror:
description: Mirror traffic to canary before shifting
type: boolean
match:
description: A/B testing match conditions
anyOf:
- type: string
- type: array
metrics:
description: Prometheus query list for this canary
type: array
properties:
items:
@@ -117,54 +213,109 @@ spec:
required: ['name', 'threshold']
properties:
name:
description: Name of the Prometheus metric
type: string
interval:
description: Interval of the promql query
type: string
pattern: "^[0-9]+(m|s)"
threshold:
description: Max scalar value accepted for this metric
type: number
query:
description: Prometheus query
type: string
webhooks:
description: Webhook list for this canary
type: array
properties:
items:
type: object
required: ['name', 'url', 'timeout']
required: ["name", "url"]
properties:
name:
description: Name of the webhook
type: string
type:
description: Type of the webhook pre, post or during rollout
type: string
enum:
- ""
- confirm-rollout
- pre-rollout
- rollout
- confirm-promotion
- post-rollout
url:
description: URL address of this webhook
type: string
format: url
timeout:
description: Request timeout for this webhook
type: string
pattern: "^[0-9]+(m|s)"
metadata:
description: Metadata (key-value pairs) for this webhook
anyOf:
- type: string
- type: object
status:
properties:
phase:
description: Analysis phase of this canary
type: string
enum:
- ""
- Initializing
- Initialized
- Waiting
- Progressing
- Promoting
- Finalising
- Succeeded
- Failed
canaryWeight:
description: Traffic weight percentage routed to canary
type: number
failedChecks:
description: Failed check count of the current canary analysis
type: number
iterations:
description: Iteration count of the current canary analysis
type: number
lastAppliedSpec:
description: LastAppliedSpec of this canary
type: string
lastTransitionTime:
description: LastTransitionTime of this canary
format: date-time
type: string
conditions:
description: Status conditions of this canary
type: array
properties:
items:
type: object
required: ['type', 'status', 'reason']
properties:
lastTransitionTime:
description: LastTransitionTime of this condition
format: date-time
type: string
lastUpdateTime:
description: LastUpdateTime of this condition
format: date-time
type: string
message:
description: Message associated with this condition
type: string
reason:
description: Reason for the current status of this condition
type: string
status:
description: Status of this condition
type: string
type:
description: Type of this condition
type: string
{{- end }}

View File

@@ -8,7 +8,7 @@ metadata:
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
replicas: 1
replicas: {{ .Values.leaderElection.replicaCount }}
strategy:
type: Recreate
selector:
@@ -20,8 +20,26 @@ spec:
labels:
app.kubernetes.io/name: {{ template "flagger.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
annotations:
{{- if .Values.podAnnotations }}
{{ toYaml .Values.podAnnotations | indent 8 }}
{{- end }}
spec:
serviceAccountName: {{ template "flagger.serviceAccountName" . }}
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template "flagger.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
topologyKey: kubernetes.io/hostname
{{- if .Values.image.pullSecret }}
imagePullSecrets:
- name: {{ .Values.image.pullSecret }}
{{- end }}
containers:
- name: flagger
securityContext:
@@ -43,6 +61,9 @@ spec:
{{- else }}
- -metrics-server={{ .Values.metricsServer }}
{{- end }}
{{- if .Values.selectorLabels }}
- -selector-labels={{ .Values.selectorLabels }}
{{- end }}
{{- if .Values.namespace }}
- -namespace={{ .Values.namespace }}
{{- end }}
@@ -51,6 +72,16 @@ spec:
- -slack-user={{ .Values.slack.user }}
- -slack-channel={{ .Values.slack.channel }}
{{- end }}
{{- if .Values.msteams.url }}
- -msteams-url={{ .Values.msteams.url }}
{{- end }}
{{- if .Values.leaderElection.enabled }}
- -enable-leader-election=true
- -leader-election-namespace={{ .Release.Namespace }}
{{- end }}
{{- if .Values.ingressAnnotationsPrefix }}
- -ingress-annotations-prefix={{ .Values.ingressAnnotationsPrefix }}
{{- end }}
livenessProbe:
exec:
command:
@@ -71,14 +102,14 @@ spec:
- --spider
- http://localhost:8080/healthz
timeoutSeconds: 5
{{- if .Values.env }}
env:
{{ toYaml .Values.env | indent 12 }}
{{- end }}
resources:
{{ toYaml .Values.resources | indent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{ toYaml . | indent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{ toYaml . | indent 8 }}
{{- end }}
{{- with .Values.tolerations }}

View File

@@ -133,38 +133,22 @@ data:
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: kubernetes;https
# Scrape config for nodes
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
# scrape config for cAdvisor
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
@@ -174,6 +158,14 @@ data:
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
# exclude high cardinality metrics
metric_relabel_configs:
- source_labels: [__name__]
regex: (container|machine)_(cpu|memory|network|fs)_(.+)
action: keep
- source_labels: [__name__]
regex: container_memory_failures_total
action: drop
# scrape config for pods
- job_name: kubernetes-pods
@@ -238,10 +230,10 @@ spec:
serviceAccountName: {{ template "flagger.serviceAccountName" . }}-prometheus
containers:
- name: prometheus
image: "docker.io/prom/prometheus:v2.7.1"
image: "docker.io/prom/prometheus:v2.12.0"
imagePullPolicy: IfNotPresent
args:
- '--storage.tsdb.retention=6h'
- '--storage.tsdb.retention=2h'
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- containerPort: 9090

View File

@@ -0,0 +1,66 @@
{{- if .Values.rbac.pspEnabled }}
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: {{ template "flagger.fullname" . }}
labels:
helm.sh/chart: {{ template "flagger.chart" . }}
app.kubernetes.io/name: {{ template "flagger.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
spec:
privileged: false
hostIPC: false
hostNetwork: false
hostPID: false
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
allowedCapabilities:
- '*'
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: {{ template "flagger.fullname" . }}-psp
labels:
helm.sh/chart: {{ template "flagger.chart" . }}
app.kubernetes.io/name: {{ template "flagger.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
rules:
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- {{ template "flagger.fullname" . }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ template "flagger.fullname" . }}-psp
labels:
helm.sh/chart: {{ template "flagger.chart" . }}
app.kubernetes.io/name: {{ template "flagger.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ template "flagger.fullname" . }}-psp
subjects:
- kind: ServiceAccount
name: {{ template "flagger.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end }}

View File

@@ -2,23 +2,53 @@
image:
repository: weaveworks/flagger
tag: 0.15.0
tag: 0.20.2
pullPolicy: IfNotPresent
pullSecret:
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
metricsServer: "http://prometheus:9090"
# accepted values are istio, appmesh, nginx or supergloo:mesh.namespace (defaults to istio)
# accepted values are kubernetes, istio, linkerd, appmesh, nginx, gloo or supergloo:mesh.namespace (defaults to istio)
meshProvider: ""
# single namespace restriction
namespace: ""
# list of pod labels that Flagger uses to create pod selectors
# defaults to: app,name,app.kubernetes.io/name
selectorLabels: ""
slack:
user: flagger
channel:
# incoming webhook https://api.slack.com/incoming-webhooks
url:
msteams:
# MS Teams incoming webhook URL
url:
#env:
#- name: SLACK_URL
# valueFrom:
# secretKeyRef:
# name: slack
# key: url
#- name: MSTEAMS_URL
# valueFrom:
# secretKeyRef:
# name: msteams
# key: url
env: []
leaderElection:
enabled: false
replicaCount: 1
serviceAccount:
# serviceAccount.create: Whether to create a service account or not
create: true
@@ -28,6 +58,8 @@ serviceAccount:
rbac:
# rbac.create: `true` if rbac resources should be created
create: true
# rbac.pspEnabled: `true` if PodSecurityPolicy resources should be created
pspEnabled: false
crd:
# crd.create: `true` if custom resource definitions should be created
@@ -48,8 +80,6 @@ nodeSelector: {}
tolerations: []
affinity: {}
prometheus:
# to be used with AppMesh or nginx ingress
install: false

View File

@@ -1,7 +1,7 @@
apiVersion: v1
name: grafana
version: 1.2.0
appVersion: 5.4.3
version: 1.3.0
appVersion: 6.2.5
description: Grafana dashboards for monitoring Flagger canary deployments
icon: https://raw.githubusercontent.com/weaveworks/flagger/master/docs/logo/flagger-icon.png
home: https://flagger.app

View File

@@ -20,6 +20,9 @@ spec:
release: {{ .Release.Name }}
annotations:
prometheus.io/scrape: 'false'
{{- if .Values.podAnnotations }}
{{ toYaml .Values.podAnnotations | indent 8 }}
{{- end }}
spec:
containers:
- name: {{ .Chart.Name }}

View File

@@ -6,9 +6,11 @@ replicaCount: 1
image:
repository: grafana/grafana
tag: 5.4.3
tag: 6.2.5
pullPolicy: IfNotPresent
podAnnotations: {}
service:
type: ClusterIP
port: 80

View File

@@ -1,10 +1,10 @@
apiVersion: v1
name: loadtester
version: 0.4.1
appVersion: 0.4.0
version: 0.11.0
appVersion: 0.11.0
kubeVersion: ">=1.11.0-0"
engine: gotpl
description: Flagger's load testing services based on rakyll/hey that generates traffic during canary analysis when configured as a webhook.
description: Flagger's load testing services based on rakyll/hey and bojand/ghz that generates traffic during canary analysis when configured as a webhook.
home: https://docs.flagger.app
icon: https://raw.githubusercontent.com/weaveworks/flagger/master/docs/logo/flagger-icon.png
sources:

View File

@@ -18,9 +18,14 @@ spec:
app: {{ include "loadtester.name" . }}
annotations:
appmesh.k8s.aws/ports: "444"
{{- if .Values.podAnnotations }}
{{ toYaml .Values.podAnnotations | indent 8 }}
{{- end }}
spec:
{{- if .Values.serviceAccountName }}
serviceAccountName: {{ .Values.serviceAccountName }}
{{- else if .Values.rbac.create }}
serviceAccountName: {{ include "loadtester.fullname" . }}
{{- end }}
containers:
- name: {{ .Chart.Name }}

View File

@@ -0,0 +1,54 @@
---
{{- if .Values.rbac.create }}
apiVersion: rbac.authorization.k8s.io/v1
{{- if eq .Values.rbac.scope "cluster" }}
kind: ClusterRole
{{- else }}
kind: Role
{{- end }}
metadata:
name: {{ template "loadtester.fullname" . }}
labels:
helm.sh/chart: {{ template "loadtester.chart" . }}
app.kubernetes.io/name: {{ template "loadtester.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
rules:
{{ toYaml .Values.rbac.rules | indent 2 }}
---
apiVersion: rbac.authorization.k8s.io/v1
{{- if eq .Values.rbac.scope "cluster" }}
kind: ClusterRoleBinding
{{- else }}
kind: RoleBinding
{{- end }}
metadata:
name: {{ template "loadtester.fullname" . }}
labels:
helm.sh/chart: {{ template "loadtester.chart" . }}
app.kubernetes.io/name: {{ template "loadtester.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
roleRef:
apiGroup: rbac.authorization.k8s.io
{{- if eq .Values.rbac.scope "cluster" }}
kind: ClusterRole
{{- else }}
kind: Role
{{- end }}
name: {{ template "loadtester.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ template "loadtester.fullname" . }}
namespace: {{ .Release.Namespace }}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ template "loadtester.fullname" . }}
labels:
helm.sh/chart: {{ template "loadtester.chart" . }}
app.kubernetes.io/name: {{ template "loadtester.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

View File

@@ -2,9 +2,13 @@ replicaCount: 1
image:
repository: weaveworks/flagger-loadtester
tag: 0.4.0
tag: 0.11.0
pullPolicy: IfNotPresent
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
logLevel: info
cmd:
timeout: 1h
@@ -27,6 +31,20 @@ tolerations: []
affinity: {}
rbac:
# rbac.create: `true` if rbac resources should be created
create: false
# rbac.scope: `cluster` to create cluster-scope rbac resources (ClusterRole/ClusterRoleBinding)
# otherwise, namespace-scope rbac resources will be created (Role/RoleBinding)
scope:
# rbac.rules: array of rules to apply to the role. example:
# rules:
# - apiGroups: [""]
# resources: ["pods"]
# verbs: ["list", "get"]
rules: []
# name of an existing service account to use - if not creating rbac resources
serviceAccountName: ""
# App Mesh virtual node settings

View File

@@ -1,10 +1,10 @@
apiVersion: v1
version: 2.1.0
appVersion: 1.4.0
version: 3.1.0
appVersion: 3.1.0
name: podinfo
engine: gotpl
description: Flagger canary deployment demo chart
home: https://github.com/weaveworks/flagger
home: https://flagger.app
maintainers:
- email: stefanprodan@users.noreply.github.com
name: stefanprodan

View File

@@ -26,6 +26,9 @@ spec:
hosts:
- {{ .Values.canary.istioIngress.host }}
{{- end }}
trafficPolicy:
tls:
mode: {{ .Values.canary.istioTLS }}
canaryAnalysis:
interval: {{ .Values.canary.analysis.interval }}
threshold: {{ .Values.canary.analysis.threshold }}

View File

@@ -21,10 +21,13 @@ spec:
app: {{ template "podinfo.fullname" . }}
annotations:
prometheus.io/scrape: 'true'
{{- if .Values.podAnnotations }}
{{ toYaml .Values.podAnnotations | indent 8 }}
{{- end }}
spec:
terminationGracePeriodSeconds: 30
containers:
- name: {{ .Chart.Name }}
- name: podinfo
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
@@ -34,6 +37,9 @@ spec:
- --random-delay={{ .Values.faults.delay }}
- --random-error={{ .Values.faults.error }}
- --config-path=/podinfo/config
{{- range .Values.backends }}
- --backend-url={{ . }}
{{- end }}
env:
{{- if .Values.message }}
- name: PODINFO_UI_MESSAGE

View File

@@ -0,0 +1,29 @@
apiVersion: v1
kind: Pod
metadata:
name: {{ template "podinfo.fullname" . }}-jwt-test-{{ randAlphaNum 5 | lower }}
labels:
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app: {{ template "podinfo.name" . }}
annotations:
"helm.sh/hook": test-success
sidecar.istio.io/inject: "false"
linkerd.io/inject: disabled
appmesh.k8s.aws/sidecarInjectorWebhook: disabled
spec:
containers:
- name: tools
image: giantswarm/tiny-tools
command:
- sh
- -c
- |
TOKEN=$(curl -sd 'test' ${PODINFO_SVC}/token | jq -r .token) &&
curl -H "Authorization: Bearer ${TOKEN}" ${PODINFO_SVC}/token/validate | grep test
env:
- name: PODINFO_SVC
value: {{ template "podinfo.fullname" . }}:{{ .Values.service.port }}
restartPolicy: Never

View File

@@ -1,22 +0,0 @@
{{- $url := printf "%s%s.%s:%v" (include "podinfo.fullname" .) (include "podinfo.suffix" .) .Release.Namespace .Values.service.port -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "podinfo.fullname" . }}-tests
labels:
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app: {{ template "podinfo.name" . }}
data:
run.sh: |-
@test "HTTP POST /echo" {
run curl --retry 3 --connect-timeout 2 -sSX POST -d 'test' {{ $url }}/echo
[ $output = "test" ]
}
@test "HTTP POST /store" {
curl --retry 3 --connect-timeout 2 -sSX POST -d 'test' {{ $url }}/store
}
@test "HTTP GET /" {
curl --retry 3 --connect-timeout 2 -sS {{ $url }} | grep hostname
}

View File

@@ -1,43 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: {{ template "podinfo.fullname" . }}-tests-{{ randAlphaNum 5 | lower }}
annotations:
"helm.sh/hook": test-success
sidecar.istio.io/inject: "false"
labels:
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app: {{ template "podinfo.name" . }}
spec:
initContainers:
- name: "test-framework"
image: "dduportal/bats:0.4.0"
command:
- "bash"
- "-c"
- |
set -ex
# copy bats to tools dir
cp -R /usr/local/libexec/ /tools/bats/
volumeMounts:
- mountPath: /tools
name: tools
containers:
- name: {{ .Release.Name }}-ui-test
image: dduportal/bats:0.4.0
command: ["/tools/bats/bats", "-t", "/tests/run.sh"]
volumeMounts:
- mountPath: /tests
name: tests
readOnly: true
- mountPath: /tools
name: tools
volumes:
- name: tests
configMap:
name: {{ template "podinfo.fullname" . }}-tests
- name: tools
emptyDir: {}
restartPolicy: Never

View File

@@ -1,22 +1,27 @@
# Default values for podinfo.
image:
repository: quay.io/stefanprodan/podinfo
tag: 1.4.0
repository: stefanprodan/podinfo
tag: 3.1.0
pullPolicy: IfNotPresent
podAnnotations: {}
service:
enabled: false
type: ClusterIP
port: 9898
hpa:
enabled: true
minReplicas: 2
maxReplicas: 2
maxReplicas: 4
cpu: 80
memory: 512Mi
canary:
enabled: true
enabled: false
# Istio traffic policy tls can be DISABLE or ISTIO_MUTUAL
istioTLS: DISABLE
istioIngress:
enabled: false
# Istio ingress gateway name
@@ -67,6 +72,7 @@ fullnameOverride: ""
logLevel: info
backend: #http://backend-podinfo:9898/echo
backends: []
message: #UI greetings
faults:

View File

@@ -1,11 +1,15 @@
package main
import (
"context"
"flag"
"fmt"
"log"
"os"
"strings"
"time"
"github.com/Masterminds/semver"
clientset "github.com/weaveworks/flagger/pkg/client/clientset/versioned"
informers "github.com/weaveworks/flagger/pkg/client/informers/externalversions"
"github.com/weaveworks/flagger/pkg/controller"
@@ -17,28 +21,38 @@ import (
"github.com/weaveworks/flagger/pkg/signals"
"github.com/weaveworks/flagger/pkg/version"
"go.uber.org/zap"
"k8s.io/apimachinery/pkg/util/uuid"
"k8s.io/client-go/kubernetes"
_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/tools/leaderelection"
"k8s.io/client-go/tools/leaderelection/resourcelock"
"k8s.io/client-go/transport"
_ "k8s.io/code-generator/cmd/client-gen/generators"
)
var (
masterURL string
kubeconfig string
metricsServer string
controlLoopInterval time.Duration
logLevel string
port string
slackURL string
slackUser string
slackChannel string
threadiness int
zapReplaceGlobals bool
zapEncoding string
namespace string
meshProvider string
selectorLabels string
masterURL string
kubeconfig string
metricsServer string
controlLoopInterval time.Duration
logLevel string
port string
msteamsURL string
slackURL string
slackUser string
slackChannel string
threadiness int
zapReplaceGlobals bool
zapEncoding string
namespace string
meshProvider string
selectorLabels string
ingressAnnotationsPrefix string
enableLeaderElection bool
leaderElectionNamespace string
ver bool
)
func init() {
@@ -51,17 +65,27 @@ func init() {
flag.StringVar(&slackURL, "slack-url", "", "Slack hook URL.")
flag.StringVar(&slackUser, "slack-user", "flagger", "Slack user name.")
flag.StringVar(&slackChannel, "slack-channel", "", "Slack channel.")
flag.StringVar(&msteamsURL, "msteams-url", "", "MS Teams incoming webhook URL.")
flag.IntVar(&threadiness, "threadiness", 2, "Worker concurrency.")
flag.BoolVar(&zapReplaceGlobals, "zap-replace-globals", false, "Whether to change the logging level of the global zap logger.")
flag.StringVar(&zapEncoding, "zap-encoding", "json", "Zap logger encoding.")
flag.StringVar(&namespace, "namespace", "", "Namespace that flagger would watch canary object.")
flag.StringVar(&meshProvider, "mesh-provider", "istio", "Service mesh provider, can be istio, appmesh, supergloo, nginx or smi.")
flag.StringVar(&meshProvider, "mesh-provider", "istio", "Service mesh provider, can be istio, linkerd, appmesh, supergloo, nginx or smi.")
flag.StringVar(&selectorLabels, "selector-labels", "app,name,app.kubernetes.io/name", "List of pod labels that Flagger uses to create pod selectors.")
flag.StringVar(&ingressAnnotationsPrefix, "ingress-annotations-prefix", "nginx.ingress.kubernetes.io", "Annotations prefix for ingresses.")
flag.BoolVar(&enableLeaderElection, "enable-leader-election", false, "Enable leader election.")
flag.StringVar(&leaderElectionNamespace, "leader-election-namespace", "kube-system", "Namespace used to create the leader election config map.")
flag.BoolVar(&ver, "version", false, "Print version")
}
func main() {
flag.Parse()
if ver {
fmt.Println("Flagger version", version.VERSION, "revision ", version.REVISION)
os.Exit(0)
}
logger, err := logger.NewLoggerWithEncoding(logLevel, zapEncoding)
if err != nil {
log.Fatalf("Error creating logger: %v", err)
@@ -105,6 +129,26 @@ func main() {
logger.Fatalf("Error calling Kubernetes API: %v", err)
}
k8sVersionConstraint := "^1.11.0"
// We append -alpha.1 to the end of our version constraint so that prebuilds of later versions
// are considered valid for our purposes, as well as some managed solutions like EKS where they provide
// a version like `v1.12.6-eks-d69f1b`. It doesn't matter what the prelease value is here, just that it
// exists in our constraint.
semverConstraint, err := semver.NewConstraint(k8sVersionConstraint + "-alpha.1")
if err != nil {
logger.Fatalf("Error parsing kubernetes version constraint: %v", err)
}
k8sSemver, err := semver.NewVersion(ver.GitVersion)
if err != nil {
logger.Fatalf("Error parsing kubernetes version as a semantic version: %v", err)
}
if !semverConstraint.Check(k8sSemver) {
logger.Fatalf("Unsupported version of kubernetes detected. Expected %s, got %v", k8sVersionConstraint, ver)
}
labels := strings.Split(selectorLabels, ",")
if len(labels) < 1 {
logger.Fatalf("At least one selector label is required")
@@ -127,20 +171,13 @@ func main() {
logger.Errorf("Metrics server %s unreachable %v", metricsServer, err)
}
var slack *notifier.Slack
if slackURL != "" {
slack, err = notifier.NewSlack(slackURL, slackUser, slackChannel)
if err != nil {
logger.Errorf("Notifier %v", err)
} else {
logger.Infof("Slack notifications enabled for channel %s", slack.Channel)
}
}
// setup Slack or MS Teams notifications
notifierClient := initNotifier(logger)
// start HTTP server
go server.ListenAndServe(port, 3*time.Second, logger, stopCh)
routerFactory := router.NewFactory(cfg, kubeClient, flaggerClient, logger, meshClient)
routerFactory := router.NewFactory(cfg, kubeClient, flaggerClient, ingressAnnotationsPrefix, logger, meshClient)
c := controller.NewController(
kubeClient,
@@ -148,9 +185,8 @@ func main() {
flaggerClient,
canaryInformer,
controlLoopInterval,
metricsServer,
logger,
slack,
notifierClient,
routerFactory,
observerFactory,
meshProvider,
@@ -169,12 +205,109 @@ func main() {
}
}
// start controller
go func(ctrl *controller.Controller) {
if err := ctrl.Run(threadiness, stopCh); err != nil {
// leader election context
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// prevents new requests when leadership is lost
cfg.Wrap(transport.ContextCanceller(ctx, fmt.Errorf("the leader is shutting down")))
// cancel leader election context on shutdown signals
go func() {
<-stopCh
cancel()
}()
// wrap controller run
runController := func() {
if err := c.Run(threadiness, stopCh); err != nil {
logger.Fatalf("Error running controller: %v", err)
}
}(c)
}
<-stopCh
// run controller when this instance wins the leader election
if enableLeaderElection {
ns := leaderElectionNamespace
if namespace != "" {
ns = namespace
}
startLeaderElection(ctx, runController, ns, kubeClient, logger)
} else {
runController()
}
}
func startLeaderElection(ctx context.Context, run func(), ns string, kubeClient kubernetes.Interface, logger *zap.SugaredLogger) {
configMapName := "flagger-leader-election"
id, err := os.Hostname()
if err != nil {
logger.Fatalf("Error running controller: %v", err)
}
id = id + "_" + string(uuid.NewUUID())
lock, err := resourcelock.New(
resourcelock.ConfigMapsResourceLock,
ns,
configMapName,
kubeClient.CoreV1(),
kubeClient.CoordinationV1(),
resourcelock.ResourceLockConfig{
Identity: id,
},
)
if err != nil {
logger.Fatalf("Error running controller: %v", err)
}
logger.Infof("Starting leader election id: %s configmap: %s namespace: %s", id, configMapName, ns)
leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
Lock: lock,
ReleaseOnCancel: true,
LeaseDuration: 60 * time.Second,
RenewDeadline: 15 * time.Second,
RetryPeriod: 5 * time.Second,
Callbacks: leaderelection.LeaderCallbacks{
OnStartedLeading: func(ctx context.Context) {
logger.Info("Acting as elected leader")
run()
},
OnStoppedLeading: func() {
logger.Infof("Leadership lost")
os.Exit(1)
},
OnNewLeader: func(identity string) {
if identity != id {
logger.Infof("Another instance has been elected as leader: %v", identity)
}
},
},
})
}
func initNotifier(logger *zap.SugaredLogger) (client notifier.Interface) {
provider := "slack"
notifierURL := fromEnv("SLACK_URL", slackURL)
if msteamsURL != "" || os.Getenv("MSTEAMS_URL") != "" {
provider = "msteams"
notifierURL = fromEnv("MSTEAMS_URL", msteamsURL)
}
notifierFactory := notifier.NewFactory(notifierURL, slackUser, slackChannel)
if notifierURL != "" {
var err error
client, err = notifierFactory.Notifier(provider)
if err != nil {
logger.Errorf("Notifier %v", err)
} else {
logger.Infof("Notifications enabled for %s", notifierURL[0:30])
}
}
return
}
func fromEnv(envVar string, defaultVal string) string {
if os.Getenv(envVar) != "" {
return os.Getenv(envVar)
}
return defaultVal
}

View File

@@ -10,7 +10,7 @@ import (
"time"
)
var VERSION = "0.4.0"
var VERSION = "0.11.0"
var (
logLevel string
port string
@@ -47,5 +47,7 @@ func main() {
go taskRunner.Start(100*time.Millisecond, stopCh)
logger.Infof("Starting load tester v%s API on port %s", VERSION, port)
loadtester.ListenAndServe(port, time.Minute, logger, taskRunner, stopCh)
gateStorage := loadtester.NewGateStorage("in-memory")
loadtester.ListenAndServe(port, time.Minute, logger, taskRunner, gateStorage, stopCh)
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 158 KiB

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 210 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 49 KiB

View File

@@ -5,13 +5,12 @@ description: Flagger is a progressive delivery Kubernetes operator
# Introduction
[Flagger](https://github.com/weaveworks/flagger) is a **Kubernetes** operator that automates the promotion of canary
deployments using **Istio**, **App Mesh** or **NGINX** routing for traffic shifting and **Prometheus** metrics for canary analysis.
The canary analysis can be extended with webhooks for running
system integration/acceptance tests, load tests, or any other custom validation.
deployments using **Istio**, **Linkerd**, **App Mesh**, **NGINX** or **Gloo** routing for traffic shifting and **Prometheus** metrics for canary analysis.
The canary analysis can be extended with webhooks for running system integration/acceptance tests, load tests, or any other custom validation.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance
indicators like HTTP requests success rate, requests average duration and pods health.
Based on analysis of the **KPIs** a canary is promoted or aborted, and the analysis result is published to **Slack**.
Based on analysis of the **KPIs** a canary is promoted or aborted, and the analysis result is published to **Slack** or **MS Teams**.
![Flagger overview diagram](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-overview.png)

View File

@@ -2,6 +2,7 @@
* [Introduction](README.md)
* [How it works](how-it-works.md)
* [FAQ](faq.md)
## Install
@@ -14,9 +15,11 @@
* [Istio Canary Deployments](usage/progressive-delivery.md)
* [Istio A/B Testing](usage/ab-testing.md)
* [Linkerd Canary Deployments](usage/linkerd-progressive-delivery.md)
* [App Mesh Canary Deployments](usage/appmesh-progressive-delivery.md)
* [NGINX Canary Deployments](usage/nginx-progressive-delivery.md)
* [Gloo Canary Deployments](usage/gloo-progressive-delivery.md)
* [Blue/Green Deployments](usage/blue-green.md)
* [Monitoring](usage/monitoring.md)
* [Alerting](usage/alerting.md)

View File

@@ -1,21 +1,641 @@
# Frequently asked questions
**Can Flagger be part of my integration tests?**
> Yes, Flagger supports webhooks to do integration testing.
### Deployment Strategies
**What if I only want to target beta testers?**
> That's a feature in Flagger, not in App Mesh. It's on the App Mesh roadmap.
**Which deployment strategies are supported by Flagger?**
**When do I use A/B testing when Canary?**
> One advantage of using A/B testing is that each version remains separated and routes aren't mixed.
>
> Using a Canary deployment can lead to behaviour like this one observed by a
> user:
>
> [..] during a canary deployment of our nodejs app, the version that is being served <50% traffic reports mime type mismatch errors in the browser (js as "text/html")
> When the deployment Passes/ Fails (doesn't really matter) the version that stays alive works as expected. If anyone has any tips or direction I would greatly appreciate it. Even if its as simple as I'm looking in the wrong place. Thanks in advance!
>
> The issue was that we were not maintaining session affinity while serving files for our frontend. Which resulted in any redirects or refreshes occasionally returning a mismatched app.*.js file (generated from vue)
>
> Read up on [A/B testing](https://docs.flagger.app/usage/ab-testing).
Flagger can run automated application analysis, promotion and rollback for the following deployment strategies:
* Canary (progressive traffic shifting)
* Istio, Linkerd, App Mesh, NGINX, Gloo
* Canary (traffic mirroring)
* Istio
* A/B Testing (HTTP headers and cookies traffic routing)
* Istio, App Mesh, NGINX
* Blue/Green (traffic switch)
* Kubernetes CNI, Istio, Linkerd, App Mesh, NGINX, Gloo
For Canary deployments and A/B testing you'll need a Layer 7 traffic management solution like a service mesh or an ingress controller.
For Blue/Green deployments no service mesh or ingress controller is required.
**When should I use A/B testing instead of progressive traffic shifting?**
For frontend applications that require session affinity you should use HTTP headers or cookies match conditions
to ensure a set of users will stay on the same version for the whole duration of the canary analysis.
A/B testing is supported by Istio and NGINX only.
Istio example:
```yaml
canaryAnalysis:
# schedule interval (default 60s)
interval: 1m
# total number of iterations
iterations: 10
# max number of failed iterations before rollback
threshold: 2
# canary match condition
match:
- headers:
x-canary:
regex: ".*insider.*"
- headers:
cookie:
regex: "^(.*?;)?(canary=always)(;.*)?$"
```
App Mesh example:
```yaml
canaryAnalysis:
interval: 1m
threshold: 10
iterations: 2
match:
- headers:
user-agent:
regex: ".*Chrome.*"
```
Note that App Mesh supports a single condition.
NGINX example:
```yaml
canaryAnalysis:
interval: 1m
threshold: 10
iterations: 2
match:
- headers:
x-canary:
exact: "insider"
- headers:
cookie:
exact: "canary"
```
Note that the NGINX ingress controller supports only exact matching for a single header and the cookie value is set to `always`.
The above configurations will route users with the x-canary header or canary cookie to the canary instance during analysis:
```bash
curl -H 'X-Canary: insider' http://app.example.com
curl -b 'canary=always' http://app.example.com
```
**Can I use Flagger to manage applications that live outside of a service mesh?**
For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments
with Kubernetes L4 networking.
Blue/Green example:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
spec:
provider: kubernetes
canaryAnalysis:
interval: 30s
threshold: 2
iterations: 10
metrics:
- name: request-success-rate
threshold: 99
interval: 1m
- name: request-duration
threshold: 500
interval: 30s
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"
```
The above configuration will run an analysis for five minutes.
Flagger starts the load test for the canary service (green version) and checks the Prometheus metrics every 30 seconds.
If the analysis result is positive, Flagger will promote the canary (green version) to primary (blue version).
**When can I use traffic mirroring?**
Traffic Mirroring is a pre-stage in a Canary (progressive traffic shifting) or
Blue/Green deployment strategy. Traffic mirroring will copy each incoming
request, sending one request to the primary and one to the canary service.
The response from the primary is sent back to the user. The response from the canary
is discarded. Metrics are collected on both requests so that the deployment will
only proceed if the canary metrics are healthy.
Mirroring is supported by Istio only.
In Istio, mirrored requests have `-shadow` appended to the `Host` (HTTP) or
`Authority` (HTTP/2) header; for example requests to `podinfo.test` that are
mirrored will be reported in telemetry with a destination host `podinfo.test-shadow`.
Mirroring must only be used for requests that are **idempotent** or capable of
being processed twice (once by the primary and once by the canary). Reads are
idempotent. Before using mirroring on requests that may be writes, you should
consider what will happen if a write is duplicated and handled by the primary
and canary.
To use mirroring, set `spec.canaryAnalysis.mirror` to `true`. Example for
traffic shifting:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
spec:
provider: istio
canaryAnalysis:
mirror: true
interval: 30s
stepWeight: 20
maxWeight: 50
```
### Kubernetes services
**How is an application exposed inside the cluster?**
Assuming the app name is podinfo you can define a canary like:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
service:
# ClusterIP port number (required)
port: 9898
# container port name or number
targetPort: http
# port name can be http or grpc (default http)
portName: http
```
Based on the canary spec service, Flagger generates the following Kubernetes ClusterIP service:
* `<targetRef.name>.<namespace>.svc.cluster.local`
selector `app=<name>-primary`
* `<targetRef.name>-primary.<namespace>.svc.cluster.local`
selector `app=<name>-primary`
* `<targetRef.name>-canary.<namespace>.svc.cluster.local`
selector `app=<name>`
This ensures that traffic coming from a namespace outside the mesh to `podinfo.test:9898`
will be routed to the latest stable release of your app.
```yaml
apiVersion: v1
kind: Service
metadata:
name: podinfo
spec:
type: ClusterIP
selector:
app: podinfo-primary
ports:
- name: http
port: 9898
protocol: TCP
targetPort: http
---
apiVersion: v1
kind: Service
metadata:
name: podinfo-primary
spec:
type: ClusterIP
selector:
app: podinfo-primary
ports:
- name: http
port: 9898
protocol: TCP
targetPort: http
---
apiVersion: v1
kind: Service
metadata:
name: podinfo-canary
spec:
type: ClusterIP
selector:
app: podinfo
ports:
- name: http
port: 9898
protocol: TCP
targetPort: http
```
The `podinfo-canary.test:9898` address is available only during the
canary analysis and can be used for conformance testing or load testing.
### Multiple ports
**My application listens on multiple ports, how can I expose them inside the cluster?**
If port discovery is enabled, Flagger scans the deployment spec and extracts the containers
ports excluding the port specified in the canary service and Envoy sidecar ports.
`These ports will be used when generating the ClusterIP services.
For a deployment that exposes two ports:
```yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9899"
spec:
containers:
- name: app
ports:
- containerPort: 8080
- containerPort: 9090
```
You can enable port discovery so that Prometheus will be able to reach port `9090` over mTLS:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
spec:
service:
# container port used for canary analysis
port: 8080
# port name can be http or grpc (default http)
portName: http
# add all the other container ports
# to the ClusterIP services (default false)
portDiscovery: true
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
```
Both port `8080` and `9090` will be added to the ClusterIP services.
### Label selectors
**What labels selectors are supported by Flagger?**
The target deployment must have a single label selector in the format `app: <DEPLOYMENT-NAME>`:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
spec:
selector:
matchLabels:
app: podinfo
template:
metadata:
labels:
app: podinfo
```
Besides `app` Flagger supports `name` and `app.kubernetes.io/name` selectors. If you use a different
convention you can specify your label with the `-selector-labels` flag.
**Is pod affinity and anti affinity supported?**
For pod affinity to work you need to use a different label than the `app`, `name` or `app.kubernetes.io/name`.
Anti affinity example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
spec:
selector:
matchLabels:
app: podinfo
affinity: podinfo
template:
metadata:
labels:
app: podinfo
affinity: podinfo
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
affinity: podinfo
topologyKey: kubernetes.io/hostname
```
### Istio routing
**How does Flagger interact with Istio?**
Flagger creates an Istio Virtual Service and Destination Rules based on the Canary service spec.
The service configuration lets you expose an app inside or outside the mesh.
You can also define traffic policies, HTTP match conditions, URI rewrite rules, CORS policies, timeout and retries.
The following spec exposes the `frontend` workload inside the mesh on `frontend.test.svc.cluster.local:9898`
and outside the mesh on `frontend.example.com`. You'll have to specify an Istio ingress gateway for external hosts.
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: frontend
namespace: test
spec:
service:
# container port
port: 9898
# service port name (optional, will default to "http")
portName: http-frontend
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
# Istio virtual service host names (optional)
hosts:
- frontend.example.com
# Istio traffic policy
trafficPolicy:
tls:
# use ISTIO_MUTUAL when mTLS is enabled
mode: DISABLE
# HTTP match conditions (optional)
match:
- uri:
prefix: /
# HTTP rewrite (optional)
rewrite:
uri: /
# Istio retry policy (optional)
retries:
attempts: 3
perTryTimeout: 1s
retryOn: "gateway-error,connect-failure,refused-stream"
# Add headers (optional)
headers:
request:
add:
x-some-header: "value"
# cross-origin resource sharing policy (optional)
corsPolicy:
allowOrigin:
- example.com
allowMethods:
- GET
allowCredentials: false
allowHeaders:
- x-some-header
maxAge: 24h
```
For the above spec Flagger will generate the following virtual service:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: frontend
namespace: test
ownerReferences:
- apiVersion: flagger.app/v1alpha3
blockOwnerDeletion: true
controller: true
kind: Canary
name: podinfo
uid: 3a4a40dd-3875-11e9-8e1d-42010a9c0fd1
spec:
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
hosts:
- frontend.example.com
- frontend
http:
- appendHeaders:
x-some-header: "value"
corsPolicy:
allowHeaders:
- x-some-header
allowMethods:
- GET
allowOrigin:
- example.com
maxAge: 24h
match:
- uri:
prefix: /
rewrite:
uri: /
route:
- destination:
host: podinfo-primary
weight: 100
- destination:
host: podinfo-canary
weight: 0
retries:
attempts: 3
perTryTimeout: 1s
retryOn: "gateway-error,connect-failure,refused-stream"
```
For each destination in the virtual service a rule is generated:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: frontend-primary
namespace: test
spec:
host: frontend-primary
trafficPolicy:
tls:
mode: DISABLE
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: frontend-canary
namespace: test
spec:
host: frontend-canary
trafficPolicy:
tls:
mode: DISABLE
```
Flagger keeps in sync the virtual service and destination rules with the canary service spec.
Any direct modification to the virtual service spec will be overwritten.
To expose a workload inside the mesh on `http://backend.test.svc.cluster.local:9898`,
the service spec can contain only the container port and the traffic policy:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: backend
namespace: test
spec:
service:
port: 9898
trafficPolicy:
tls:
mode: DISABLE
```
Based on the above spec, Flagger will create several ClusterIP services like:
```yaml
apiVersion: v1
kind: Service
metadata:
name: backend-primary
ownerReferences:
- apiVersion: flagger.app/v1alpha3
blockOwnerDeletion: true
controller: true
kind: Canary
name: backend
uid: 2ca1a9c7-2ef6-11e9-bd01-42010a9c0145
spec:
type: ClusterIP
ports:
- name: http
port: 9898
protocol: TCP
targetPort: 9898
selector:
app: backend-primary
```
Flagger works for user facing apps exposed outside the cluster via an ingress gateway
and for backend HTTP APIs that are accessible only from inside the mesh.
### Istio Ingress Gateway
**How can I expose multiple canaries on the same external domain?**
Assuming you have two apps, one that servers the main website and one that serves the REST API.
For each app you can define a canary object as:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: website
spec:
service:
port: 8080
gateways:
- public-gateway.istio-system.svc.cluster.local
hosts:
- my-site.com
match:
- uri:
prefix: /
rewrite:
uri: /
---
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: webapi
spec:
service:
port: 8080
gateways:
- public-gateway.istio-system.svc.cluster.local
hosts:
- my-site.com
match:
- uri:
prefix: /api
rewrite:
uri: /
```
Based on the above configuration, Flagger will create two virtual services bounded to the same ingress gateway and external host.
Istio Pilot will [merge](https://istio.io/help/ops/traffic-management/deploy-guidelines/#multiple-virtual-services-and-destination-rules-for-the-same-host)
the two services and the website rule will be moved to the end of the list in the merged configuration.
Note that host merging only works if the canaries are bounded to a ingress gateway other than the `mesh` gateway.
### Istio Mutual TLS
**How can I enable mTLS for a canary?**
When deploying Istio with global mTLS enabled, you have to set the TLS mode to `ISTIO_MUTUAL`:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
spec:
service:
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
```
If you run Istio in permissive mode you can disable TLS:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
spec:
service:
trafficPolicy:
tls:
mode: DISABLE
```
**If Flagger is outside of the mesh, how can it start the load test?**
In order for Flagger to be able to call the load tester service from outside the mesh, you need to disable mTLS on port 80:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: flagger-loadtester
namespace: test
spec:
host: "flagger-loadtester.test.svc.cluster.local"
trafficPolicy:
tls:
mode: DISABLE
---
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: flagger-loadtester
namespace: test
spec:
targets:
- name: flagger-loadtester
ports:
- number: 80
```

View File

@@ -2,9 +2,7 @@
[Flagger](https://github.com/weaveworks/flagger) takes a Kubernetes deployment and optionally
a horizontal pod autoscaler \(HPA\) and creates a series of objects
\(Kubernetes deployments, ClusterIP services and Istio or App Mesh virtual services\) to drive the canary analysis and promotion.
![Flagger Canary Process](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-hpa.png)
\(Kubernetes deployments, ClusterIP services, virtual service, traffic split or ingress\) to drive the canary analysis and promotion.
### Canary Custom Resource
@@ -17,6 +15,9 @@ metadata:
name: podinfo
namespace: test
spec:
# service mesh provider (optional)
# can be: kubernetes, istio, linkerd, appmesh, nginx, gloo, supergloo
provider: linkerd
# deployment reference
targetRef:
apiVersion: apps/v1
@@ -31,16 +32,15 @@ spec:
kind: HorizontalPodAutoscaler
name: podinfo
service:
# container port
# ClusterIP port number
port: 9898
# service port name (optional, will default to "http")
portName: http-podinfo
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
# Istio virtual service host names (optional)
hosts:
- podinfo.example.com
# ClusterIP port name can be http or grpc (default http)
portName: http
# container port number or name (optional)
targetPort: 9898
# add all the other container ports
# to the ClusterIP services (default false)
portDiscovery: false
# promote the canary without analysing it (default false)
skipAnalysis: false
# define the canary analysis timing and KPIs
@@ -67,15 +67,13 @@ spec:
# milliseconds
threshold: 500
interval: 30s
# external checks (optional)
# testing (optional)
webhooks:
- name: integration-tests
url: http://podinfo.test:9898/echo
timeout: 1m
# key-value pairs (optional)
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
test: "all"
token: "16688eb5e9f289f1991c"
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"
```
**Note** that the target deployment must have a single label selector in the format `app: <DEPLOYMENT-NAME>`:
@@ -98,187 +96,77 @@ spec:
Besides `app` Flagger supports `name` and `app.kubernetes.io/name` selectors. If you use a different
convention you can specify your label with the `-selector-labels` flag.
The target deployment should expose a TCP port that will be used by Flagger to create the ClusterIP Service and
the Istio Virtual Service. The container port from the target deployment should match the `service.port` value.
The target deployment should expose a TCP port that will be used by Flagger to create the ClusterIP Services.
The container port from the target deployment should match the `service.port` or `service.targetPort`.
### Istio routing
### Canary status
Flagger creates an Istio Virtual Service and Destination Rules based on the Canary service spec.
The service configuration lets you expose an app inside or outside the mesh.
You can also define traffic policies, HTTP match conditions, URI rewrite rules, CORS policies, timeout and retries.
Get the current status of canary deployments cluster wide:
The following spec exposes the `frontend` workload inside the mesh on `frontend.test.svc.cluster.local:9898`
and outside the mesh on `frontend.example.com`. You'll have to specify an Istio ingress gateway for external hosts.
```bash
kubectl get canaries --all-namespaces
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: frontend
namespace: test
spec:
service:
# container port
port: 9898
# service port name (optional, will default to "http")
portName: http-frontend
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
# Istio virtual service host names (optional)
hosts:
- frontend.example.com
# Istio traffic policy (optional)
trafficPolicy:
loadBalancer:
simple: LEAST_CONN
# HTTP match conditions (optional)
match:
- uri:
prefix: /
# HTTP rewrite (optional)
rewrite:
uri: /
# Envoy timeout and retry policy (optional)
headers:
request:
add:
x-envoy-upstream-rq-timeout-ms: "15000"
x-envoy-max-retries: "10"
x-envoy-retry-on: "gateway-error,connect-failure,refused-stream"
# cross-origin resource sharing policy (optional)
corsPolicy:
allowOrigin:
- example.com
allowMethods:
- GET
allowCredentials: false
allowHeaders:
- x-some-header
maxAge: 24h
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-06-30T14:05:07Z
prod frontend Succeeded 0 2019-06-30T16:15:07Z
prod backend Failed 0 2019-06-30T17:05:07Z
```
For the above spec Flagger will generate the following virtual service:
The status condition reflects the last know state of the canary analysis:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: frontend
namespace: test
ownerReferences:
- apiVersion: flagger.app/v1alpha3
blockOwnerDeletion: true
controller: true
kind: Canary
name: podinfo
uid: 3a4a40dd-3875-11e9-8e1d-42010a9c0fd1
spec:
gateways:
- public-gateway.istio-system.svc.cluster.local
- mesh
hosts:
- frontend.example.com
- frontend
http:
- appendHeaders:
x-envoy-max-retries: "10"
x-envoy-retry-on: gateway-error,connect-failure,refused-stream
x-envoy-upstream-rq-timeout-ms: "15000"
corsPolicy:
allowHeaders:
- x-some-header
allowMethods:
- GET
allowOrigin:
- example.com
maxAge: 24h
match:
- uri:
prefix: /
rewrite:
uri: /
route:
- destination:
host: podinfo-primary
weight: 100
- destination:
host: podinfo-canary
weight: 0
```bash
kubectl -n test get canary/podinfo -oyaml | awk '/status/,0'
```
For each destination in the virtual service a rule is generated:
A successful rollout status:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: frontend-primary
namespace: test
spec:
host: frontend-primary
trafficPolicy:
loadBalancer:
simple: LEAST_CONN
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: frontend-canary
namespace: test
spec:
host: frontend-canary
trafficPolicy:
loadBalancer:
simple: LEAST_CONN
status:
canaryWeight: 0
failedChecks: 0
iterations: 0
lastAppliedSpec: "14788816656920327485"
lastPromotedSpec: "14788816656920327485"
conditions:
- lastTransitionTime: "2019-07-10T08:23:18Z"
lastUpdateTime: "2019-07-10T08:23:18Z"
message: Canary analysis completed successfully, promotion finished.
reason: Succeeded
status: "True"
type: Promoted
```
Flagger keeps in sync the virtual service and destination rules with the canary service spec.
Any direct modification to the virtual service spec will be overwritten.
The `Promoted` status condition can have one of the following reasons:
Initialized, Waiting, Progressing, Promoting, Finalising, Succeeded or Failed.
A failed canary will have the promoted status set to `false`,
the reason to `failed` and the last applied spec will be different to the last promoted one.
To expose a workload inside the mesh on `http://backend.test.svc.cluster.local:9898`,
the service spec can contain only the container port:
Wait for a successful rollout:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: backend
namespace: test
spec:
service:
port: 9898
```bash
kubectl wait canary/podinfo --for=condition=promoted
```
Based on the above spec, Flagger will create several ClusterIP services like:
CI example:
```yaml
apiVersion: v1
kind: Service
metadata:
name: backend-primary
ownerReferences:
- apiVersion: flagger.app/v1alpha3
blockOwnerDeletion: true
controller: true
kind: Canary
name: backend
uid: 2ca1a9c7-2ef6-11e9-bd01-42010a9c0145
spec:
type: ClusterIP
ports:
- name: http
port: 9898
protocol: TCP
targetPort: 9898
selector:
app: backend-primary
```bash
# update the container image
kubectl set image deployment/podinfo podinfod=stefanprodan/podinfo:3.0.1
# wait for Flagger to detect the change
ok=false
until ${ok}; do
kubectl get canary/podinfo | grep 'Progressing' && ok=true || ok=false
sleep 5
done
# wait for the canary analysis to finish
kubectl wait canary/podinfo --for=condition=promoted --timeout=5m
# check if the deployment was successful
kubectl get canary/podinfo | grep Succeeded
```
Flagger works for user facing apps exposed outside the cluster via an ingress gateway
and for backend HTTP APIs that are accessible only from inside the mesh.
### Canary Stages
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
@@ -292,12 +180,13 @@ A canary deployment is triggered by changes in any of the following objects:
Gated canary promotion stages:
* scan for canary deployments
* check Istio virtual service routes are mapped to primary and canary ClusterIP services
* check primary and canary deployments status
* check primary and canary deployment status
* halt advancement if a rolling update is underway
* halt advancement if pods are unhealthy
* call pre-rollout webhooks are check results
* halt advancement if any hook returned a non HTTP 2xx result
* call confirm-rollout webhooks and check results
* halt advancement if any hook returns a non HTTP 2xx result
* call pre-rollout webhooks and check results
* halt advancement if any hook returns a non HTTP 2xx result
* increment the failed checks counter
* increase canary traffic weight percentage from 0% to 5% (step weight)
* call rollout webhooks and check results
@@ -314,8 +203,11 @@ Gated canary promotion stages:
* halt advancement if any webhook call fails
* halt advancement while canary request success rate is under the threshold
* halt advancement while canary request duration P99 is over the threshold
* halt advancement while any custom metric check fails
* halt advancement if the primary or canary deployment becomes unhealthy
* halt advancement while canary deployment is being scaled up/down by HPA
* call confirm-promotion webhooks and check results
* halt advancement if any hook returns a non HTTP 2xx result
* promote canary to primary
* copy ConfigMaps and Secrets from canary to primary
* copy canary deployment spec template over primary
@@ -325,7 +217,7 @@ Gated canary promotion stages:
* scale to zero the canary deployment
* mark rollout as finished
* call post-rollout webhooks
* post the analysis result to Slack
* post the analysis result to Slack or MS Teams
* wait for the canary deployment to be updated and start over
### Canary Analysis
@@ -412,6 +304,45 @@ interval * threshold
Make sure that the analysis threshold is lower than the number of iterations.
### Blue/Green deployments
For applications that are not deployed on a service mesh, Flagger can orchestrate blue/green style deployments
with Kubernetes L4 networking. When using Istio you have the option to mirror traffic between blue and green.
You can use the blue/green deployment strategy by replacing `stepWeight/maxWeight` with `iterations` in the `canaryAnalysis` spec:
```yaml
canaryAnalysis:
# schedule interval (default 60s)
interval: 1m
# total number of iterations
iterations: 10
# max number of failed iterations before rollback
threshold: 2
# Traffic shadowing (compatible with Istio only)
mirror: true
```
With the above configuration Flagger will run conformance and load tests on the canary pods for ten minutes.
If the metrics analysis succeeds, live traffic will be switched from the old version to the new one when the
canary is promoted.
The blue/green deployment strategy is supported for all service mesh providers.
Blue/Green rollout steps for service mesh:
* scale up the canary (green)
* run conformance tests for the canary pods
* run load tests and metric checks for the canary pods
* route traffic to canary
* promote canary spec over primary (blue)
* wait for primary rollout
* route traffic to primary
* scale down canary
After the analysis finishes, the traffic is routed to the canary (green) before triggering the primary (blue)
rolling update, this ensures a smooth transition to the new version avoiding dropping in-flight requests during
the Kubernetes deployment rollout.
### HTTP Metrics
The canary analysis is using the following Prometheus queries:
@@ -455,7 +386,7 @@ sum(
)
```
App Mesh query:
Envoy query (App Mesh or Gloo):
```javascript
sum(
@@ -463,7 +394,7 @@ sum(
envoy_cluster_upstream_rq{
kubernetes_namespace="$namespace",
kubernetes_pod_name=~"$workload",
response_code!~"5.*"
envoy_response_code!~"5.*"
}[$interval]
)
)
@@ -508,7 +439,7 @@ histogram_quantile(0.99,
)
```
App Mesh query:
Envoy query (App Mesh or Gloo):
```javascript
histogram_quantile(0.99,
@@ -604,11 +535,16 @@ The canary analysis can be extended with webhooks. Flagger will call each webhoo
determine from the response status code (HTTP 2xx) if the canary is failing or not.
There are three types of hooks:
* Confirm-rollout hooks are executed before scaling up the canary deployment and can be used for manual approval.
The rollout is paused until the hook returns a successful HTTP status code.
* Pre-rollout hooks are executed before routing traffic to canary.
The canary advancement is paused if a pre-rollout hook fails and if the number of failures reach the
threshold the canary will be rollback.
* Rollout hooks are executed during the analysis on each iteration before the metric checks.
If a rollout hook call fails the canary advancement is paused and eventfully rolled back.
* Confirm-promotion hooks are executed before the promotion step.
The canary promotion is paused until the hooks return HTTP 200.
While the promotion is paused, Flagger will continue to run the metrics checks and rollout hooks.
* Post-rollout hooks are executed after the canary has been promoted or rolled back.
If a post rollout hook fails the error is logged.
@@ -617,6 +553,9 @@ Spec:
```yaml
canaryAnalysis:
webhooks:
- name: "start gate"
type: confirm-rollout
url: http://flagger-loadtester.test/gate/approve
- name: "smoke test"
type: pre-rollout
url: http://flagger-helmtester.kube-system/
@@ -630,6 +569,9 @@ Spec:
timeout: 15s
metadata:
cmd: "hey -z 1m -q 5 -c 2 http://podinfo-canary.test:9898/"
- name: "promotion gate"
type: confirm-promotion
url: http://flagger-loadtester.test/gate/approve
- name: "notify"
type: post-rollout
url: http://telegram.bot:8080/
@@ -673,7 +615,7 @@ that generates traffic during analysis when configured as a webhook.
![Flagger Load Testing Webhook](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-load-testing.png)
First you need to deploy the load test runner in a namespace with Istio sidecar injection enabled:
First you need to deploy the load test runner in a namespace with sidecar injection enabled:
```bash
export REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
@@ -716,7 +658,7 @@ When the canary analysis starts, Flagger will call the webhooks and the load tes
in the background, if they are not already running. This will ensure that during the
analysis, the `podinfo-canary.test` service will receive a steady stream of GET and POST requests.
If your workload is exposed outside the mesh with the Istio Gateway and TLS you can point `hey` to the
If your workload is exposed outside the mesh you can point `hey` to the
public URL and use HTTP2.
```yaml
@@ -729,6 +671,30 @@ webhooks:
cmd: "hey -z 1m -q 10 -c 2 -h2 https://podinfo.example.com/"
```
For gRPC services you can use [bojand/ghz](https://github.com/bojand/ghz) which is a similar tool to Hey but for gPRC:
```yaml
webhooks:
- name: grpc-load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "ghz -z 1m -q 10 -c 2 --insecure podinfo.test:9898"
```
`ghz` uses reflection to identify which gRPC method to call. If you do not wish to enable reflection for your gRPC service you can implement a standardized health check from the [grpc-proto](https://github.com/grpc/grpc-proto) library. To use this [health check schema](https://github.com/grpc/grpc-proto/blob/master/grpc/health/v1/health.proto) without reflection you can pass a parameter to `ghz` like this
```yaml
webhooks:
- name: grpc-load-test-no-reflection
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "ghz --insecure --proto=/tmp/ghz/health.proto --call=grpc.health.v1.Health/Check podinfo.test:9898"
```
The load tester can run arbitrary commands as long as the binary is present in the container image.
For example if you you want to replace `hey` with another CLI, you can create your own Docker image:
@@ -801,6 +767,20 @@ Now you can add pre-rollout webhooks to the canary analysis spec:
When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.
If the helm test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.
If you are using Helm v3, you'll have to create a dedicated service account and add the release namespace to the test command:
```yaml
canaryAnalysis:
webhooks:
- name: "smoke test"
type: pre-rollout
url: http://flagger-helmtester.kube-system/
timeout: 3m
metadata:
type: "helmv3"
cmd: "test run {{ .Release.Name }} --cleanup -n {{ .Release.Namespace }}"
```
As an alternative to Helm you can use the [Bash Automated Testing System](https://github.com/bats-core/bats-core) to run your tests.
```yaml
@@ -816,3 +796,79 @@ As an alternative to Helm you can use the [Bash Automated Testing System](https:
```
Note that you should create a ConfigMap with your Bats tests and mount it inside the tester container.
### Manual Gating
For manual approval of a canary deployment you can use the `confirm-rollout` and `confirm-promotion` webhooks.
The confirmation rollout hooks are executed before the pre-rollout hooks.
Flagger will halt the canary traffic shifting and analysis until the confirm webhook returns HTTP status 200.
Manual gating with Flagger's tester:
```yaml
canaryAnalysis:
webhooks:
- name: "gate"
type: confirm-rollout
url: http://flagger-loadtester.test/gate/halt
```
The `/gate/halt` returns HTTP 403 thus blocking the rollout.
If you have notifications enabled, Flagger will post a message to Slack or MS Teams if a canary rollout is waiting for approval.
Change the URL to `/gate/approve` to start the canary analysis:
```yaml
canaryAnalysis:
webhooks:
- name: "gate"
type: confirm-rollout
url: http://flagger-loadtester.test/gate/approve
```
Manual gating can be driven with Flagger's tester API. Set the confirmation URL to `/gate/check`:
```yaml
canaryAnalysis:
webhooks:
- name: "ask for confirmation"
type: confirm-rollout
url: http://flagger-loadtester.test/gate/check
```
By default the gate is closed, you can start or resume the canary rollout with:
```bash
kubectl -n test exec -it flagger-loadtester-xxxx-xxxx sh
curl -d '{"name": "podinfo","namespace":"test"}' http://localhost:8080/gate/open
```
You can pause the rollout at any time with:
```bash
curl -d '{"name": "podinfo","namespace":"test"}' http://localhost:8080/gate/close
```
If a canary analysis is paused the status will change to waiting:
```bash
kubectl get canary/podinfo
NAME STATUS WEIGHT
podinfo Waiting 0
```
The `confirm-promotion` hook type can be used to manually approve the canary promotion.
While the promotion is paused, Flagger will continue to run the metrics checks and load tests.
```yaml
canaryAnalysis:
webhooks:
- name: "promotion gate"
type: confirm-promotion
url: http://flagger-loadtester.test/gate/halt
```
If you have notifications enabled, Flagger will post a message to Slack or MS Teams if a canary promotion is waiting for approval.

View File

@@ -14,19 +14,10 @@ The App Mesh integration with EKS is made out of the following components:
* Admission controller - injects the Envoy sidecar and assigns Kubernetes pods to App Mesh virtual nodes
* Metrics server - Prometheus instance that collects and stores Envoy's metrics
Prerequisites:
* jq
* homebrew
* openssl
* kubectl
* AWS CLI (default region us-west-2)
### Create a Kubernetes cluster
In order to create an EKS cluster you can use [eksctl](https://eksctl.io).
Eksctl is an open source command-line utility made by Weaveworks in collaboration with Amazon,
its a Kubernetes-native tool written in Go.
Eksctl is an open source command-line utility made by Weaveworks in collaboration with Amazon.
On MacOS you can install eksctl with Homebrew:
@@ -40,6 +31,8 @@ Create an EKS cluster:
```bash
eksctl create cluster --name=appmesh \
--region=us-west-2 \
--nodes 3 \
--node-volume-size=120 \
--appmesh-access
```
@@ -98,21 +91,39 @@ kubectl -n kube-system top pods
### Install the App Mesh components
Run the App Mesh installer:
Create the `appmesh-system` namespace:
```bash
curl -fsSL https://git.io/get-app-mesh-eks.sh | bash -
```sh
kubectl create ns appmesh-system
```
The installer does the following:
Apply the App Mesh CRDs:
* creates the `appmesh-system` namespace
* generates a certificate signed by Kubernetes CA
* registers the App Mesh mutating webhook
* deploys the App Mesh webhook in `appmesh-system` namespace
* deploys the App Mesh CRDs
* deploys the App Mesh controller in `appmesh-system` namespace
* creates a mesh called `global`
```sh
kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/appmesh-controller/crds/crds.yaml
```
Add the EKS repository to Helm:
```sh
helm repo add eks https://aws.github.io/eks-charts
```
Install the App Mesh CRD controller:
```sh
helm upgrade -i appmesh-controller eks/appmesh-controller \
--wait --namespace appmesh-system --version 0.2.0
```
Install the App Mesh admission controller:
```sh
helm upgrade -i appmesh-inject eks/appmesh-inject \
--wait --namespace appmesh-system \
--set mesh.create=true \
--set mesh.name=global
```
Verify that the global mesh is active:
@@ -125,6 +136,16 @@ Status:
Type: MeshActive
```
In order to collect the App Mesh metrics that Flagger needs to run the canary analysis,
you'll need to setup a Prometheus instance to scrape the Envoy sidecars.
Install the App Mesh Prometheus:
```sh
helm upgrade -i appmesh-prometheus eks/appmesh-prometheus \
--wait --namespace appmesh-system
```
### Install Flagger and Grafana
Add Flagger Helm repository:
@@ -133,25 +154,28 @@ Add Flagger Helm repository:
helm repo add flagger https://flagger.app
```
Deploy Flagger and Prometheus in the _**appmesh-system**_ namespace:
Install Flagger's Canary CRD:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=appmesh-system \
--set meshProvider=appmesh \
--set prometheus.install=true
```yaml
kubectl apply -f https://raw.githubusercontent.com/weaveworks/flagger/master/artifacts/flagger/crd.yaml
```
In order to collect the App Mesh metrics that Flagger needs to run the canary analysis,
you'll need to setup a Prometheus instance to scrape the Envoy sidecars.
You can enable **Slack** notifications with:
Deploy Flagger in the _**appmesh-system**_ namespace:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=appmesh-system \
--set crd.create=false \
--set meshProvider=appmesh \
--set metricsServer=http://prometheus.appmesh:9090 \
--set metricsServer=http://appmesh-prometheus:9090
```
You can enable Slack or MS Teams notifications with:
```bash
helm upgrade -i flagger flagger/flagger \
--reuse-values \
--namespace=appmesh-system \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \
--set slack.user=flagger
@@ -163,7 +187,7 @@ Deploy Grafana in the _**appmesh-system**_ namespace:
```bash
helm upgrade -i flagger-grafana flagger/grafana \
--namespace=appmesh-system \
--set url=http://flagger-prometheus.appmesh-system:9090
--set url=http://appmesh-prometheus:9090
```
You can access Grafana using port forwarding:

View File

@@ -186,7 +186,7 @@ Install cert-manager's CRDs:
```bash
CERT_REPO=https://raw.githubusercontent.com/jetstack/cert-manager
kubectl apply -f ${CERT_REPO}/release-0.7/deploy/manifests/00-crds.yaml
kubectl apply -f ${CERT_REPO}/release-0.10/deploy/manifests/00-crds.yaml
```
Create the cert-manager namespace and disable resource validation:
@@ -204,7 +204,7 @@ helm repo add jetstack https://charts.jetstack.io && \
helm repo update && \
helm upgrade -i cert-manager \
--namespace cert-manager \
--version v0.7.0 \
--version v0.10.0 \
jetstack/cert-manager
```
@@ -333,10 +333,17 @@ The GKE Istio add-on does not include a Prometheus instance that scrapes the Ist
Because Flagger uses the Istio HTTP metrics to run the canary analysis you have to deploy the following
Prometheus configuration that's similar to the one that comes with the official Istio Helm chart.
```bash
REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
Find the GKE Istio version with:
kubectl apply -f ${REPO}/artifacts/gke/istio-prometheus.yaml
```bash
kubectl -n istio-system get deploy istio-pilot -oyaml | grep image:
```
Install Prometheus in istio-system namespace:
```bash
kubectl -n istio-system apply -f \
https://storage.googleapis.com/gke-release/istio/release/1.0.6-gke.3/patches/install-prometheus.yaml
```
### Install Flagger and Grafana
@@ -347,11 +354,18 @@ Add Flagger Helm repository:
helm repo add flagger https://flagger.app
```
Install Flagger's Canary CRD:
```yaml
kubectl apply -f https://raw.githubusercontent.com/weaveworks/flagger/master/artifacts/flagger/crd.yaml
```
Deploy Flagger in the `istio-system` namespace with Slack notifications enabled:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set metricsServer=http://prometheus.istio-system:9090 \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \

View File

@@ -1,6 +1,6 @@
# Flagger install on Kubernetes
This guide walks you through setting up Flagger on a Kubernetes cluster.
This guide walks you through setting up Flagger on a Kubernetes cluster with Helm or Kustomize.
### Prerequisites
@@ -9,18 +9,7 @@ Flagger requires a Kubernetes cluster **v1.11** or newer with the following admi
* MutatingAdmissionWebhook
* ValidatingAdmissionWebhook
Flagger depends on [Istio](https://istio.io/docs/setup/kubernetes/quick-start/) **v1.0.3** or newer
with traffic management, telemetry and Prometheus enabled.
A minimal Istio installation should contain the following services:
* istio-pilot
* istio-ingressgateway
* istio-sidecar-injector
* istio-telemetry
* prometheus
### Install Flagger
### Install Flagger with Helm
Add Flagger Helm repository:
@@ -28,26 +17,54 @@ Add Flagger Helm repository:
helm repo add flagger https://flagger.app
```
Deploy Flagger in the _**istio-system**_ namespace:
Install Flagger's Canary CRD:
```yaml
kubectl apply -f https://raw.githubusercontent.com/weaveworks/flagger/master/artifacts/flagger/crd.yaml
```
Deploy Flagger for Istio:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set metricsServer=http://prometheus.istio-system:9090
--set crd.create=false \
--set meshProvider=istio \
--set metricsServer=http://prometheus:9090
```
You can install Flagger in any namespace as long as it can talk to the Istio Prometheus service on port 9090.
Deploy Flagger for Linkerd:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=linkerd \
--set crd.create=false \
--set meshProvider=linkerd \
--set metricsServer=http://linkerd-prometheus:9090
```
You can install Flagger in any namespace as long as it can talk to the Prometheus service on port 9090.
Enable **Slack** notifications:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \
--set slack.user=flagger
```
Enable **Microsoft Teams** notifications:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set msteams.url=https://outlook.office.com/webhook/YOUR/TEAMS/WEBHOOK
```
If you don't have Tiller you can use the helm template command and apply the generated yaml with kubectl:
```bash
@@ -81,7 +98,7 @@ If you want to remove all the objects created by Flagger you have delete the Can
kubectl delete crd canaries.flagger.app
```
### Install Grafana
### Install Grafana with Helm
Flagger comes with a Grafana dashboard made for monitoring the canary analysis.
@@ -115,31 +132,117 @@ You can access Grafana using port forwarding:
kubectl -n istio-system port-forward svc/flagger-grafana 3000:80
```
### Install Load Tester
### Install Flagger with Kustomize
Flagger comes with an optional load testing service that generates traffic
during canary analysis when configured as a webhook.
As an alternative to Helm, Flagger can be installed with Kustomize.
Deploy the load test runner with Helm:
**Service mesh specific installers**
Install Flagger for Istio:
```bash
helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test \
--set cmd.timeout=1h
kubectl apply -k github.com/weaveworks/flagger//kustomize/istio
```
Deploy with kubectl:
This deploys Flagger in the `istio-system` namespace and sets the metrics server URL to Istio's Prometheus instance.
Note that you'll need kubectl 1.14 to run the above the command or you can download the
[kustomize binary](https://github.com/kubernetes-sigs/kustomize/releases) and run:
```bash
helm fetch --untar --untardir . flagger/loadtester &&
helm template loadtester \
--name flagger-loadtester \
--namespace=test
> $HOME/flagger-loadtester.yaml
# apply
kubectl apply -f $HOME/flagger-loadtester.yaml
kustomize build github.com/weaveworks/flagger//kustomize/istio | kubectl apply -f -
```
> **Note** that the load tester should be deployed in a namespace with Istio sidecar injection enabled.
Install Flagger for AWS App Mesh:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/appmesh
```
This deploys Flagger and Prometheus (configured to scrape the App Mesh Envoy sidecars) in the `appmesh-system` namespace.
Install Flagger for Linkerd:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/linkerd
```
This deploys Flagger in the `linkerd` namespace and sets the metrics server URL to Linkerd's Prometheus instance.
If you want to install a specific Flagger release, add the version number to the URL:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/linkerd?ref=0.18.0
```
**Generic installer**
Install Flagger and Prometheus:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/kubernetes
```
This deploys Flagger and Prometheus in the `flagger-system` namespace,
sets the metrics server URL to `http://flagger-prometheus.flagger-system:9090` and the mesh provider to `kubernetes`.
The Prometheus instance has a two hours data retention and is configured to scrape all pods in your cluster that
have the `prometheus.io/scrape: "true"` annotation.
To target a different provider you can specify it in the canary custom resource:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: app
namespace: test
spec:
# can be: kubernetes, istio, linkerd, appmesh, nginx, gloo
# use the kubernetes provider for Blue/Green style deployments
provider: nginx
```
**Customized installer**
Create a kustomization file using flagger as base:
```bash
cat > kustomization.yaml <<EOF
namespace: istio-system
bases:
- github.com/weaveworks/flagger/kustomize/base/flagger
patchesStrategicMerge:
- patch.yaml
EOF
```
Create a patch and enable Slack notifications by setting the slack channel and hook URL:
```bash
cat > patch.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
spec:
template:
spec:
containers:
- name: flagger
args:
- -mesh-provider=istio
- -metrics-server=http://prometheus.istio-system:9090
- -slack-user=flagger
- -slack-channel=alerts
- -slack-url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
EOF
```
Install Flagger with Slack:
```bash
kubectl apply -k .
```
If you want to use MS Teams instead of Slack, replace `-slack-url` with `-msteams-url` and set the webhook address to `https://outlook.office.com/webhook/YOUR/TEAMS/WEBHOOK`.

View File

@@ -115,11 +115,18 @@ Add Flagger Helm repository:
helm repo add flagger https://flagger.app
```
Install Flagger's Canary CRD:
```yaml
kubectl apply -f https://raw.githubusercontent.com/weaveworks/flagger/master/artifacts/flagger/crd.yaml
```
Deploy Flagger in the _**istio-system**_ namespace and set the service mesh provider to SuperGloo:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set crd.create=false \
--set metricsServer=http://prometheus.istio-system:9090 \
--set meshProvider=supergloo:istio.supergloo-system
```

View File

@@ -51,7 +51,7 @@ helm upgrade -i frontend flagger/podinfo \
--namespace test \
--set nameOverride=frontend \
--set backend=http://backend.test:9898/echo \
--set canary.loadtest.enabled=true \
--set canary.enabled=true \
--set canary.istioIngress.enabled=true \
--set canary.istioIngress.gateway=public-gateway.istio-system.svc.cluster.local \
--set canary.istioIngress.host=frontend.istio.example.com
@@ -91,7 +91,7 @@ Now let's install the `backend` release without exposing it outside the mesh:
helm upgrade -i backend flagger/podinfo \
--namespace test \
--set nameOverride=backend \
--set canary.loadtest.enabled=true \
--set canary.enabled=true \
--set canary.istioIngress.enabled=false
```
@@ -138,7 +138,7 @@ helm upgrade -i frontend flagger/podinfo/ \
--reuse-values \
--set canary.loadtest.enabled=true \
--set canary.helmtest.enabled=true \
--set image.tag=1.4.1
--set image.tag=3.1.1
```
Flagger detects that the deployment revision changed and starts the canary analysis:
@@ -177,6 +177,7 @@ Now trigger a canary deployment for the `backend` app, but this time you'll chan
helm upgrade -i backend flagger/podinfo/ \
--namespace test \
--reuse-values \
--set canary.loadtest.enabled=true \
--set canary.helmtest.enabled=true \
--set httpServer.timeout=25s
```
@@ -283,17 +284,17 @@ metadata:
namespace: test
annotations:
flux.weave.works/automated: "true"
flux.weave.works/tag.chart-image: semver:~1.4
flux.weave.works/tag.chart-image: semver:~3.1
spec:
releaseName: frontend
chart:
repository: https://stefanprodan.github.io/flagger/
name: podinfo
version: 2.0.0
git: https://github.com/weaveowrks/flagger
ref: master
path: charts/podinfo
values:
image:
repository: quay.io/stefanprodan/podinfo
tag: 1.4.0
repository: stefanprodan/podinfo
tag: 3.1.0
backend: http://backend-podinfo:9898/echo
canary:
enabled: true
@@ -311,25 +312,26 @@ In the `chart` section I've defined the release source by specifying the Helm re
In the `values` section I've overwritten the defaults set in values.yaml.
With the `flux.weave.works` annotations I instruct Flux to automate this release.
When an image tag in the sem ver range of `1.4.0 - 1.4.99` is pushed to Quay,
When an image tag in the sem ver range of `3.1.0 - 3.1.99` is pushed to Docker Hub,
Flux will upgrade the Helm release and from there Flagger will pick up the change and start a canary deployment.
Install [Weave Flux](https://github.com/weaveworks/flux) and its Helm Operator by specifying your Git repo URL:
```bash
helm repo add weaveworks https://weaveworks.github.io/flux
helm repo add fluxcd https://charts.fluxcd.io
helm install --name flux \
--set helmOperator.create=true \
--set helmOperator.createCRD=true \
--set git.url=git@github.com:<USERNAME>/<REPOSITORY> \
--namespace flux \
weaveworks/flux
--namespace fluxcd \
fluxcd/flux
```
At startup Flux generates a SSH key and logs the public key. Find the SSH public key with:
```bash
kubectl -n flux logs deployment/flux | grep identity.pub | cut -d '"' -f2
kubectl -n fluxcd logs deployment/flux | grep identity.pub | cut -d '"' -f2
```
In order to sync your cluster state with Git you need to copy the public key and create a
@@ -343,9 +345,9 @@ launch the `frontend` and `backend` apps.
A CI/CD pipeline for the `frontend` release could look like this:
* cut a release from the master branch of the podinfo code repo with the git tag `1.4.1`
* CI builds the image and pushes the `podinfo:1.4.1` image to the container registry
* Flux scans the registry and updates the Helm release `image.tag` to `1.4.1`
* cut a release from the master branch of the podinfo code repo with the git tag `3.1.1`
* CI builds the image and pushes the `podinfo:3.1.1` image to the container registry
* Flux scans the registry and updates the Helm release `image.tag` to `3.1.1`
* Flux commits and push the change to the cluster repo
* Flux applies the updated Helm release on the cluster
* Flux Helm Operator picks up the change and calls Tiller to upgrade the release
@@ -353,9 +355,9 @@ A CI/CD pipeline for the `frontend` release could look like this:
* Flagger runs the helm test before routing traffic to the canary service
* Flagger starts the load test and runs the canary analysis
* Based on the analysis result the canary deployment is promoted to production or rolled back
* Flagger sends a Slack notification with the canary result
* Flagger sends a Slack or MS Teams notification with the canary result
If the canary fails, fix the bug, do another patch release eg `1.4.2` and the whole process will run again.
If the canary fails, fix the bug, do another patch release eg `3.1.2` and the whole process will run again.
A canary deployment can fail due to any of the following reasons:

View File

@@ -4,42 +4,16 @@ This guide shows you how to use the SMI Istio adapter and Flagger to automate ca
### Prerequisites
Flagger requires a Kubernetes cluster **v1.11** or newer with the following admission controllers enabled:
* Kubernetes > 1.13
* Istio > 1.0
* MutatingAdmissionWebhook
* ValidatingAdmissionWebhook
### Install Istio SMI adapter
Flagger depends on [Istio](https://istio.io/docs/setup/kubernetes/quick-start/) **v1.0.3** or newer
with traffic management, telemetry and Prometheus enabled.
A minimal Istio installation should contain the following services:
* istio-pilot
* istio-ingressgateway
* istio-sidecar-injector
* istio-telemetry
* prometheus
### Install Istio and the SMI adapter
Add Istio Helm repository:
Install the SMI adapter:
```bash
helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.1.5/charts
```
Install Istio CRDs:
```bash
helm upgrade -i istio-init istio.io/istio-init --wait --namespace istio-system
kubectl -n istio-system wait --for=condition=complete job/istio-init-crd-11
```
Install Istio:
```bash
helm upgrade -i istio istio.io/istio --wait --namespace istio-system
kubectl apply -f https://raw.githubusercontent.com/deislabs/smi-adapter-istio/master/deploy/crds/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/deislabs/smi-adapter-istio/master/deploy/operator-and-rbac.yaml
```
Create a generic Istio gateway to expose services outside the mesh on HTTP:
@@ -74,14 +48,6 @@ Find the Gateway load balancer IP and add a DNS record for it:
kubectl -n istio-system get svc/istio-ingressgateway -ojson | jq -r .status.loadBalancer.ingress[0].ip
```
Install the SMI adapter:
```bash
REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
kubectl apply -f ${REPO}/artifacts/smi/istio-adapter.yaml
```
### Install Flagger and Grafana
Add Flagger Helm repository:
@@ -95,7 +61,6 @@ Deploy Flagger in the _**istio-system**_ namespace:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set image.tag=master-12d84b2 \
--set meshProvider=smi:istio
```
@@ -119,24 +84,23 @@ kubectl -n istio-system port-forward svc/flagger-grafana 3000:80
Create a test namespace with Istio sidecar injection enabled:
```bash
export REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
Create a test namespace and enable Linkerd proxy injection:
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml
```bash
kubectl create ns test
kubectl label namespace test istio-injection=enabled
```
Create a deployment and a horizontal pod autoscaler:
```bash
kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Deploy the load testing service to generate traffic during the canary analysis:
```bash
kubectl -n test apply -f ${REPO}/artifacts/loadtester/deployment.yaml
kubectl -n test apply -f ${REPO}/artifacts/loadtester/service.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/tester
```
Create a canary custom resource (replace example.com with your own domain):
@@ -236,7 +200,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.1
podinfod=quay.io/stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
@@ -287,7 +251,7 @@ Create a tester pod and exec into it:
```bash
kubectl -n test run tester \
--image=quay.io/stefanprodan/podinfo:1.2.1 \
--image=quay.io/stefanprodan/podinfo:3.1.2 \
-- ./podinfo --port=9898
kubectl -n test exec -it tester-xx-xx sh

View File

@@ -13,23 +13,20 @@ This is particularly useful for frontend applications that require session affin
Create a test namespace with Istio sidecar injection enabled:
```bash
export REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml
kubectl create ns test
kubectl label namespace test istio-injection=enabled
```
Create a deployment and a horizontal pod autoscaler:
```bash
kubectl apply -f ${REPO}/artifacts/ab-testing/deployment.yaml
kubectl apply -f ${REPO}/artifacts/ab-testing/hpa.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Deploy the load testing service to generate traffic during the canary analysis:
```bash
kubectl -n test apply -f ${REPO}/artifacts/loadtester/deployment.yaml
kubectl -n test apply -f ${REPO}/artifacts/loadtester/service.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/tester
```
Create a canary custom resource (replace example.com with your own domain):
@@ -38,14 +35,14 @@ Create a canary custom resource (replace example.com with your own domain):
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: abtest
name: podinfo
namespace: test
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: abtest
name: podinfo
# the maximum time in seconds for the canary deployment
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
@@ -53,7 +50,7 @@ spec:
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: abtest
name: podinfo
service:
# container port
port: 9898
@@ -63,6 +60,11 @@ spec:
# Istio virtual service host names (optional)
hosts:
- app.example.com
# Istio traffic policy (optional)
trafficPolicy:
tls:
# use ISTIO_MUTUAL when mTLS is enabled
mode: DISABLE
canaryAnalysis:
# schedule interval (default 60s)
interval: 1m
@@ -110,19 +112,19 @@ After a couple of seconds Flagger will create the canary objects:
```bash
# applied
deployment.apps/abtest
horizontalpodautoscaler.autoscaling/abtest
canary.flagger.app/abtest
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated
deployment.apps/abtest-primary
horizontalpodautoscaler.autoscaling/abtest-primary
service/abtest
service/abtest-canary
service/abtest-primary
destinationrule.networking.istio.io/abtest-canary
destinationrule.networking.istio.io/abtest-primary
virtualservice.networking.istio.io/abtest
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
destinationrule.networking.istio.io/podinfo-canary
destinationrule.networking.istio.io/podinfo-primary
virtualservice.networking.istio.io/podinfo
```
### Automated canary promotion
@@ -131,7 +133,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/abtest \
podinfod=quay.io/stefanprodan/podinfo:1.4.1
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
@@ -145,22 +147,22 @@ Status:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger New revision detected abtest.test
Normal Synced 3m flagger Scaling up abtest.test
Warning Synced 3m flagger Waiting for abtest.test rollout to finish: 0 of 1 updated replicas are available
Normal Synced 3m flagger Advance abtest.test canary iteration 1/10
Normal Synced 3m flagger Advance abtest.test canary iteration 2/10
Normal Synced 3m flagger Advance abtest.test canary iteration 3/10
Normal Synced 2m flagger Advance abtest.test canary iteration 4/10
Normal Synced 2m flagger Advance abtest.test canary iteration 5/10
Normal Synced 1m flagger Advance abtest.test canary iteration 6/10
Normal Synced 1m flagger Advance abtest.test canary iteration 7/10
Normal Synced 55s flagger Advance abtest.test canary iteration 8/10
Normal Synced 45s flagger Advance abtest.test canary iteration 9/10
Normal Synced 35s flagger Advance abtest.test canary iteration 10/10
Normal Synced 25s flagger Copying abtest.test template spec to abtest-primary.test
Normal Synced 3m flagger New revision detected podinfo.test
Normal Synced 3m flagger Scaling up podinfo.test
Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
Normal Synced 2m flagger Advance podinfo.test canary iteration 4/10
Normal Synced 2m flagger Advance podinfo.test canary iteration 5/10
Normal Synced 1m flagger Advance podinfo.test canary iteration 6/10
Normal Synced 1m flagger Advance podinfo.test canary iteration 7/10
Normal Synced 55s flagger Advance podinfo.test canary iteration 8/10
Normal Synced 45s flagger Advance podinfo.test canary iteration 9/10
Normal Synced 35s flagger Advance podinfo.test canary iteration 10/10
Normal Synced 25s flagger Copying podinfo.test template spec to abtest-primary.test
Warning Synced 15s flagger Waiting for abtest-primary.test rollout to finish: 1 of 2 updated replicas are available
Normal Synced 5s flagger Promotion completed! Scaling down abtest.test
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
```
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
@@ -204,12 +206,12 @@ Status:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger Starting canary deployment for abtest.test
Normal Synced 3m flagger Advance abtest.test canary iteration 1/10
Normal Synced 3m flagger Advance abtest.test canary iteration 2/10
Normal Synced 3m flagger Advance abtest.test canary iteration 3/10
Normal Synced 3m flagger Halt abtest.test advancement success rate 69.17% < 99%
Normal Synced 2m flagger Halt abtest.test advancement success rate 61.39% < 99%
Warning Synced 2m flagger Rolling back abtest.test failed checks threshold reached 2
Warning Synced 1m flagger Canary failed! Scaling down abtest.test
Normal Synced 3m flagger Starting canary deployment for podinfo.test
Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
Warning Synced 2m flagger Rolling back podinfo.test failed checks threshold reached 2
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
```

View File

@@ -6,7 +6,6 @@ Flagger can be configured to send Slack notifications:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \
--set slack.user=flagger
@@ -22,6 +21,23 @@ maximum number of failed checks:
![Slack Notifications](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/screens/slack-canary-failed.png)
### Microsoft Teams
Flagger can be configured to send notifications to Microsoft Teams:
```bash
helm upgrade -i flagger flagger/flagger \
--set msteams.url=https://outlook.office.com/webhook/YOUR/TEAMS/WEBHOOK
```
Flagger will post a message card to MS Teams when a new revision has been detected and if the canary analysis failed or succeeded:
![MS Teams Notifications](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/screens/flagger-ms-teams-notifications.png)
And you'll get a notification on rollback:
![MS Teams Notifications](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/screens/flagger-ms-teams-failed.png)
### Prometheus Alert Manager
Besides Slack, you can use Alertmanager to trigger alerts when a canary deployment failed:

View File

@@ -37,8 +37,9 @@ Deploy the load testing service to generate traffic during the canary analysis:
```bash
helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test \
--set meshName=global.appmesh-system \
--set "backends[0]=podinfo.test"
--set meshName=global \
--set "backends[0]=podinfo.test" \
--set "backends[1]=podinfo-canary.test"
```
Create a canary custom resource:
@@ -66,11 +67,19 @@ spec:
service:
# container port
port: 9898
# container port name (optional)
# can be http or grpc
portName: http
# App Mesh reference
meshName: global.appmesh-system
meshName: global
# App Mesh egress (optional)
backends:
- backend.test
# App Mesh retry policy (optional)
retries:
attempts: 3
perTryTimeout: 1s
retryOn: "gateway-error,client-error,stream-error"
# define the canary analysis timing and KPIs
canaryAnalysis:
# schedule interval (default 60s)
@@ -90,13 +99,25 @@ spec:
# percentage (0-100)
threshold: 99
interval: 1m
# external checks (optional)
- name: request-duration
# maximum req duration P99
# milliseconds
threshold: 500
interval: 30s
# testing (optional)
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"
```
Save the above resource as podinfo-canary.yaml and then apply it:
@@ -125,14 +146,18 @@ virtualnode.appmesh.k8s.aws/podinfo
virtualnode.appmesh.k8s.aws/podinfo-canary
virtualnode.appmesh.k8s.aws/podinfo-primary
virtualservice.appmesh.k8s.aws/podinfo.test
virtualservice.appmesh.k8s.aws/podinfo-canary.test
```
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test` will be routed
to the primary pods. During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
The App Mesh specific settings are:
```yaml
service:
port: 9898
meshName: global.appmesh-system
meshName: global
backends:
- backend1.test
- backend2.test
@@ -176,7 +201,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.1
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
@@ -189,28 +214,34 @@ Status:
Failed Checks: 0
Phase: Succeeded
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger New revision detected podinfo.test
Normal Synced 3m flagger Scaling up podinfo.test
Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Normal Synced 3m flagger Advance podinfo.test canary weight 5
Normal Synced 3m flagger Advance podinfo.test canary weight 10
Normal Synced 3m flagger Advance podinfo.test canary weight 15
Normal Synced 2m flagger Advance podinfo.test canary weight 20
Normal Synced 2m flagger Advance podinfo.test canary weight 25
Normal Synced 1m flagger Advance podinfo.test canary weight 30
Normal Synced 1m flagger Advance podinfo.test canary weight 35
Normal Synced 55s flagger Advance podinfo.test canary weight 40
Normal Synced 45s flagger Advance podinfo.test canary weight 45
Normal Synced 35s flagger Advance podinfo.test canary weight 50
Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
New revision detected! Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Routing all traffic to primary
Promotion completed! Scaling down podinfo.test
```
When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
A canary deployment is triggered by changes in any of the following objects:
* Deployment PodSpec (container image, command, ports, env, resources, etc)
* ConfigMaps mounted as volumes or mapped to environment variables
* Secrets mounted as volumes or mapped to environment variables
During the analysis the canarys progress can be monitored with Grafana. The App Mesh dashboard URL is
http://localhost:3000/d/flagger-appmesh/appmesh-canary?refresh=10s&orgId=1&var-namespace=test&var-primary=podinfo-primary&var-canary=podinfo
@@ -222,9 +253,9 @@ You can monitor all canaries with:
watch kubectl get canaries --all-namespaces
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-03-16T14:05:07Z
prod frontend Succeeded 0 2019-03-15T16:15:07Z
prod backend Failed 0 2019-03-14T17:05:07Z
test podinfo Progressing 15 2019-10-02T14:05:07Z
prod frontend Succeeded 0 2019-10-02T16:15:07Z
prod backend Failed 0 2019-10-02T17:05:07Z
```
If youve enabled the Slack notifications, you should receive the following messages:
@@ -239,19 +270,25 @@ Trigger a canary deployment:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.2
podinfod=stefanprodan/podinfo:3.1.2
```
Exec into the load tester pod with:
```bash
kubectl -n test exec -it flagger-loadtester-xx-xx sh
kubectl -n test exec -it deploy/flagger-loadtester bash
```
Generate HTTP 500 errors:
```bash
hey -z 1m -c 5 -q 5 http://podinfo.test:9898/status/500
hey -z 1m -c 5 -q 5 http://podinfo-canary.test:9898/status/500
```
Generate latency:
```bash
watch -n 1 curl http://podinfo-canary.test:9898/delay/1
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
@@ -262,25 +299,95 @@ kubectl -n test describe canary/podinfo
Status:
Canary Weight: 0
Failed Checks: 10
Failed Checks: 5
Phase: Failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger Starting canary deployment for podinfo.test
Normal Synced 3m flagger Advance podinfo.test canary weight 5
Normal Synced 3m flagger Advance podinfo.test canary weight 10
Normal Synced 3m flagger Advance podinfo.test canary weight 15
Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% < 99%
Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% < 99%
Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
Starting canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement request duration 1.20s > 0.5s
Halt podinfo.test advancement request duration 1.45s > 0.5s
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
If youve enabled the Slack notifications, youll receive a message if the progress deadline is exceeded,
or if the analysis reached the maximum number of failed checks:
![Flagger Slack Notifications](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/screens/slack-canary-failed.png)
### A/B Testing
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
This is particularly useful for frontend applications that require session affinity.
![Flagger A/B Testing Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-abtest-steps.png)
Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:
```yaml
canaryAnalysis:
interval: 1m
threshold: 10
iterations: 10
match:
- headers:
x-canary:
exact: "insider"
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
metadata:
cmd: "hey -z 1m -q 10 -c 2 -H 'X-Canary: insider' http://podinfo.test:9898/"
```
The above configuration will run an analysis for ten minutes targeting users that have a `X-Canary: insider` header.
You can also use a HTTP cookie, to target all users with a `canary` cookie set to `insider` the match condition should be:
```yaml
match:
- headers:
cookie:
regex: "^(.*?;)?(canary=insider)(;.*)?$"
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
metadata:
cmd: "hey -z 1m -q 10 -c 2 -H 'Cookie: canary=insider' http://podinfo.test:9898/"
```
Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.3
```
Flagger detects that the deployment revision changed and starts the A/B test:
```text
kubectl -n appmesh-system logs deploy/flagger -f | jq .msg
New revision detected! Starting canary analysis for podinfo.test
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Routing all traffic to primary
Promotion completed! Scaling down podinfo.test
```

View File

@@ -0,0 +1,355 @@
# Blue/Green Deployments
This guide shows you how to automate Blue/Green deployments with Flagger and Kubernetes.
For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments
with Kubernetes L4 networking.
When using a service mesh blue/green can be used as specified [here](https://docs.flagger.app/how-it-works#blue-green-deployments).
![Flagger Blue/Green Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-bluegreen-steps.png)
### Prerequisites
Flagger requires a Kubernetes cluster **v1.11** or newer.
Install Flagger and the Prometheus add-on:
```bash
helm repo add flagger https://flagger.app
helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set prometheus.install=true \
--set meshProvider=kubernetes
```
If you already have a Prometheus instance running in your cluster,
you can point Flagger to the ClusterIP service with:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set metricsServer=http://prometheus.monitoring:9090
```
Optionally you can enable Slack notifications:
```bash
helm upgrade -i flagger flagger/flagger \
--reuse-values \
--namespace flagger \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \
--set slack.user=flagger
```
### Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployment and ClusterIP services).
These objects expose the application inside the cluster and drive the canary analysis and Blue/Green promotion.
Create a test namespace:
```bash
kubectl create ns test
```
Create a deployment and a horizontal pod autoscaler:
```bash
export REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml
```
Deploy the load testing service to generate traffic during the analysis:
```bash
kubectl -n test apply -f ${REPO}/artifacts/loadtester/deployment.yaml
kubectl -n test apply -f ${REPO}/artifacts/loadtester/service.yaml
```
Create a canary custom resource:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
# service mesh provider can be: kubernetes, istio, appmesh, nginx, gloo
provider: kubernetes
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
# the maximum time in seconds for the canary deployment
# to make progress before rollback (default 600s)
progressDeadlineSeconds: 60
# HPA reference (optional)
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: podinfo
service:
port: 9898
portDiscovery: true
canaryAnalysis:
# schedule interval (default 60s)
interval: 30s
# max number of failed checks before rollback
threshold: 2
# number of checks to run before rollback
iterations: 10
# Prometheus checks based on
# http_request_duration_seconds histogram
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)
# percentage (0-100)
threshold: 99
interval: 1m
- name: request-duration
# maximum req duration P99
# milliseconds
threshold: 500
interval: 30s
# acceptance/load testing hooks
webhooks:
- name: smoke-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 15s
metadata:
type: bash
cmd: "curl -sd 'anon' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"
```
The above configuration will run an analysis for five minutes.
Save the above resource as podinfo-canary.yaml and then apply it:
```bash
kubectl apply -f ./podinfo-canary.yaml
```
After a couple of seconds Flagger will create the canary objects:
```bash
# applied
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo
# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
```
Blue/Green scenario:
* on bootstrap, Flagger will create three ClusterIP services (`app-primary`,` app-canary`, `app`) and a shadow deployment named `app-primary` that represents the blue version
* when a new version is detected, Flagger would scale up the green version and run the conformance tests (the tests should target the `app-canary` ClusterIP service to reach the green version)
* if the conformance tests are passing, Flagger would start the load tests and validate them with custom Prometheus queries
* if the load test analysis is successful, Flagger will promote the new version to `app-primary` and scale down the green version
### Automated Blue/Green promotion
Trigger a deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
```text
kubectl -n test describe canary/podinfo
Events:
New revision detected podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Pre-rollout check acceptance-test passed
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test
```
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
You can monitor all canaries with:
```bash
watch kubectl get canaries --all-namespaces
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 100 2019-06-16T14:05:07Z
prod frontend Succeeded 0 2019-06-15T16:15:07Z
prod backend Failed 0 2019-06-14T17:05:07Z
```
### Automated rollback
During the analysis you can generate HTTP 500 errors and high latency to test Flagger's rollback.
Exec into the load tester pod with:
```bash
kubectl -n test exec -it flagger-loadtester-xx-xx sh
```
Generate HTTP 500 errors:
```bash
watch curl http://podinfo-canary.test:9898/status/500
```
Generate latency:
```bash
watch curl http://podinfo-canary.test:9898/delay/1
```
When the number of failed checks reaches the analysis threshold,
the green version is scaled to zero and the rollout is marked as failed.
```text
kubectl -n test describe canary/podinfo
Status:
Failed Checks: 2
Phase: Failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger New revision detected podinfo.test
Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
Warning Synced 2m flagger Rolling back podinfo.test failed checks threshold reached 2
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
```
### Custom metrics
The analysis can be extended with Prometheus queries. The demo app is instrumented with Prometheus
so you can create a custom check that will use the HTTP request duration histogram to validate the canary (green version).
Edit the canary analysis and add the following metric:
```yaml
canaryAnalysis:
metrics:
- name: "404s percentage"
threshold: 5
query: |
100 - sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
status!="404"
}[1m]
)
)
/
sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[1m]
)
) * 100
```
The above configuration validates the canary (green version) by checking if the HTTP 404 req/sec percentage is below 5
percent of the total traffic. If the 404s rate reaches the 5% threshold, then the rollout is rolled back.
Trigger a deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.3
```
Generate 404s:
```bash
watch curl http://podinfo-canary.test:9898/status/400
```
Watch Flagger logs:
```
kubectl -n flagger logs deployment/flagger -f | jq .msg
New revision detected podinfo.test
Scaling up podinfo.test
Advance podinfo.test canary iteration 1/10
Halt podinfo.test advancement 404s percentage 6.20 > 5
Halt podinfo.test advancement 404s percentage 6.45 > 5
Rolling back podinfo.test failed checks threshold reached 2
Canary failed! Scaling down podinfo.test
```
If you have Slack configured, Flagger will send a notification with the reason why the canary failed.
### Conformance Testing with Helm
Flagger comes with a testing service that can run Helm tests when configured as a pre-rollout webhook.
Deploy the Helm test runner in the `kube-system` namespace using the `tiller` service account:
```bash
helm repo add flagger https://flagger.app
helm upgrade -i flagger-helmtester flagger/loadtester \
--namespace=kube-system \
--set serviceAccountName=tiller
```
When deployed the Helm tester API will be available at `http://flagger-helmtester.kube-system/`.
Add a helm test pre-rollout hook to your chart:
```yaml
canaryAnalysis:
webhooks:
- name: "conformance testing"
type: pre-rollout
url: http://flagger-helmtester.kube-system/
timeout: 3m
metadata:
type: "helm"
cmd: "test {{ .Release.Name }} --cleanup"
```
When the canary analysis starts, Flagger will call the pre-rollout webhooks.
If the helm test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.

View File

@@ -54,15 +54,13 @@ kubectl create ns test
Create a deployment and a horizontal pod autoscaler:
```bash
kubectl apply -f ${REPO}/artifacts/gloo/deployment.yaml
kubectl apply -f ${REPO}/artifacts/gloo/hpa.yaml
kubectl -n test apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Deploy the load testing service to generate traffic during the canary analysis:
```bash
helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test
kubectl -n test apply -k github.com/weaveworks/flagger//kustomize/tester
```
Create an virtual service definition that references an upstream group that will be generated by Flagger
@@ -103,6 +101,7 @@ metadata:
name: podinfo
namespace: test
spec:
provider: gloo
# deployment reference
targetRef:
apiVersion: apps/v1
@@ -117,8 +116,10 @@ spec:
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
service:
# container port
# ClusterIP port number
port: 9898
# container port number or name (optional)
targetPort: 9898
canaryAnalysis:
# schedule interval (default 60s)
interval: 10s
@@ -142,14 +143,21 @@ spec:
# milliseconds
threshold: 500
interval: 30s
# load testing (optional)
# testing (optinal)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 10s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://app.example.com/"
cmd: "hey -z 2m -q 5 -c 2 -host app.example.com http://gateway-proxy-v2.gloo-system"
```
Save the above resource as podinfo-canary.yaml and then apply it:
@@ -197,7 +205,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.1
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
@@ -251,19 +259,19 @@ Trigger another canary deployment:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.2
podinfod=stefanprodan/podinfo:3.1.2
```
Generate HTTP 500 errors:
```bash
watch curl http://app.example.com/status/500
watch curl -H 'Host: app.example.com' http://gateway-proxy-v2.gloo-system/status/500
```
Generate high latency:
```bash
watch curl http://app.example.com/delay/2
watch curl -H 'Host: app.example.com' http://gateway-proxy-v2.gloo-system/delay/2
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
@@ -334,13 +342,13 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.3
podinfod=stefanprodan/podinfo:3.1.3
```
Generate 404s:
```bash
watch curl http://app.example.com/status/400
watch curl -H 'Host: app.example.com' http://gateway-proxy-v2.gloo-system/status/400
```
Watch Flagger logs:
@@ -362,5 +370,3 @@ Canary failed! Scaling down podinfo.test
```
If you have Slack configured, Flagger will send a notification with the reason why the canary failed.

View File

@@ -0,0 +1,473 @@
# Linkerd Canary Deployments
This guide shows you how to use Linkerd and Flagger to automate canary deployments.
![Flagger Linkerd Traffic Split](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-linkerd-traffic-split.png)
### Prerequisites
Flagger requires a Kubernetes cluster **v1.11** or newer and Linkerd **2.4** or newer.
Install Flagger in the linkerd namespace:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/linkerd
```
Note that you'll need kubectl 1.14 or newer to run the above command.
To enable Slack or MS Teams notifications,
see Flagger's [install docs](https://docs.flagger.app/install/flagger-install-on-kubernetes) for Kustomize or Helm options.
### Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services and SMI traffic split).
These objects expose the application inside the mesh and drive the canary analysis and promotion.
Create a test namespace and enable Linkerd proxy injection:
```bash
kubectl create ns test
kubectl annotate namespace test linkerd.io/inject=enabled
```
Install the load testing service to generate traffic during the canary analysis:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/tester
```
Create a deployment and a horizontal pod autoscaler:
```bash
kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Create a canary custom resource for the podinfo deployment:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
# deployment reference
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
# HPA reference (optional)
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: podinfo
# the maximum time in seconds for the canary deployment
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
service:
# ClusterIP port number
port: 9898
# container port number or name (optional)
targetPort: 9898
canaryAnalysis:
# schedule interval (default 60s)
interval: 30s
# max number of failed metric checks before rollback
threshold: 5
# max traffic percentage routed to canary
# percentage (0-100)
maxWeight: 50
# canary increment step
# percentage (0-100)
stepWeight: 5
# Linkerd Prometheus checks
metrics:
- name: request-success-rate
# minimum req success rate (non 5xx responses)
# percentage (0-100)
threshold: 99
interval: 1m
- name: request-duration
# maximum req duration P99
# milliseconds
threshold: 500
interval: 30s
# testing (optional)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
- name: load-test
type: rollout
url: http://flagger-loadtester.test/
metadata:
cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"
```
Save the above resource as podinfo-canary.yaml and then apply it:
```bash
kubectl apply -f ./podinfo-canary.yaml
```
When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.
The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every half a minute.
After a couple of seconds Flagger will create the canary objects:
```bash
# applied
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingresses.extensions/podinfo
canary.flagger.app/podinfo
# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
trafficsplits.split.smi-spec.io/podinfo
```
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test` will be routed
to the primary pods. During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
### Automated canary promotion
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators
like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
```text
kubectl -n test describe canary/podinfo
Status:
Canary Weight: 0
Failed Checks: 0
Phase: Succeeded
Events:
New revision detected! Scaling up podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Advance podinfo.test canary weight 25
Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
Advance podinfo.test canary weight 30
Advance podinfo.test canary weight 35
Advance podinfo.test canary weight 40
Advance podinfo.test canary weight 45
Advance podinfo.test canary weight 50
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test
```
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
A canary deployment is triggered by changes in any of the following objects:
* Deployment PodSpec (container image, command, ports, env, resources, etc)
* ConfigMaps mounted as volumes or mapped to environment variables
* Secrets mounted as volumes or mapped to environment variables
You can monitor all canaries with:
```bash
watch kubectl get canaries --all-namespaces
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-06-30T14:05:07Z
prod frontend Succeeded 0 2019-06-30T16:15:07Z
prod backend Failed 0 2019-06-30T17:05:07Z
```
### Automated rollback
During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses and rolls back the faulted version.
Trigger another canary deployment:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.2
```
Exec into the load tester pod with:
```bash
kubectl -n test exec -it flagger-loadtester-xx-xx sh
```
Generate HTTP 500 errors:
```bash
watch -n 1 curl http://podinfo-canary.test:9898/status/500
```
Generate latency:
```bash
watch -n 1 curl http://podinfo-canary.test:9898/delay/1
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```text
kubectl -n test describe canary/podinfo
Status:
Canary Weight: 0
Failed Checks: 10
Phase: Failed
Events:
Starting canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement request duration 1.20s > 0.5s
Halt podinfo.test advancement request duration 1.45s > 0.5s
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
### Custom metrics
The canary analysis can be extended with Prometheus queries.
Let's a define a check for not found errors. Edit the canary analysis and add the following metric:
```yaml
canaryAnalysis:
metrics:
- name: "404s percentage"
threshold: 3
query: |
100 - sum(
rate(
response_total{
namespace="test",
deployment="podinfo",
status_code!="404",
direction="inbound"
}[1m]
)
)
/
sum(
rate(
response_total{
namespace="test",
deployment="podinfo",
direction="inbound"
}[1m]
)
)
* 100
```
The above configuration validates the canary version by checking if the HTTP 404 req/sec percentage is below
three percent of the total traffic. If the 404s rate reaches the 3% threshold, then the analysis is aborted and the
canary is marked as failed.
Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.3
```
Generate 404s:
```bash
watch -n 1 curl http://podinfo-canary:9898/status/404
```
Watch Flagger logs:
```
kubectl -n linkerd logs deployment/flagger -f | jq .msg
Starting canary deployment for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Halt podinfo.test advancement 404s percentage 6.20 > 3
Halt podinfo.test advancement 404s percentage 6.45 > 3
Halt podinfo.test advancement 404s percentage 7.22 > 3
Halt podinfo.test advancement 404s percentage 6.50 > 3
Halt podinfo.test advancement 404s percentage 6.34 > 3
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
If you have Slack configured, Flagger will send a notification with the reason why the canary failed.
### Linkerd Ingress
There are two ingress controllers that are compatible with both Flagger and Linkerd: NGINX and Gloo.
Install NGINX:
```bash
helm upgrade -i nginx-ingress stable/nginx-ingress \
--namespace ingress-nginx
```
Create an ingress definition for podinfo that rewrites the incoming header to the internal service name (required by Linkerd):
```yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: podinfo
namespace: test
labels:
app: podinfo
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:9898;
proxy_hide_header l5d-remote-ip;
proxy_hide_header l5d-server-id;
spec:
rules:
- host: app.example.com
http:
paths:
- backend:
serviceName: podinfo
servicePort: 9898
```
When using an ingress controller, the Linkerd traffic split does not apply to incoming traffic since NGINX in running outside of
the mesh. In order to run a canary analysis for a frontend app, Flagger creates a shadow ingress and sets the NGINX specific annotations.
### A/B Testing
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
This is particularly useful for frontend applications that require session affinity.
![Flagger Linkerd Ingress](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-nginx-linkerd.png)
Edit podinfo canary analysis, set the provider to `nginx`, add the ingress reference, remove the max/step weight and add the match conditions and iterations:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
# ingress reference
provider: nginx
ingressRef:
apiVersion: extensions/v1beta1
kind: Ingress
name: podinfo
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
autoscalerRef:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
name: podinfo
service:
# container port
port: 9898
canaryAnalysis:
interval: 1m
threshold: 10
iterations: 10
match:
# curl -H 'X-Canary: always' http://app.example.com
- headers:
x-canary:
exact: "always"
# curl -b 'canary=always' http://app.example.com
- headers:
cookie:
exact: "canary"
# Linkerd Prometheus checks
metrics:
- name: request-success-rate
threshold: 99
interval: 1m
- name: request-duration
threshold: 500
interval: 30s
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
- name: load-test
type: rollout
url: http://flagger-loadtester.test/
metadata:
cmd: "hey -z 2m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com"
```
The above configuration will run an analysis for ten minutes targeting users that have a `canary` cookie set to `always` or
those that call the service using the `X-Canary: always` header.
**Note** that the load test now targets the external address and uses the canary cookie.
Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.4
```
Flagger detects that the deployment revision changed and starts the A/B testing:
```text
kubectl -n test describe canary/podinfo
Events:
Starting canary deployment for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test
```

View File

@@ -56,8 +56,7 @@ kubectl create ns test
Create a deployment and a horizontal pod autoscaler:
```bash
kubectl apply -f ${REPO}/artifacts/nginx/deployment.yaml
kubectl apply -f ${REPO}/artifacts/nginx/hpa.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Deploy the load testing service to generate traffic during the canary analysis:
@@ -86,7 +85,7 @@ spec:
paths:
- backend:
serviceName: podinfo
servicePort: 9898
servicePort: 80
```
Save the above resource as podinfo-ingress.yaml and then apply it:
@@ -104,6 +103,7 @@ metadata:
name: podinfo
namespace: test
spec:
provider: nginx
# deployment reference
targetRef:
apiVersion: apps/v1
@@ -123,8 +123,10 @@ spec:
# to make progress before it is rollback (default 600s)
progressDeadlineSeconds: 60
service:
# container port
port: 9898
# ClusterIP port number
port: 80
# container port number or name
targetPort: 9898
canaryAnalysis:
# schedule interval (default 60s)
interval: 10s
@@ -143,13 +145,19 @@ spec:
# percentage (0-100)
threshold: 99
interval: 1m
# load testing (optional)
# testing (optional)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 http://app.example.com/"
```
@@ -189,7 +197,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.1
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
@@ -243,7 +251,7 @@ Trigger another canary deployment:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.2
podinfod=stefanprodan/podinfo:3.1.2
```
Generate HTTP 500 errors:
@@ -313,7 +321,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.3
podinfod=stefanprodan/podinfo:3.1.3
```
Generate high response latency:
@@ -372,12 +380,10 @@ Edit the canary analysis, remove the max/step weight and add the match condition
interval: 1m
webhooks:
- name: load-test
url: http://localhost:8888/
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
type: cmd
cmd: "hey -z 1m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com/"
logCmdOutput: "true"
```
The above configuration will run an analysis for ten minutes targeting users that have a `canary` cookie set to `always` or
@@ -387,7 +393,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.5.0
podinfod=stefanprodan/podinfo:3.1.4
```
Flagger detects that the deployment revision changed and starts the A/B testing:
@@ -418,4 +424,3 @@ Events:
Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
```

View File

@@ -2,28 +2,31 @@
This guide shows you how to use Istio and Flagger to automate canary deployments.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
### Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services, Istio destination rules and virtual services).
These objects expose the application inside the mesh and drive the canary analysis and promotion.
Create a test namespace with Istio sidecar injection enabled:
```bash
export REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml
kubectl create ns test
kubectl label namespace test istio-injection=enabled
```
Create a deployment and a horizontal pod autoscaler:
```bash
kubectl apply -f ${REPO}/artifacts/canaries/deployment.yaml
kubectl apply -f ${REPO}/artifacts/canaries/hpa.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Deploy the load testing service to generate traffic during the canary analysis:
```bash
kubectl -n test apply -f ${REPO}/artifacts/loadtester/deployment.yaml
kubectl -n test apply -f ${REPO}/artifacts/loadtester/service.yaml
kubectl apply -k github.com/weaveworks/flagger//kustomize/tester
```
Create a canary custom resource (replace example.com with your own domain):
@@ -49,14 +52,26 @@ spec:
kind: HorizontalPodAutoscaler
name: podinfo
service:
# container port
# service port number
port: 9898
# container port number or name (optional)
targetPort: 9898
# Istio gateways (optional)
gateways:
- public-gateway.istio-system.svc.cluster.local
# Istio virtual service host names (optional)
hosts:
- app.example.com
# Istio traffic policy (optional)
trafficPolicy:
tls:
# use ISTIO_MUTUAL when mTLS is enabled
mode: DISABLE
# Istio retry policy (optional)
retries:
attempts: 3
perTryTimeout: 1s
retryOn: "gateway-error,connect-failure,refused-stream"
canaryAnalysis:
# schedule interval (default 60s)
interval: 1m
@@ -79,8 +94,15 @@ spec:
# milliseconds
threshold: 500
interval: 30s
# generate traffic during analysis
# testing (optional)
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
@@ -94,6 +116,11 @@ Save the above resource as podinfo-canary.yaml and then apply it:
kubectl apply -f ./podinfo-canary.yaml
```
When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.
The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every minute.
![Flagger Canary Process](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-hpa.png)
After a couple of seconds Flagger will create the canary objects:
```bash
@@ -119,7 +146,7 @@ Trigger a canary deployment by updating the container image:
```bash
kubectl -n test set image deployment/podinfo \
podinfod=quay.io/stefanprodan/podinfo:1.4.1
podinfod=stefanprodan/podinfo:3.1.1
```
Flagger detects that the deployment revision changed and starts a new rollout:
@@ -154,6 +181,11 @@ Events:
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
A canary deployment is triggered by changes in any of the following objects:
* Deployment PodSpec (container image, command, ports, env, resources, etc)
* ConfigMaps mounted as volumes or mapped to environment variables
* Secrets mounted as volumes or mapped to environment variables
You can monitor all canaries with:
```bash
@@ -169,14 +201,17 @@ prod backend Failed 0 2019-01-14T17:05:07Z
During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses the rollout.
Create a tester pod and exec into it:
Trigger another canary deployment:
```bash
kubectl -n test run tester \
--image=quay.io/stefanprodan/podinfo:1.2.1 \
-- ./podinfo --port=9898
kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:3.1.2
```
kubectl -n test exec -it tester-xx-xx sh
Exec into the load tester pod with:
```bash
kubectl -n test exec -it flagger-loadtester-xx-xx sh
```
Generate HTTP 500 errors:
@@ -217,3 +252,81 @@ Events:
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
```
### Traffic mirroring
![Flagger Canary Traffic Shadowing](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-traffic-mirroring.png)
For applications that perform read operations, Flagger can be configured to drive canary releases with traffic mirroring.
Istio traffic mirroring will copy each incoming request, sending one request to the primary and one to the canary service.
The response from the primary is sent back to the user and the response from the canary is discarded.
Metrics are collected on both requests so that the deployment will only proceed if the canary metrics are within the threshold values.
Note that mirroring should be used for requests that are **idempotent** or capable of being processed twice
(once by the primary and once by the canary).
You can enable mirroring by replacing `stepWeight/maxWeight` with `iterations` and
by setting `canaryAnalysis.mirror` to `true`:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
canaryAnalysis:
# schedule interval
interval: 1m
# max number of failed metric checks before rollback
threshold: 5
# total number of iterations
iterations: 10
# enable traffic shadowing
mirror: true
metrics:
- name: request-success-rate
threshold: 99
interval: 1m
- name: request-duration
threshold: 500
interval: 1m
webhooks:
- name: acceptance-test
type: pre-rollout
url: http://flagger-loadtester.test/
timeout: 30s
metadata:
type: bash
cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
- name: load-test
url: http://flagger-loadtester.test/
timeout: 5s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"
```
With the above configuration, Flagger will run a canary release with the following steps:
* detect new revision (deployment spec, secrets or configmaps changes)
* scale from zero the canary deployment
* wait for the HPA to set the canary minimum replicas
* check canary pods health
* run the acceptance tests
* abort the canary release if tests fail
* start the load tests
* mirror traffic from primary to canary
* check request success rate and request duration every minute
* abort the canary release if the metrics check failure threshold is reached
* stop traffic mirroring after the number of iterations is reached
* route live traffic to the canary pods
* promote the canary (update the primary secrets, configmaps and deployment spec)
* wait for the primary deployment rollout to finish
* wait for the HPA to set the primary minimum replicas
* check primary pods health
* switch live traffic back to primary
* scale to zero the canary
* send notification with the canary analysis result
The above procedure can be extended with [custom metrics](https://docs.flagger.app/how-it-works#custom-metrics) checks,
[webhooks](https://docs.flagger.app/how-it-works#webhooks),
[manual promotion](https://docs.flagger.app/how-it-works#manual-gating) approval and
[Slack or MS Teams](https://docs.flagger.app/usage/alerting) notifications.

Some files were not shown because too many files have changed in this diff Show More