github/flagger
1
0
Fork 0
mirror of https://github.com/fluxcd/flagger.git synced 2026-04-15 06:57:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: use metric providers in tutorials

Browse Source
This commit is contained in:
stefanprodan
2020-02-29 11:56:13 +02:00
parent c4a9712b81
commit bf0499e8a6
13 changed files with 271 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitbook.yaml
+1
View File
@@ -1,6 +1,7 @@
root: ./docs/gitbook
redirects:
how-it-works: usage/how-it-works.md
usage/progressive-delivery: tutorials/istio-progressive-delivery.md
usage/ab-testing: tutorials/istio-ab-testing.md
usage/blue-green: tutorials/kubernetes-blue-green.md
docs/gitbook/SUMMARY.md
+1 -2
View File
@@ -1,7 +1,6 @@
# Table of contents
* [Introduction](README.md)
* [How it works](how-it-works.md)
* [FAQ](faq.md)
## Install
@@ -9,10 +8,10 @@
* [Flagger Install on Kubernetes](install/flagger-install-on-kubernetes.md)
* [Flagger Install on GKE Istio](install/flagger-install-on-google-cloud.md)
* [Flagger Install on EKS App Mesh](install/flagger-install-on-eks-appmesh.md)
* [Flagger Install with SuperGloo](install/flagger-install-with-supergloo.md)
## Usage
* [How it works](usage/how-it-works.md)
* [Deployment Strategies](usage/deployment-strategies.md)
* [Metrics Analysis](usage/metrics.md)
* [Webhooks](usage/webhooks.md)
docs/gitbook/faq.md
+5
View File
@@ -336,6 +336,11 @@ histogram_quantile(0.99,
> **Note** that the metric interval should be lower or equal to the control loop interval.
**Can I use custom metrics?**
The analysis can be extended with metrics provided by Prometheus, Datadog and AWS CloudWatch. For more details
on how custom metrics can be used please read the [metrics docs](usage/metrics.md).
### Istio routing
**How does Flagger interact with Istio?**
docs/gitbook/tutorials/appmesh-progressive-delivery.md
+1
View File
@@ -389,3 +389,4 @@ Routing all traffic to primary
Promotion completed! Scaling down podinfo.test
```
For an in-depth look at the analysis process read the [usage docs](../usage/how-it-works.md).
docs/gitbook/tutorials/contour-progressive-delivery.md
+31 -16
View File
@@ -22,7 +22,7 @@ Install Flagger using Kustomize \(kubectl 1.14\) in the `projectcontour` namespa
kubectl apply -k github.com/weaveworks/flagger//kustomize/contour
```
The above command will deploy Flagger and Prometheus configured to scrape the Contour's Envoy instances. You can also enable Slack or MS Teams notifications, see the Kustomize install [docs](https://docs.flagger.app/install/flagger-install-on-kubernetes#install-flagger-with-kustomize).
The above command will deploy Flagger and Prometheus configured to scrape the Contour's Envoy instances.
Or you can install Flagger using Helm:
@@ -32,15 +32,17 @@ helm repo add flagger https://flagger.app
helm upgrade -i flagger flagger/flagger \
--namespace projectcontour \
--set meshProvider=contour \
--set prometheus.install=true \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \
--set slack.user=flagger
--set prometheus.install=true
```
You can also enable Slack, Discord, Rocket or MS Teams notifications,
see the alerting [docs](../usage/alerting.md).
## Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\), then creates a series of objects \(Kubernetes deployments, ClusterIP services and Contour HTTPProxy\). These objects expose the application in the cluster and drive the canary analysis and promotion.
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services and Contour HTTPProxy).
These objects expose the application in the cluster and drive the canary analysis and promotion.
Create a test namespace:
@@ -60,7 +62,7 @@ Create a deployment and a horizontal pod autoscaler:
kubectl apply -k github.com/weaveworks/flagger//kustomize/podinfo
```
Create a canary custom resource \(replace `app.example.com` with your own domain\):
Create a canary custom resource (replace `app.example.com` with your own domain):
```yaml
apiVersion: flagger.app/v1beta1
@@ -155,7 +157,9 @@ service/podinfo-primary
httpproxy.projectcontour.io/podinfo
```
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test` will be routed to the primary pods. During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test`
will be routed to the primary pods.
During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
## Expose the app outside the cluster
@@ -167,9 +171,11 @@ export ADDRESS="$(kubectl -n projectcontour get svc/envoy -ojson \
echo $ADDRESS
```
Configure your DNS server with a CNAME record \(AWS\) or A record \(GKE/AKS/DOKS\) and point a domain e.g. `app.example.com` to the LB address.
Configure your DNS server with a CNAME record \(AWS\) or A record (GKE/AKS/DOKS)
and point a domain e.g. `app.example.com` to the LB address.
Create a HTTPProxy definition and include the podinfo proxy generated by Flagger \(replace `app.example.com` with your own domain\):
Create a HTTPProxy definition and include the podinfo proxy generated by Flagger
(replace `app.example.com` with your own domain):
```yaml
apiVersion: projectcontour.io/v1
@@ -205,17 +211,21 @@ podinfo-ingress app.example.com valid
Now you can access podinfo UI using your domain address.
Note that you should be using HTTPS when exposing production workloads on internet. You can obtain free TLS certs from Let's Encrypt, read this [guide](https://github.com/stefanprodan/eks-contour-ingress) on how to configure cert-manager to secure Contour with TLS certificates.
Note that you should be using HTTPS when exposing production workloads on internet.
You can obtain free TLS certs from Let's Encrypt, read this [guide](https://github.com/stefanprodan/eks-contour-ingress)
on how to configure cert-manager to secure Contour with TLS certificates.
## Automated canary promotion
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring
key performance indicators like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
A canary deployment is triggered by changes in any of the following objects:
* Deployment PodSpec \(container image, command, ports, env, resources, etc\)
* Deployment PodSpec (container image, command, ports, env, resources, etc)
* ConfigMaps and Secrets mounted as volumes or mapped to environment variables
Trigger a canary deployment by updating the container image:
@@ -300,7 +310,8 @@ Generate latency:
watch -n 1 curl http://app.example.com/delay/1
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```text
kubectl -n projectcontour logs deploy/flagger -f | jq .msg
@@ -319,13 +330,16 @@ Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
If you’ve enabled the Slack notifications, you’ll receive a message if the progress deadline is exceeded, or if the analysis reached the maximum number of failed checks:
If you’ve enabled the Slack notifications, you’ll receive a message if the progress deadline is exceeded,
or if the analysis reached the maximum number of failed checks:
![Flagger Slack Notifications](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/screens/slack-canary-failed.png)
## A/B Testing
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users. This is particularly useful for frontend applications that require session affinity.
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
This is particularly useful for frontend applications that require session affinity.
![Flagger A/B Testing Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-abtest-steps.png)
@@ -421,3 +435,4 @@ match:
suffix: "Firefox/71.0"
```
For an in-depth look at the analysis process read the [usage docs](../usage/how-it-works.md).
docs/gitbook/tutorials/crossover-progressive-delivery.md
+15 -6
View File
@@ -64,7 +64,10 @@ helm upgrade -i flagger flagger/flagger \
## Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\), then creates a series of objects \(Kubernetes deployments, ClusterIP services, SMI traffic splits\). These objects expose the application on the mesh and drive the canary analysis and promotion. There's no SMI object you need to create by yourself.
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services, SMI traffic splits).
These objects expose the application on the mesh and drive the canary analysis and promotion.
There's no SMI object you need to create by yourself.
Create a deployment and a horizontal pod autoscaler:
@@ -171,17 +174,21 @@ service/podinfo-primary
trafficsplits.split.smi-spec.io/podinfo
```
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test` will be routed to the primary pods. During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test`
will be routed to the primary pods. During the canary analysis,
the `podinfo-canary.test` address can be used to target directly the canary pods.
## Automated canary promotion
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring
key performance indicators like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
A canary deployment is triggered by changes in any of the following objects:
* Deployment PodSpec \(container image, command, ports, env, resources, etc\)
* Deployment PodSpec (container image, command, ports, env, resources, etc)
* ConfigMaps and Secrets mounted as volumes or mapped to environment variables
Trigger a canary deployment by updating the container image:
@@ -288,7 +295,8 @@ Generate latency:
watch -n 1 curl -H 'Host: podinfo.test' http://envoy.test:10000/delay/1
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```text
kubectl -n test logs deploy/flagger -f | jq .msg
@@ -307,7 +315,8 @@ Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
If you’ve enabled the Slack notifications, you’ll receive a message if the progress deadline is exceeded, or if the analysis reached the maximum number of failed checks:
If you’ve enabled the Slack notifications, you’ll receive a message if the progress deadline is exceeded,
or if the analysis reached the maximum number of failed checks:
![Flagger Slack Notifications](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/screens/slack-canary-failed.png)
docs/gitbook/tutorials/gloo-progressive-delivery.md
+58 -40
View File
@@ -28,20 +28,11 @@ helm upgrade -i flagger flagger/flagger \
--set meshProvider=gloo
```
Optionally you can enable Slack notifications:
```bash
helm upgrade -i flagger flagger/flagger \
--reuse-values \
--namespace gloo-system \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=general \
--set slack.user=flagger
```
## Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\), then creates a series of objects \(Kubernetes deployments, ClusterIP services and Gloo upstream groups\). These objects expose the application outside the cluster and drive the canary analysis and promotion.
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services and Gloo upstream groups).
These objects expose the application outside the cluster and drive the canary analysis and promotion.
Create a test namespace:
@@ -61,7 +52,8 @@ Deploy the load testing service to generate traffic during the canary analysis:
kubectl -n test apply -k github.com/weaveworks/flagger//kustomize/tester
```
Create an virtual service definition that references an upstream group that will be generated by Flagger \(replace `app.example.com` with your own domain\):
Create an virtual service definition that references an upstream group that will be generated by Flagger
(replace `app.example.com` with your own domain):
```yaml
apiVersion: gateway.solo.io/v1
@@ -88,7 +80,7 @@ Save the above resource as podinfo-virtualservice.yaml and then apply it:
kubectl apply -f ./podinfo-virtualservice.yaml
```
Create a canary custom resource \(replace `app.example.com` with your own domain\):
Create a canary custom resource (replace `app.example.com` with your own domain):
```yaml
apiVersion: flagger.app/v1beta1
@@ -188,7 +180,9 @@ podinfo Initialized 0 2019-05-17T08:09:51Z
## Automated canary promotion
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring
key performance indicators like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
@@ -265,7 +259,8 @@ Generate high latency:
watch curl -H 'Host: app.example.com' http://gateway-proxy-v2.gloo-system/delay/2
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```text
kubectl -n test describe canary/podinfo
@@ -294,37 +289,57 @@ Events:
The canary analysis can be extended with Prometheus queries.
The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary.
The demo app is instrumented with Prometheus so you can create a custom check that will use
the HTTP request duration histogram to validate the canary.
Create a metric template and apply it on the cluster:
```yaml
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: not-found-percentage
namespace: test
spec:
provider:
type: prometheus
address: http://flagger-promethues.gloo-system:9090
query: |
100 - sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="{{ namespace }}",
kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
status!="{{ interval }}"
}[1m]
)
)
/
sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="{{ namespace }}",
kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[{{ interval }}]
)
) * 100
```
Edit the canary analysis and add the following metric:
```yaml
analysis:
metrics:
- name: "404s percentage"
threshold: 5
query: |
100 - sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
status!="404"
}[1m]
)
)
/
sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[1m]
)
) * 100
- name: "404s percentage"
templateRef:
name: not-found-percentage
thresholdRange:
max: 5
interval: 1m
```
The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.
The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is
below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.
Trigger a canary deployment by updating the container image:
@@ -357,5 +372,8 @@ Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
If you have Slack configured, Flagger will send a notification with the reason why the canary failed.
If you have [alerting](../usage/alerting.md) configured,
Flagger will send a notification with the reason why the canary failed.
For an in-depth look at the analysis process read the [usage docs](../usage/how-it-works.md).
docs/gitbook/tutorials/istio-ab-testing.md
+4
View File
@@ -213,3 +213,7 @@ Events:
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
```
The above procedure can be extended with [custom metrics](../usage/metrics.md) checks,
[webhooks](../usage/webhooks.md),
[manual promotion](../usage/webhooks.md#manual-gating) approval and
[Slack or MS Teams](../usage/alerting.md) notifications.
docs/gitbook/tutorials/istio-progressive-delivery.md
+4 -2
View File
@@ -319,5 +319,7 @@ With the above configuration, Flagger will run a canary release with the followi
* scale to zero the canary
* send notification with the canary analysis result
The above procedure can be extended with [custom metrics](https://docs.flagger.app/how-it-works#custom-metrics) checks, [webhooks](https://docs.flagger.app/how-it-works#webhooks), [manual promotion](https://docs.flagger.app/how-it-works#manual-gating) approval and [Slack or MS Teams](https://docs.flagger.app/usage/alerting) notifications.
The above procedure can be extended with [custom metrics](../usage/metrics.md) checks,
[webhooks](../usage/webhooks.md),
[manual promotion](../usage/webhooks.md#manual-gating) approval and
[Slack or MS Teams](../usage/alerting.md) notifications.
docs/gitbook/tutorials/kubernetes-blue-green.md
+60 -29
View File
@@ -2,7 +2,9 @@
This guide shows you how to automate Blue/Green deployments with Flagger and Kubernetes.
For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments with Kubernetes L4 networking. When using a service mesh blue/green can be used as specified [here](https://docs.flagger.app/how-it-works#blue-green-deployments).
For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments
with Kubernetes L4 networking. When using a service mesh blue/green can be used as
specified [here](../usage/deployment-strategies.md).
![Flagger Blue/Green Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-bluegreen-steps.png)
@@ -42,7 +44,9 @@ helm upgrade -i flagger flagger/flagger \
## Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\), then creates a series of objects \(Kubernetes deployment and ClusterIP services\). These objects expose the application inside the cluster and drive the canary analysis and Blue/Green promotion.
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployment and ClusterIP services).
These objects expose the application inside the cluster and drive the canary analysis and Blue/Green promotion.
Create a test namespace:
@@ -156,8 +160,10 @@ service/podinfo-primary
Blue/Green scenario:
* on bootstrap, Flagger will create three ClusterIP services \(`app-primary`,`app-canary`, `app`\) and a shadow deployment named `app-primary` that represents the blue version
* when a new version is detected, Flagger would scale up the green version and run the conformance tests \(the tests should target the `app-canary` ClusterIP service to reach the green version\)
* on bootstrap, Flagger will create three ClusterIP services (`app-primary`,`app-canary`, `app`)
and a shadow deployment named `app-primary` that represents the blue version
* when a new version is detected, Flagger would scale up the green version and run the conformance tests
(the tests should target the `app-canary` ClusterIP service to reach the green version)
* if the conformance tests are passing, Flagger would start the load tests and validate them with custom Prometheus queries
* if the load test analysis is successful, Flagger will promote the new version to `app-primary` and scale down the green version
@@ -253,37 +259,58 @@ Events:
## Custom metrics
The analysis can be extended with Prometheus queries. The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary \(green version\).
The analysis can be extended with Prometheus queries. The demo app is instrumented with Prometheus so you can
create a custom check that will use the HTTP request duration histogram to validate the canary (green version).
Create a metric template and apply it on the cluster:
```yaml
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: not-found-percentage
namespace: test
spec:
provider:
type: prometheus
address: http://flagger-promethues.flagger:9090
query: |
100 - sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="{{ namespace }}",
kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
status!="{{ interval }}"
}[1m]
)
)
/
sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="{{ namespace }}",
kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[{{ interval }}]
)
) * 100
```
Edit the canary analysis and add the following metric:
```yaml
analysis:
metrics:
- name: "404s percentage"
threshold: 5
query: |
100 - sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
status!="404"
}[1m]
)
)
/
sum(
rate(
http_request_duration_seconds_count{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[1m]
)
) * 100
- name: "404s percentage"
templateRef:
name: not-found-percentage
thresholdRange:
max: 5
interval: 1m
```
The above configuration validates the canary \(green version\) by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the rollout is rolled back.
The above configuration validates the canary (green version) by checking if the HTTP 404 req/sec percentage is
below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the rollout is rolled back.
Trigger a deployment by updating the container image:
@@ -312,7 +339,8 @@ Rolling back podinfo.test failed checks threshold reached 2
Canary failed! Scaling down podinfo.test
```
If you have Slack configured, Flagger will send a notification with the reason why the canary failed.
If you have [alerting](../usage/alerting.md) configured,
Flagger will send a notification with the reason why the canary failed.
## Conformance Testing with Helm
@@ -344,5 +372,8 @@ Add a helm test pre-rollout hook to your chart:
cmd: "test {{ .Release.Name }} --cleanup"
```
When the canary analysis starts, Flagger will call the pre-rollout webhooks. If the helm test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.
When the canary analysis starts, Flagger will call the pre-rollout webhooks.
If the helm test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.
For an in-depth look at the analysis process read the [usage docs](../usage/how-it-works.md).
docs/gitbook/tutorials/linkerd-progressive-delivery.md
+34 -14
View File
@@ -16,11 +16,11 @@ kubectl apply -k github.com/weaveworks/flagger//kustomize/linkerd
Note that you'll need kubectl 1.14 or newer to run the above command.
To enable Slack or MS Teams notifications, see Flagger's [install docs](https://docs.flagger.app/install/flagger-install-on-kubernetes) for Kustomize or Helm options.
## Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\), then creates a series of objects \(Kubernetes deployments, ClusterIP services and SMI traffic split\). These objects expose the application inside the mesh and drive the canary analysis and promotion.
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services and SMI traffic split).
These objects expose the application inside the mesh and drive the canary analysis and promotion.
Create a test namespace and enable Linkerd proxy injection:
@@ -113,7 +113,8 @@ Save the above resource as podinfo-canary.yaml and then apply it:
kubectl apply -f ./podinfo-canary.yaml
```
When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary. The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every half a minute.
When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.
The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every half a minute.
After a couple of seconds Flagger will create the canary objects:
@@ -133,11 +134,15 @@ service/podinfo-primary
trafficsplits.split.smi-spec.io/podinfo
```
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test` will be routed to the primary pods. During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
After the boostrap, the podinfo deployment will be scaled to zero and the traffic to `podinfo.test`
will be routed to the primary pods.
During the canary analysis, the `podinfo-canary.test` address can be used to target directly the canary pods.
## Automated canary promotion
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring
key performance indicators like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
@@ -198,7 +203,8 @@ prod backend Failed 0 2019-06-30T17:05:07Z
## Automated rollback
During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses and rolls back the faulted version.
During the canary analysis you can generate HTTP 500 errors and high latency to
test if Flagger pauses and rolls back the faulted version.
Trigger another canary deployment:
@@ -225,7 +231,8 @@ Generate latency:
watch -n 1 curl http://podinfo-canary.test:9898/delay/1
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```text
kubectl -n test describe canary/podinfo
@@ -284,7 +291,9 @@ Let's a define a check for not found errors. Edit the canary analysis and add th
* 100
```
The above configuration validates the canary version by checking if the HTTP 404 req/sec percentage is below three percent of the total traffic. If the 404s rate reaches the 3% threshold, then the analysis is aborted and the canary is marked as failed.
The above configuration validates the canary version by checking if the HTTP 404 req/sec percentage
is below three percent of the total traffic.
If the 404s rate reaches the 3% threshold, then the analysis is aborted and the canary is marked as failed.
Trigger a canary deployment by updating the container image:
@@ -329,7 +338,8 @@ helm upgrade -i nginx-ingress stable/nginx-ingress \
--namespace ingress-nginx
```
Create an ingress definition for podinfo that rewrites the incoming header to the internal service name \(required by Linkerd\):
Create an ingress definition for podinfo that rewrites the incoming header
to the internal service name (required by Linkerd):
```yaml
apiVersion: extensions/v1beta1
@@ -355,15 +365,20 @@ spec:
servicePort: 9898
```
When using an ingress controller, the Linkerd traffic split does not apply to incoming traffic since NGINX in running outside of the mesh. In order to run a canary analysis for a frontend app, Flagger creates a shadow ingress and sets the NGINX specific annotations.
When using an ingress controller, the Linkerd traffic split does not apply to incoming traffic
since NGINX in running outside of the mesh. In order to run a canary analysis for a frontend app,
Flagger creates a shadow ingress and sets the NGINX specific annotations.
## A/B Testing
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users. This is particularly useful for frontend applications that require session affinity.
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
This is particularly useful for frontend applications that require session affinity.
![Flagger Linkerd Ingress](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-nginx-linkerd.png)
Edit podinfo canary analysis, set the provider to `nginx`, add the ingress reference, remove the max/step weight and add the match conditions and iterations:
Edit podinfo canary analysis, set the provider to `nginx`, add the ingress reference,
remove the max/step weight and add the match conditions and iterations:
```yaml
apiVersion: flagger.app/v1beta1
@@ -425,7 +440,8 @@ spec:
cmd: "hey -z 2m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com"
```
The above configuration will run an analysis for ten minutes targeting users that have a `canary` cookie set to `always` or those that call the service using the `X-Canary: always` header.
The above configuration will run an analysis for ten minutes targeting users that have
a `canary` cookie set to `always` or those that call the service using the `X-Canary: always` header.
**Note** that the load test now targets the external address and uses the canary cookie.
@@ -459,3 +475,7 @@ Events:
Promotion completed! Scaling down podinfo.test
```
The above procedure can be extended with [custom metrics](../usage/metrics.md) checks,
[webhooks](../usage/webhooks.md),
[manual promotion](../usage/webhooks.md#manual-gating) approval and
[Slack or MS Teams](../usage/alerting.md) notifications.
docs/gitbook/tutorials/nginx-progressive-delivery.md
+52 -21
View File
@@ -44,7 +44,9 @@ helm upgrade -i flagger flagger/flagger \
## Bootstrap
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\), then creates a series of objects \(Kubernetes deployments, ClusterIP services and canary ingress\). These objects expose the application outside the cluster and drive the canary analysis and promotion.
Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA),
then creates a series of objects (Kubernetes deployments, ClusterIP services and canary ingress).
These objects expose the application outside the cluster and drive the canary analysis and promotion.
Create a test namespace:
@@ -186,7 +188,9 @@ ingresses.extensions/podinfo-canary
## Automated canary promotion
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.
Flagger implements a control loop that gradually shifts traffic to the canary while measuring
key performance indicators like HTTP requests success rate, requests average duration and pod health.
Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack or MS Teams.
![Flagger Canary Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-canary-steps.png)
@@ -257,7 +261,8 @@ Generate HTTP 500 errors:
watch curl http://app.example.com/status/500
```
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
the canary is scaled to zero and the rollout is marked as failed.
```text
kubectl -n test describe canary/podinfo
@@ -286,30 +291,49 @@ Events:
The canary analysis can be extended with Prometheus queries.
The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary.
The demo app is instrumented with Prometheus so you can create a custom check that will use the
HTTP request duration histogram to validate the canary.
Edit the canary analysis and add the following metric:
Create a metric template and apply it on the cluster:
```yaml
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: latency
namespace: test
spec:
provider:
type: prometheus
address: http://flagger-promethues.ingress-nginx:9090
query: |
histogram_quantile(0.99,
sum(
rate(
http_request_duration_seconds_bucket{
kubernetes_namespace="{{ namespace }}",
kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[1m]
)
) by (le)
)
```
Edit the canary analysis and add the latency check:
```yaml
analysis:
metrics:
- name: "latency"
threshold: 0.5
templateRef:
name: latency
thresholdRange:
max: 0.5
interval: 1m
query: |
histogram_quantile(0.99,
sum(
rate(
http_request_duration_seconds_bucket{
kubernetes_namespace="test",
kubernetes_pod_name=~"podinfo-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
}[1m]
)
) by (le)
)
```
The threshold is set to 500ms so if the average request duration in the last minute goes over half a second then the analysis will fail and the canary will not be promoted.
The threshold is set to 500ms so if the average request duration in the last minute goes over half a second
then the analysis will fail and the canary will not be promoted.
Trigger a canary deployment by updating the container image:
@@ -342,11 +366,13 @@ Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
```
If you have Slack configured, Flagger will send a notification with the reason why the canary failed.
If you have alerting configured, Flagger will send a notification with the reason why the canary failed.
## A/B Testing
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users. This is particularly useful for frontend applications that require session affinity.
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions.
In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users.
This is particularly useful for frontend applications that require session affinity.
![Flagger A/B Testing Stages](https://raw.githubusercontent.com/weaveworks/flagger/master/docs/diagrams/flagger-abtest-steps.png)
@@ -378,7 +404,8 @@ Edit the canary analysis, remove the max/step weight and add the match condition
cmd: "hey -z 1m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com/"
```
The above configuration will run an analysis for ten minutes targeting users that have a `canary` cookie set to `always` or those that call the service using the `X-Canary: insider` header.
The above configuration will run an analysis for ten minutes targeting users that have a `canary` cookie
set to `always` or those that call the service using the `X-Canary: insider` header.
Trigger a canary deployment by updating the container image:
@@ -416,3 +443,7 @@ Events:
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
```
The above procedure can be extended with [custom metrics](../usage/metrics.md) checks,
[webhooks](../usage/webhooks.md),
[manual promotion](../usage/webhooks.md#manual-gating) approval and
[Slack or MS Teams](../usage/alerting.md) notifications.
docs/gitbook/how-it-works.md → docs/gitbook/usage/how-it-works.md
+5 -5
View File
@@ -168,7 +168,7 @@ spec:
When using **Istio** as the mesh provider, you can also specify
HTTP header operations, CORS and traffic policies, Istio gateways and hosts.
The Istio routing configuration can be found [here](faq.md#istio-routing).
The Istio routing configuration can be found [here](../faq.md#istio-routing).
### Canary status
@@ -241,10 +241,10 @@ kubectl get canary/podinfo | grep Succeeded
### Canary analysis
The canary analysis defines:
* the type of [deployment strategy](usage/deployment-strategies.md)
* the [metrics](usage/metrics.md) used to validate the canary version
* the [webhooks](usage/webhooks.md) used for conformance testing, load testing and manual gating
* the [alerting settings](usage/alerting.md)
* the type of [deployment strategy](deployment-strategies.md)
* the [metrics](metrics.md) used to validate the canary version
* the [webhooks](webhooks.md) used for conformance testing, load testing and manual gating
* the [alerting settings](alerting.md)
Spec: