Document the CRD target ref and control loop interval

This commit is contained in:
stefanprodan
2019-01-05 10:22:00 +02:00
parent b67b49fde6
commit 41f0ba0247
8 changed files with 60 additions and 25 deletions
View File
+2 -2
View File
@@ -5,8 +5,8 @@
## Install
* [Install Flagger](install/installing-flagger.md)
* [Install Grafana](install/installing-grafana.md)
* [Install Flagger](install/install-flagger.md)
* [Install Grafana](install/install-grafana.md)
* [Install Istio](install/install-istio.md)
## Usage
+31 -9
View File
@@ -2,7 +2,7 @@
[Flagger](https://github.com/stefanprodan/flagger) takes a Kubernetes deployment and optionally a horizontal pod autoscaler \(HPA\) and creates a series of objects \(Kubernetes deployments, ClusterIP services and Istio virtual services\) to drive the canary analysis and promotion.
![flagger-canary-hpa](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/diagrams/flagger-canary-hpa.png)
![Flagger Canary Process](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/diagrams/flagger-canary-hpa.png)
### Canary Custom Resource
@@ -36,7 +36,7 @@ spec:
- public-gateway.istio-system.svc.cluster.local
# Istio virtual service host names (optional)
hosts:
- app.iowa.weavedx.com
- podinfo.example.com
canaryAnalysis:
# max number of failed metric checks before rollback
threshold: 10
@@ -63,14 +63,35 @@ spec:
- name: integration-tests
url: http://podinfo.test:9898/echo
timeout: 1m
# key-value pairs (optional)
metadata:
test: "all"
token: "16688eb5e9f289f1991c"
```
**Note** that the target deployment must have a single label selector in the format `app: <DEPLOYMENT-NAME>`:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
spec:
selector:
matchLabels:
app: podinfo
template:
metadata:
labels:
app: podinfo
```
The target deployment should expose a TCP port that will be used by Flagger to create the ClusterIP Service and
the Istio Virtual Service. The container port from the target deployment should match the `service.port` value.
### Canary Deployment
![flagger-canary-steps](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/diagrams/flagger-canary-steps.png)
![Flagger Canary Stages](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/diagrams/flagger-canary-steps.png)
Gated canary promotion stages:
@@ -99,7 +120,7 @@ Gated canary promotion stages:
* halt advancement if pods are unhealthy
* route all traffic to primary
* scale to zero the canary deployment
* mark rollout as finished
* mark the canary deployment as finished
* wait for the canary deployment to be updated \(revision bump\) and start over
You can change the canary analysis _max weight_ and the _step weight_ percentage in the Flagger's custom resource.
@@ -120,9 +141,10 @@ Spec:
maxWeight: 50
# canary increment step
# percentage (0-100)
stepWeight: 5
stepWeight: 2
```
The above analysis, if it succeeds, will run for 25 minutes while validating the HTTP metrics and webhooks every minute.
You can determine the minimum time that it takes to validate and promote a canary deployment using this formula:
```
@@ -208,12 +230,12 @@ histogram_quantile(0.99,
)
```
**Note** that the metric interval should be lower or equal to the control loop interval.
> **Note** that the metric interval should be lower or equal to the control loop interval.
### Webhooks
The canary analysis can be extended with webhooks.
Flagger would call each webhook URL and determine from the response status code (HTTP 2xx) if the canary is failing or not.
Flagger will call each webhook URL and determine from the response status code (HTTP 2xx) if the canary is failing or not.
Spec:
@@ -222,7 +244,7 @@ Spec:
webhooks:
- name: integration-tests
url: http://podinfo.test:9898/echo
timeout: 1m
timeout: 30s
metadata:
test: "all"
token: "16688eb5e9f289f1991c"
@@ -234,7 +256,7 @@ Spec:
key2: "val2"
```
**Note** that the sum of all webhooks timeouts should be lower than the control loop interval.
> **Note** that the sum of all webhooks timeouts should be lower than the control loop interval.
Webhook payload (HTTP POST):
@@ -1,6 +1,8 @@
# Install Flagger
Before installing Flagger make sure you have [Istio](https://istio.io) running with Prometheus enabled. If you are new to Istio you can follow this GKE guide [Istio service mesh walk-through](https://docs.flagger.app/install/install-istio).
Before installing Flagger make sure you have [Istio](https://istio.io) running with Prometheus enabled.
If you are new to Istio you can follow this GKE guide
[Istio service mesh walk-through](https://docs.flagger.app/install/install-istio).
**Prerequisites**
@@ -61,11 +63,12 @@ helm delete --purge flagger
The command removes all the Kubernetes components associated with the chart and deletes the release.
{% hint style="info" %}
On uninstall the Flagger CRD will not be removed. Deleting the CRD will make Kubernetes remove all the objects owned by the CRD like Istio virtual services, Kubernetes deployments and ClusterIP services.
{% endhint %}
> **Note** that on uninstall the Canary CRD will not be removed.
Deleting the CRD will make Kubernetes remove all the objects owned by Flagger like Istio virtual services,
Kubernetes deployments and ClusterIP services.
If you want to remove all the objects created by Flagger you have delete the canary CRD with kubectl:
If you want to remove all the objects created by Flagger you have delete the Canary CRD with kubectl:
```text
kubectl delete crd canaries.flagger.app
+13 -5
View File
@@ -1,12 +1,14 @@
# Install Istio
This guide walks you through setting up Istio with Jaeger, Prometheus, Grafana and Lets Encrypt TLS for ingress gateway on Google Kubernetes Engine.
This guide walks you through setting up Istio with Jaeger, Prometheus, Grafana and
Lets Encrypt TLS for ingress gateway on Google Kubernetes Engine.
![Istio GKE diagram](https://raw.githubusercontent.com/stefanprodan/istio-gke/master/docs/screens/istio-gcp-overview.png)
### Prerequisites
You will be creating a cluster on Googles Kubernetes Engine \(GKE\), if you dont have an account you can sign up [here](https://cloud.google.com/free/) for free credits.
You will be creating a cluster on Googles Kubernetes Engine \(GKE\),
if you dont have an account you can sign up [here](https://cloud.google.com/free/) for free credits.
Login into GCP, create a project and enable billing for it.
@@ -64,7 +66,9 @@ gcloud container clusters create istio \
--scopes=gke-default,compute-rw,storage-rw
```
The above command will create a default node pool consisting of `n1-highcpu-4` \(vCPU: 4, RAM 3.60GB, DISK: 30GB\) preemptible VMs. Preemptible VMs are up to 80% cheaper than regular instances and are terminated and replaced after a maximum of 24 hours.
The above command will create a default node pool consisting of `n1-highcpu-4` \(vCPU: 4, RAM 3.60GB, DISK: 30GB\)
preemptible VMs. Preemptible VMs are up to 80% cheaper than regular instances and are terminated and replaced
after a maximum of 24 hours.
Set up credentials for `kubectl`:
@@ -330,7 +334,8 @@ kubectl create secret generic cert-manager-credentials \
--namespace=istio-system
```
Create a letsencrypt issuer for CloudDNS \(replace `email@example.com` with a valid email address and `my-gcp-project`with your project ID\):
Create a letsencrypt issuer for CloudDNS \(replace `email@example.com` with a valid email address and
`my-gcp-project`with your project ID\):
```yaml
apiVersion: v1alpha2
@@ -403,7 +408,10 @@ Recreate Istio ingress gateway pods:
kubectl -n istio-system delete pods -l istio=ingressgateway
```
Note that Istio gateway doesn't reload the certificates from the TLS secret on cert-manager renewal. Since the GKE cluster is made out of preemptible VMs the gateway pods will be replaced once every 24h, if your not using preemptible nodes then you need to manually kill the gateway pods every two months before the certificate expires.
Note that Istio gateway doesn't reload the certificates from the TLS secret on cert-manager renewal.
Since the GKE cluster is made out of preemptible VMs the gateway pods will be replaced once every 24h,
if your not using preemptible nodes then you need to manually kill the gateway pods every two months
before the certificate expires.
### Expose services outside the service mesh
+4 -2
View File
@@ -12,11 +12,13 @@ helm upgrade -i flagger flagger/flagger \
--set slack.user=flagger
```
Once configured with a Slack incoming **webhook**, Flagger will post messages when a canary deployment has been initialised, when a new revision has been detected and if the canary analysis failed or succeeded.
Once configured with a Slack incoming **webhook**, Flagger will post messages when a canary deployment
has been initialised, when a new revision has been detected and if the canary analysis failed or succeeded.
![flagger-slack](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/screens/slack-canary-notifications.png)
A canary deployment will be rolled back if the progress deadline exceeded or if the analysis reached the maximum number of failed checks:
A canary deployment will be rolled back if the progress deadline exceeded or if the analysis reached the
maximum number of failed checks:
![flagger-slack-errors](https://raw.githubusercontent.com/stefanprodan/flagger/master/docs/screens/slack-canary-failed.png)
+2 -2
View File
@@ -44,7 +44,8 @@ Promotion completed! podinfo.test
### Metrics
Flagger exposes Prometheus metrics that can be used to determine the canary analysis status and the destination weight values:
Flagger exposes Prometheus metrics that can be used to determine the canary analysis status and
the destination weight values:
```bash
# Canaries total gauge
@@ -65,5 +66,4 @@ flagger_canary_duration_seconds_sum{name="podinfo",namespace="test"} 17.3561329
flagger_canary_duration_seconds_count{name="podinfo",namespace="test"} 6
```
####