|
|
|
@@ -0,0 +1,206 @@
|
|
|
|
|
# App Mesh Canary Deployments
|
|
|
|
|
|
|
|
|
|
This guide shows you how to use App Mesh and Flagger to automate canary deployments.
|
|
|
|
|
|
|
|
|
|
Create a mesh called `global` in the `appmesh-system` namespace:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
export REPO=https://raw.githubusercontent.com/weaveworks/flagger/master
|
|
|
|
|
|
|
|
|
|
kubectl apply -f ${REPO}/artifacts/appmesh/global-mesh.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Create a test namespace with App Mesh sidecar injection enabled:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl apply -f ${REPO}/artifacts/namespaces/test.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Create a deployment and a horizontal pod autoscaler:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl apply -f ${REPO}/artifacts/appmesh/deployment.yaml
|
|
|
|
|
kubectl apply -f ${REPO}/artifacts/appmesh/hpa.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Deploy the load testing service to generate traffic during the canary analysis:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
helm upgrade -i flagger-loadtester flagger/loadtester \
|
|
|
|
|
--namespace=test \
|
|
|
|
|
--set meshName=global.appmesh-system \
|
|
|
|
|
--set backends[0]=frontend.test
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Create a canary custom resource:
|
|
|
|
|
|
|
|
|
|
```yaml
|
|
|
|
|
apiVersion: flagger.app/v1alpha3
|
|
|
|
|
kind: Canary
|
|
|
|
|
metadata:
|
|
|
|
|
name: podinfo
|
|
|
|
|
namespace: test
|
|
|
|
|
spec:
|
|
|
|
|
# deployment reference
|
|
|
|
|
targetRef:
|
|
|
|
|
apiVersion: apps/v1
|
|
|
|
|
kind: Deployment
|
|
|
|
|
name: podinfo
|
|
|
|
|
# the maximum time in seconds for the canary deployment
|
|
|
|
|
# to make progress before it is rollback (default 600s)
|
|
|
|
|
progressDeadlineSeconds: 60
|
|
|
|
|
# HPA reference (optional)
|
|
|
|
|
autoscalerRef:
|
|
|
|
|
apiVersion: autoscaling/v2beta1
|
|
|
|
|
kind: HorizontalPodAutoscaler
|
|
|
|
|
name: podinfo
|
|
|
|
|
service:
|
|
|
|
|
# container port
|
|
|
|
|
port: 9898
|
|
|
|
|
# App Mesh reference
|
|
|
|
|
meshName: global.appmesh-system
|
|
|
|
|
# define the canary analysis timing and KPIs
|
|
|
|
|
canaryAnalysis:
|
|
|
|
|
# schedule interval (default 60s)
|
|
|
|
|
interval: 10s
|
|
|
|
|
# max number of failed metric checks before rollback
|
|
|
|
|
threshold: 10
|
|
|
|
|
# max traffic percentage routed to canary
|
|
|
|
|
# percentage (0-100)
|
|
|
|
|
maxWeight: 50
|
|
|
|
|
# canary increment step
|
|
|
|
|
# percentage (0-100)
|
|
|
|
|
stepWeight: 5
|
|
|
|
|
# App Mesh Prometheus checks
|
|
|
|
|
metrics:
|
|
|
|
|
- name: envoy_cluster_upstream_rq
|
|
|
|
|
# minimum req success rate (non 5xx responses)
|
|
|
|
|
# percentage (0-100)
|
|
|
|
|
threshold: 99
|
|
|
|
|
interval: 1m
|
|
|
|
|
# external checks (optional)
|
|
|
|
|
webhooks:
|
|
|
|
|
- name: load-test
|
|
|
|
|
url: http://flagger-loadtester.test/
|
|
|
|
|
timeout: 5s
|
|
|
|
|
metadata:
|
|
|
|
|
cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Save the above resource as podinfo-canary.yaml and then apply it:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl apply -f ./podinfo-canary.yaml
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
After a couple of seconds Flagger will create the canary objects:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# applied
|
|
|
|
|
deployment.apps/podinfo
|
|
|
|
|
horizontalpodautoscaler.autoscaling/podinfo
|
|
|
|
|
canary.flagger.app/podinfo
|
|
|
|
|
|
|
|
|
|
# generated Kubernetes objects
|
|
|
|
|
deployment.apps/podinfo-primary
|
|
|
|
|
horizontalpodautoscaler.autoscaling/podinfo-primary
|
|
|
|
|
service/podinfo
|
|
|
|
|
service/podinfo-canary
|
|
|
|
|
service/podinfo-primary
|
|
|
|
|
|
|
|
|
|
# generated App Mesh objects
|
|
|
|
|
virtualnode.appmesh.k8s.aws/podinfo
|
|
|
|
|
virtualnode.appmesh.k8s.aws/podinfo-canary
|
|
|
|
|
virtualnode.appmesh.k8s.aws/podinfo-primary
|
|
|
|
|
virtualservice.appmesh.k8s.aws/podinfo.test
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Trigger a canary deployment by updating the container image:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl -n test set image deployment/podinfo \
|
|
|
|
|
podinfod=quay.io/stefanprodan/podinfo:1.4.1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Flagger detects that the deployment revision changed and starts a new rollout:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
kubectl -n test describe canary/podinfo
|
|
|
|
|
|
|
|
|
|
Status:
|
|
|
|
|
Canary Weight: 0
|
|
|
|
|
Failed Checks: 0
|
|
|
|
|
Phase: Succeeded
|
|
|
|
|
Events:
|
|
|
|
|
Type Reason Age From Message
|
|
|
|
|
---- ------ ---- ---- -------
|
|
|
|
|
Normal Synced 3m flagger New revision detected podinfo.test
|
|
|
|
|
Normal Synced 3m flagger Scaling up podinfo.test
|
|
|
|
|
Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
|
|
|
|
|
Normal Synced 3m flagger Advance podinfo.test canary weight 5
|
|
|
|
|
Normal Synced 3m flagger Advance podinfo.test canary weight 10
|
|
|
|
|
Normal Synced 3m flagger Advance podinfo.test canary weight 15
|
|
|
|
|
Normal Synced 2m flagger Advance podinfo.test canary weight 20
|
|
|
|
|
Normal Synced 2m flagger Advance podinfo.test canary weight 25
|
|
|
|
|
Normal Synced 1m flagger Advance podinfo.test canary weight 30
|
|
|
|
|
Normal Synced 1m flagger Advance podinfo.test canary weight 35
|
|
|
|
|
Normal Synced 55s flagger Advance podinfo.test canary weight 40
|
|
|
|
|
Normal Synced 45s flagger Advance podinfo.test canary weight 45
|
|
|
|
|
Normal Synced 35s flagger Advance podinfo.test canary weight 50
|
|
|
|
|
Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
|
|
|
|
|
Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
|
|
|
|
|
Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Note** that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.
|
|
|
|
|
|
|
|
|
|
You can monitor all canaries with:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
watch kubectl get canaries --all-namespaces
|
|
|
|
|
|
|
|
|
|
NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
|
|
|
|
|
test podinfo Progressing 15 2019-03-16T14:05:07Z
|
|
|
|
|
prod frontend Succeeded 0 2019-03-15T16:15:07Z
|
|
|
|
|
prod backend Failed 0 2019-03-14T17:05:07Z
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
During the canary analysis you can generate HTTP 500 errors to test if Flagger pauses the rollout.
|
|
|
|
|
|
|
|
|
|
Exec into the load tester pod with:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
kubectl -n test exec -it flagger-loadtester-xx-xx sh
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Generate HTTP 500 errors:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
hey -z 1m -c 5 -q 5 http://podinfo.test:9898/status/500
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary,
|
|
|
|
|
the canary is scaled to zero and the rollout is marked as failed.
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
kubectl -n test describe canary/podinfo
|
|
|
|
|
|
|
|
|
|
Status:
|
|
|
|
|
Canary Weight: 0
|
|
|
|
|
Failed Checks: 10
|
|
|
|
|
Phase: Failed
|
|
|
|
|
Events:
|
|
|
|
|
Type Reason Age From Message
|
|
|
|
|
---- ------ ---- ---- -------
|
|
|
|
|
Normal Synced 3m flagger Starting canary deployment for podinfo.test
|
|
|
|
|
Normal Synced 3m flagger Advance podinfo.test canary weight 5
|
|
|
|
|
Normal Synced 3m flagger Advance podinfo.test canary weight 10
|
|
|
|
|
Normal Synced 3m flagger Advance podinfo.test canary weight 15
|
|
|
|
|
Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
|
|
|
|
|
Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
|
|
|
|
|
Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% < 99%
|
|
|
|
|
Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% < 99%
|
|
|
|
|
Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% < 99%
|
|
|
|
|
Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
|
|
|
|
|
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
|
|
|
|
|
```
|
stefanprodan