Merge pull request #232 from weaveworks/kustomize

Add Kustomize installer
This commit is contained in:
Stefan Prodan
2019-07-05 16:23:44 +03:00
committed by GitHub
21 changed files with 855 additions and 35 deletions
+2 -1
View File
@@ -71,7 +71,8 @@ version-set:
sed -i '' "s/tag: $$current/tag: $$next/g" charts/flagger/values.yaml && \
sed -i '' "s/appVersion: $$current/appVersion: $$next/g" charts/flagger/Chart.yaml && \
sed -i '' "s/version: $$current/version: $$next/g" charts/flagger/Chart.yaml && \
echo "Version $$next set in code, deployment and charts"
sed -i '' "s/newTag: $$current/newTag: $$next/g" kustomize/base/flagger/kustomization.yaml && \
echo "Version $$next set in code, deployment, chart and kustomize"
version-up:
@next="$(VERSION_MINOR).$$(($(PATCH) + 1))" && \
@@ -1,6 +1,6 @@
# Flagger install on Kubernetes
This guide walks you through setting up Flagger on a Kubernetes cluster.
This guide walks you through setting up Flagger on a Kubernetes cluster with Helm or Kustomize.
### Prerequisites
@@ -9,18 +9,7 @@ Flagger requires a Kubernetes cluster **v1.11** or newer with the following admi
* MutatingAdmissionWebhook
* ValidatingAdmissionWebhook
Flagger depends on [Istio](https://istio.io/docs/setup/kubernetes/quick-start/) **v1.0.3** or newer
with traffic management, telemetry and Prometheus enabled.
A minimal Istio installation should contain the following services:
* istio-pilot
* istio-ingressgateway
* istio-sidecar-injector
* istio-telemetry
* prometheus
### Install Flagger
### Install Flagger with Helm
Add Flagger Helm repository:
@@ -28,15 +17,25 @@ Add Flagger Helm repository:
helm repo add flagger https://flagger.app
```
Deploy Flagger in the _**istio-system**_ namespace:
Deploy Flagger for Istio:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=istio-system \
--set metricsServer=http://prometheus.istio-system:9090
--set meshProvider=istio \
--set metricsServer=http://prometheus:9090
```
You can install Flagger in any namespace as long as it can talk to the Istio Prometheus service on port 9090.
Deploy Flagger for Linkerd:
```bash
helm upgrade -i flagger flagger/flagger \
--namespace=linkerd \
--set meshProvider=linkerd \
--set metricsServer=http://linkerd-prometheus:9090
```
You can install Flagger in any namespace as long as it can talk to the Prometheus service on port 9090.
Enable **Slack** notifications:
@@ -81,7 +80,7 @@ If you want to remove all the objects created by Flagger you have delete the Can
kubectl delete crd canaries.flagger.app
```
### Install Grafana
### Install Grafana with Helm
Flagger comes with a Grafana dashboard made for monitoring the canary analysis.
@@ -115,31 +114,94 @@ You can access Grafana using port forwarding:
kubectl -n istio-system port-forward svc/flagger-grafana 3000:80
```
### Install Load Tester
### Install Flagger with Kustomize
Flagger comes with an optional load testing service that generates traffic
during canary analysis when configured as a webhook.
As an alternative to Helm, Flagger can be installed with Kustomize.
Deploy the load test runner with Helm:
**Service mesh specific installers**
Install Flagger for Istio:
```bash
helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test \
--set cmd.timeout=1h
kubectl apply -k github.com/weaveworks/flagger/kustomize/istio
```
Deploy with kubectl:
This deploys Flagger in the `istio-system` namespace and sets the metrics server URL to `http://prometheus.istio-system:9090`.
Install Flagger for Linkerd:
```bash
helm fetch --untar --untardir . flagger/loadtester &&
helm template loadtester \
--name flagger-loadtester \
--namespace=test
> $HOME/flagger-loadtester.yaml
# apply
kubectl apply -f $HOME/flagger-loadtester.yaml
kubectl apply -k github.com/weaveworks/flagger/kustomize/linkerd
```
> **Note** that the load tester should be deployed in a namespace with Istio sidecar injection enabled.
This deploys Flagger in the `linkerd` namespace and sets the metrics server URL to `http://linkerd-prometheus.linkerd:9090`.
**Generic installer**
Install Flagger and Prometheus:
```bash
kubectl apply -k github.com/weaveworks/flagger/kustomize/kubernetes
```
This deploys Flagger and Prometheus in the `flagger-system` namespace,
sets the metrics server URL to `http://flagger-prometheus.flagger-system:9090` and the mesh provider to `kubernetes`.
The Prometheus instance has a two hours data retention and is configured to scrape all pods in your cluster that
have the `prometheus.io/scrape: "true"` annotation.
To target a specific provider you need to specify it in the canary custom resource:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: app
namespace: test
spec:
# can be: kubernetes, istio, appmesh, linkerd, smi, nginx, gloo, supergloo
# use the kubernetes provider for Blue/Green style deployments
provider: nginx
```
**Customized installer**
Create a kustomization file using flagger as base:
```bash
cat > kustomization.yaml <<EOF
namespace: istio-system
bases:
- github.com/weaveworks/flagger/kustomize/base/flagger
patchesStrategicMerge:
- patch.yaml
EOF
```
Create a patch and enable Slack notifications by setting the slack channel and hook URL:
```bash
cat > patch.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
spec:
template:
spec:
containers:
- name: flagger
args:
- -mesh-provider=istio
- -metrics-server=http://prometheus.istio-system:9090
- -slack-user=flagger
- -slack-channel=alerts
- -slack-url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
EOF
```
Install Flagger with Slack:
```bash
kubectl apply -k .
```
+89
View File
@@ -0,0 +1,89 @@
# Flagger Kustomize installer
As an alternative to Helm, Flagger can be installed with Kustomize.
## Service mesh specific installers
Install Flagger for Istio:
```bash
kubectl apply -k github.com/weaveworks/flagger/kustomize/istio
```
This deploys Flagger in the `istio-system` namespace and sets the metrics server URL to `http://prometheus.istio-system:9090`.
Install Flagger for Linkerd:
```bash
kubectl apply -k github.com/weaveworks/flagger/kustomize/linkerd
```
This deploys Flagger in the `linkerd` namespace and sets the metrics server URL to `http://linkerd-prometheus.linkerd:9090`.
## Generic installer
Install Flagger and Prometheus:
```bash
kubectl apply -k github.com/weaveworks/flagger/kustomize/kubernetes
```
This deploys Flagger and Prometheus in the `flagger-system` namespace,
sets the metrics server URL to `http://flagger-prometheus.flagger-system:9090` and the mesh provider to `kubernetes`.
To target a specific provider you need to specify it in the canary custom resource:
```yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
name: app
namespace: test
spec:
# can be: kubernetes, istio, appmesh, linkerd, smi, nginx, gloo, supergloo
# use the kubernetes provider for Blue/Green style deployments
provider: nginx
```
## Customized installer
Create a kustomization file using flagger as base:
```bash
cat > kustomization.yaml <<EOF
namespace: istio-system
bases:
- github.com/weaveworks/flagger/kustomize/base/flagger
patchesStrategicMerge:
- patch.yaml
EOF
```
Create a patch and enable Slack notifications by setting the slack channel and hook URL:
```bash
cat > patch.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
spec:
template:
spec:
containers:
- name: flagger
args:
- -mesh-provider=istio
- -metrics-server=http://prometheus.istio-system:9090
- -slack-user=flagger
- -slack-channel=alerts
- -slack-url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
EOF
```
Install Flagger with Slack:
```bash
kubectl apply -k .
```
+6
View File
@@ -0,0 +1,6 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: flagger
namespace: flagger-system
+172
View File
@@ -0,0 +1,172 @@
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: canaries.flagger.app
annotations:
helm.sh/resource-policy: keep
spec:
group: flagger.app
version: v1alpha3
versions:
- name: v1alpha3
served: true
storage: true
- name: v1alpha2
served: true
storage: false
- name: v1alpha1
served: true
storage: false
names:
plural: canaries
singular: canary
kind: Canary
categories:
- all
scope: Namespaced
subresources:
status: {}
additionalPrinterColumns:
- name: Status
type: string
JSONPath: .status.phase
- name: Weight
type: string
JSONPath: .status.canaryWeight
- name: LastTransitionTime
type: string
JSONPath: .status.lastTransitionTime
validation:
openAPIV3Schema:
properties:
spec:
required:
- targetRef
- service
- canaryAnalysis
properties:
provider:
type: string
progressDeadlineSeconds:
type: number
targetRef:
type: object
required: ['apiVersion', 'kind', 'name']
properties:
apiVersion:
type: string
kind:
type: string
name:
type: string
autoscalerRef:
anyOf:
- type: string
- type: object
required: ['apiVersion', 'kind', 'name']
properties:
apiVersion:
type: string
kind:
type: string
name:
type: string
ingressRef:
anyOf:
- type: string
- type: object
required: ['apiVersion', 'kind', 'name']
properties:
apiVersion:
type: string
kind:
type: string
name:
type: string
service:
type: object
required: ['port']
properties:
port:
type: number
portName:
type: string
portDiscovery:
type: boolean
meshName:
type: string
timeout:
type: string
skipAnalysis:
type: boolean
canaryAnalysis:
properties:
interval:
type: string
pattern: "^[0-9]+(m|s)"
iterations:
type: number
threshold:
type: number
maxWeight:
type: number
stepWeight:
type: number
metrics:
type: array
properties:
items:
type: object
required: ['name', 'threshold']
properties:
name:
type: string
interval:
type: string
pattern: "^[0-9]+(m|s)"
threshold:
type: number
query:
type: string
webhooks:
type: array
properties:
items:
type: object
required: ['name', 'url', 'timeout']
properties:
name:
type: string
type:
type: string
enum:
- ""
- pre-rollout
- rollout
- post-rollout
url:
type: string
format: url
timeout:
type: string
pattern: "^[0-9]+(m|s)"
status:
properties:
phase:
type: string
enum:
- ""
- Initialized
- Progressing
- Succeeded
- Failed
canaryWeight:
type: number
failedChecks:
type: number
iterations:
type: number
lastAppliedSpec:
type: string
lastTransitionTime:
type: string
+57
View File
@@ -0,0 +1,57 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
namespace: flagger-system
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: flagger
template:
metadata:
labels:
app: flagger
annotations:
prometheus.io/scrape: "true"
spec:
serviceAccountName: flagger
containers:
- name: flagger
image: weaveworks/flagger:0.16.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
livenessProbe:
exec:
command:
- wget
- --quiet
- --tries=1
- --timeout=2
- --spider
- http://localhost:8080/healthz
timeoutSeconds: 5
readinessProbe:
exec:
command:
- wget
- --quiet
- --tries=1
- --timeout=2
- --spider
- http://localhost:8080/healthz
timeoutSeconds: 5
resources:
limits:
memory: "512Mi"
cpu: "1000m"
requests:
memory: "32Mi"
cpu: "10m"
securityContext:
readOnlyRootFilesystem: true
runAsUser: 10001
+11
View File
@@ -0,0 +1,11 @@
namespace: flagger-system
commonLabels:
app: flagger
resources:
- account.yaml
- rbac.yaml
- crd.yaml
- deployment.yaml
images:
- name: weaveworks/flagger
newTag: 0.16.0
+90
View File
@@ -0,0 +1,90 @@
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: flagger
rules:
- apiGroups:
- ""
resources:
- events
- configmaps
- secrets
- services
verbs: ["*"]
- apiGroups:
- apps
resources:
- deployments
verbs: ["*"]
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs: ["*"]
- apiGroups:
- "extensions"
resources:
- ingresses
- ingresses/status
verbs: ["*"]
- apiGroups:
- flagger.app
resources:
- canaries
- canaries/status
verbs: ["*"]
- apiGroups:
- networking.istio.io
resources:
- virtualservices
- virtualservices/status
- destinationrules
- destinationrules/status
verbs: ["*"]
- apiGroups:
- appmesh.k8s.aws
resources:
- meshes
- meshes/status
- virtualnodes
- virtualnodes/status
- virtualservices
- virtualservices/status
verbs: ["*"]
- apiGroups:
- split.smi-spec.io
resources:
- trafficsplits
verbs: ["*"]
- apiGroups:
- gloo.solo.io
resources:
- settings
- upstreams
- upstreamgroups
- proxies
- virtualservices
verbs: ["*"]
- apiGroups:
- gateway.solo.io
resources:
- virtualservices
- gateways
verbs: ["*"]
- nonResourceURLs:
- /version
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: flagger
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flagger
subjects:
- kind: ServiceAccount
name: flagger
namespace: flagger-system
+5
View File
@@ -0,0 +1,5 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: flagger-prometheus
namespace: flagger-system
+52
View File
@@ -0,0 +1,52 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger-prometheus
namespace: flagger-system
spec:
replicas: 1
selector:
matchLabels:
app: flagger-prometheus
template:
metadata:
labels:
app: flagger-prometheus
annotations:
appmesh.k8s.aws/sidecarInjectorWebhook: disabled
sidecar.istio.io/inject: "false"
spec:
serviceAccountName: flagger-prometheus
containers:
- name: prometheus
image: prom/prometheus:v2.10.0
imagePullPolicy: IfNotPresent
args:
- '--storage.tsdb.retention=2h'
- '--config.file=/etc/prometheus/prometheus.yml'
ports:
- containerPort: 9090
name: http
livenessProbe:
httpGet:
path: /-/healthy
port: 9090
readinessProbe:
httpGet:
path: /-/ready
port: 9090
resources:
requests:
cpu: 10m
memory: 128Mi
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus
- name: data-volume
mountPath: /prometheus/data
volumes:
- name: config-volume
configMap:
name: flagger-prometheus
- name: data-volume
emptyDir: {}
@@ -0,0 +1,15 @@
namespace: flagger-system
commonLabels:
app: flagger-prometheus
resources:
- account.yaml
- rbac.yaml
- service.yaml
- deployment.yaml
configMapGenerator:
- name: flagger-prometheus
files:
- prometheus.yml
images:
- name: prom/prometheus
newTag: v2.10.0
+144
View File
@@ -0,0 +1,144 @@
global:
scrape_interval: 5s
scrape_configs:
# Scrape config for AppMesh Envoy sidecar
- job_name: 'appmesh-envoy'
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_name]
action: keep
regex: '^envoy$'
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: ${1}:9901
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
# Exclude high cardinality metrics
metric_relabel_configs:
- source_labels: [ cluster_name ]
regex: '(outbound|inbound|prometheus_stats).*'
action: drop
- source_labels: [ tcp_prefix ]
regex: '(outbound|inbound|prometheus_stats).*'
action: drop
- source_labels: [ listener_address ]
regex: '(.+)'
action: drop
- source_labels: [ http_conn_manager_listener_prefix ]
regex: '(.+)'
action: drop
- source_labels: [ http_conn_manager_prefix ]
regex: '(.+)'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_tls.*'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_tcp_downstream.*'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_http_(stats|admin).*'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_cluster_(lb|retry|bind|internal|max|original).*'
action: drop
# Scrape config for API servers
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- default
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: kubernetes;https
# Scrape config for nodes
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
# scrape config for cAdvisor
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
# scrape config for pods
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- source_labels: [ __address__ ]
regex: '.*9901.*'
action: drop
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
+32
View File
@@ -0,0 +1,32 @@
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: flagger-prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: flagger-prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flagger-prometheus
subjects:
- kind: ServiceAccount
name: flagger-prometheus
namespace: flagger-system
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: Service
metadata:
name: flagger-prometheus
namespace: flagger-system
spec:
selector:
app: flagger-prometheus
ports:
- name: http
protocol: TCP
port: 9090
+5
View File
@@ -0,0 +1,5 @@
namespace: istio-system
bases:
- ../base/flagger/
patchesStrategicMerge:
- patch.yaml
+15
View File
@@ -0,0 +1,15 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
spec:
template:
spec:
containers:
- name: flagger
args:
- -mesh-provider=istio
- -metrics-server=http://prometheus:9090
- -slack-user=flagger
- -slack-channel=
- -slack-url=
+8
View File
@@ -0,0 +1,8 @@
namespace: flagger-system
resources:
- namespace.yaml
bases:
- ../base/flagger/
- ../base/prometheus/
patchesStrategicMerge:
- patch.yaml
+9
View File
@@ -0,0 +1,9 @@
apiVersion: v1
kind: Namespace
metadata:
name: flagger-system
annotations:
linkerd.io/inject: disabled
labels:
istio-injection: disabled
appmesh.k8s.aws/sidecarInjectorWebhook: disabled
+15
View File
@@ -0,0 +1,15 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
spec:
template:
spec:
containers:
- name: flagger
args:
- -mesh-provider=kubernetes
- -metrics-server=http://flagger-prometheus:9090
- -slack-user=flagger
- -slack-channel=
- -slack-url=
+5
View File
@@ -0,0 +1,5 @@
namespace: linkerd
bases:
- ../base/flagger/
patchesStrategicMerge:
- patch.yaml
+15
View File
@@ -0,0 +1,15 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: flagger
spec:
template:
spec:
containers:
- name: flagger
args:
- -mesh-provider=linkerd
- -metrics-server=http://linkerd-prometheus:9090
- -slack-user=flagger
- -slack-channel=
- -slack-url=