From 1b086f941fed57c623d96304ced41bc99a56741b Mon Sep 17 00:00:00 2001 From: Quentin Nerden Date: Thu, 12 Oct 2017 21:24:11 +0200 Subject: [PATCH] Add Fluentd > cloudwatch log forwarder (#952) * fluentd cloudwatch incubator * rename deployment -> daemonset * annotations as value * reorder values * use helm namespace * address code review * move td-agent.conf into values.yaml * Updated fluentd-cloudwatch code from PR 211 The configuration is now inside the container (from PR comment: https://github.com/kubernetes/charts/pull/211#issuecomment-286643280), I then removed the configMap holding the config. The AWS keys are added as secrets from files, using Files.Glob (from PR comment https://github.com/kubernetes/charts/pull/211#issuecomment-286662756) * Add configMap for fluentd config Copied the config from what's in /fluentd/etc in image: fluent/fluentd-kubernetes-daemonset:v0.12-cloudwatch * Update to use namespaces * Add appVersion * Add fluentd icon * Using github usernames instead of real names * Update label for configmap --- incubator/fluentd-cloudwatch/Chart.yaml | 20 +++ incubator/fluentd-cloudwatch/README.md | 71 ++++++++ .../secrets/aws_access_key_id | 1 + .../secrets/aws_secret_access_key | 1 + .../fluentd-cloudwatch/templates/NOTES.txt | 6 + .../fluentd-cloudwatch/templates/_helpers.tpl | 16 ++ .../templates/configmap.yaml | 11 ++ .../templates/daemonset.yaml | 61 +++++++ .../fluentd-cloudwatch/templates/secrets.yaml | 12 ++ incubator/fluentd-cloudwatch/values.yaml | 157 ++++++++++++++++++ 10 files changed, 356 insertions(+) create mode 100644 incubator/fluentd-cloudwatch/Chart.yaml create mode 100644 incubator/fluentd-cloudwatch/README.md create mode 100644 incubator/fluentd-cloudwatch/secrets/aws_access_key_id create mode 100644 incubator/fluentd-cloudwatch/secrets/aws_secret_access_key create mode 100644 incubator/fluentd-cloudwatch/templates/NOTES.txt create mode 100644 incubator/fluentd-cloudwatch/templates/_helpers.tpl create mode 100644 incubator/fluentd-cloudwatch/templates/configmap.yaml create mode 100644 incubator/fluentd-cloudwatch/templates/daemonset.yaml create mode 100644 incubator/fluentd-cloudwatch/templates/secrets.yaml create mode 100644 incubator/fluentd-cloudwatch/values.yaml diff --git a/incubator/fluentd-cloudwatch/Chart.yaml b/incubator/fluentd-cloudwatch/Chart.yaml new file mode 100644 index 0000000000..0e4eaa6c68 --- /dev/null +++ b/incubator/fluentd-cloudwatch/Chart.yaml @@ -0,0 +1,20 @@ +name: fluentd-cloudwatch +version: 0.1.1 +appVersion: 0.1.1 +description: A Fluentd CloudWatch Helm chart for Kubernetes. +icon: https://raw.githubusercontent.com/fluent/fluentd-docs/master/public/logo/Fluentd_square.png +keywords: +- fluentd +- cloudwatch +- logging +sources: +- https://github.com/kubernetes/charts +- https://github.com/fluent/fluentd-kubernetes-daemonset +maintainers: +- name: jmcarp + email: jm.carp@gmail.com +- name: icereval + email: michael.haselton@gmail.com +- name: kenden + email: quentin.nerden@stylight.com +engine: gotpl diff --git a/incubator/fluentd-cloudwatch/README.md b/incubator/fluentd-cloudwatch/README.md new file mode 100644 index 0000000000..a3c5a34dcd --- /dev/null +++ b/incubator/fluentd-cloudwatch/README.md @@ -0,0 +1,71 @@ +# Fluentd CloudWatch + +* Installs [Fluentd](https://www.fluentd.org/) [Cloudwatch](https://aws.amazon.com/cloudwatch/) log forwarder. + +## TL;DR; + +```console +$ helm install incubator/fluentd-cloudwatch +``` + +## Introduction + +This chart bootstraps a [Fluentd](https://www.fluentd.org/) [Cloudwatch](https://aws.amazon.com/cloudwatch/) deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. + +## Prerequisites + +- Kubernetes 1.4+ with Beta APIs enabled + +## Installing the Chart + +To install the chart with the release name `my-release`: + +```console +$ # edit secrets/aws_access_key_id and secrets/aws_access_key_id with the key/password of a AWS user with a policy to access Cloudwatch +$ helm install --name my-release incubator/fluentd-cloudwatch +``` + +The command deploys Fluentd Cloudwatch on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. + +## Uninstalling the Chart + +To uninstall/delete the `my-release` deployment: + +```console +$ helm delete my-release +``` + +The command removes all the Kubernetes components associated with the chart and deletes the release. + +## Configuration + +The following tables lists the configurable parameters of the Fluentd Cloudwatch chart and their default values. + +| Parameter | Description | Default | +| ------------------------------- | ------------------------------------------ | ---------------------------------------------------------- | +| `image` | Image | `fluent/fluentd-kubernetes-daemonset` | +| `imageTag` | Image tag | `v0.12.33-cloudwatch` | +| `imagePullPolicy` | Image pull policy | `Always` if `imageTag` is `imagePullPolicy` | +| `resources.limits.cpu` | CPU limit | `100m` | +| `resources.limits.memory` | Memory limit | `200Mi` | +| `resources.requests.cpu` | CPU request | `100m` | +| `resources.requests.memory` | Memory request | `200Mi` | +| `hostNetwork` | Host network | `false` | +| `annotations` (removed for now) | Annotations | `nil` | +| `awsRegion` | AWS Cloudwatch region | `us-east-1` | +| `fluentdConfig` | Fluentd configuration | `example configuration` | +| `logGroupName` | AWS Cloudwatch log group | `kubernetes` | + +Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, + +```console +$ helm install --name my-release \ + --set awsRegion=us-east-1 \ + incubator/fluentd-cloudwatch +``` + +Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example, + +```console +$ helm install --name my-release -f values.yaml stable/fluentd-cloudwatch +``` diff --git a/incubator/fluentd-cloudwatch/secrets/aws_access_key_id b/incubator/fluentd-cloudwatch/secrets/aws_access_key_id new file mode 100644 index 0000000000..4d7fac3bf8 --- /dev/null +++ b/incubator/fluentd-cloudwatch/secrets/aws_access_key_id @@ -0,0 +1 @@ + diff --git a/incubator/fluentd-cloudwatch/secrets/aws_secret_access_key b/incubator/fluentd-cloudwatch/secrets/aws_secret_access_key new file mode 100644 index 0000000000..41954eec7e --- /dev/null +++ b/incubator/fluentd-cloudwatch/secrets/aws_secret_access_key @@ -0,0 +1 @@ + diff --git a/incubator/fluentd-cloudwatch/templates/NOTES.txt b/incubator/fluentd-cloudwatch/templates/NOTES.txt new file mode 100644 index 0000000000..62840c5006 --- /dev/null +++ b/incubator/fluentd-cloudwatch/templates/NOTES.txt @@ -0,0 +1,6 @@ +To verify that Fluentd Cloudwatch has started, run: + + kubectl --namespace={{ .Release.Namespace }} get pods -l "app={{ template "fluentd-cloudwatch.name" . }},release={{ .Release.Name }}" + +THIS APPLICATION CAPTURES ALL CONSOLE OUTPUT AND FORWARDS IT TO AWS CLOUDWATCH. Anything that might be identifying, +including things like IP addresses, container images, and object names will NOT be anonymized. diff --git a/incubator/fluentd-cloudwatch/templates/_helpers.tpl b/incubator/fluentd-cloudwatch/templates/_helpers.tpl new file mode 100644 index 0000000000..9663251908 --- /dev/null +++ b/incubator/fluentd-cloudwatch/templates/_helpers.tpl @@ -0,0 +1,16 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "fluentd-cloudwatch.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 24 -}} +{{- end -}} + +{{/* +Create a default fully qualified app name. +We truncate at 24 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +*/}} +{{- define "fluentd-cloudwatch.fullname" -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- printf "%s-%s" .Release.Name $name | trunc 24 -}} +{{- end -}} diff --git a/incubator/fluentd-cloudwatch/templates/configmap.yaml b/incubator/fluentd-cloudwatch/templates/configmap.yaml new file mode 100644 index 0000000000..b6f89a09ee --- /dev/null +++ b/incubator/fluentd-cloudwatch/templates/configmap.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ template "fluentd-cloudwatch.fullname" . }} + labels: + app: {{ template "fluentd-cloudwatch.name" . }} + chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" + heritage: {{ .Release.Service | quote }} + release: {{ .Release.Name | quote }} +data: + fluent.conf: {{ toYaml .Values.fluentdConfig | indent 2 }} diff --git a/incubator/fluentd-cloudwatch/templates/daemonset.yaml b/incubator/fluentd-cloudwatch/templates/daemonset.yaml new file mode 100644 index 0000000000..a1596ef879 --- /dev/null +++ b/incubator/fluentd-cloudwatch/templates/daemonset.yaml @@ -0,0 +1,61 @@ +apiVersion: extensions/v1beta1 +kind: DaemonSet +metadata: + name: {{ template "fluentd-cloudwatch.fullname" . }} + labels: + app: {{ template "fluentd-cloudwatch.name" . }} + chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" + heritage: "{{ .Release.Service }}" + release: "{{ .Release.Name }}" +spec: + template: + metadata: + labels: + app: {{ template "fluentd-cloudwatch.name" . }} + release: "{{ .Release.Name }}" + annotations: +{{ toYaml .Values.annotations | indent 8 }} + spec: + containers: + - name: {{ template "fluentd-cloudwatch.fullname" . }} + image: "{{ .Values.image }}:{{ .Values.imageTag }}" + imagePullPolicy: {{ .Values.imagePullPolicy | quote }} + #hostNetwork: {{ default false .Values.hostNetwork }} + env: + - name: AWS_REGION + value: {{ .Values.awsRegion }} + - name: LOG_GROUP_NAME + value: {{ .Values.logGroupName }} + + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + key: aws_access_key_id + name: {{ template "fluentd-cloudwatch.fullname" . }} + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + key: aws_secret_access_key + name: {{ template "fluentd-cloudwatch.fullname" . }} + + resources: +{{ toYaml .Values.resources | indent 10 }} + volumeMounts: + - name: varlog + mountPath: /var/log + - name: varlibdockercontainers + mountPath: /var/lib/docker/containers + readOnly: true + - name: config-volume + mountPath: /fluentd/etc + terminationGracePeriodSeconds: 30 + volumes: + - name: varlog + hostPath: + path: /var/log + - name: varlibdockercontainers + hostPath: + path: /var/lib/docker/containers + - name: config-volume + configMap: + name: {{ template "fluentd-cloudwatch.fullname" . }} diff --git a/incubator/fluentd-cloudwatch/templates/secrets.yaml b/incubator/fluentd-cloudwatch/templates/secrets.yaml new file mode 100644 index 0000000000..9e864c60ec --- /dev/null +++ b/incubator/fluentd-cloudwatch/templates/secrets.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Secret +metadata: + name: {{ template "fluentd-cloudwatch.fullname" . }} + labels: + app: {{ template "fluentd-cloudwatch.name" . }} + chart: {{ .Chart.Name }}-{{ .Chart.Version }} + heritage: {{ .Release.Service }} + release: {{ .Release.Name }} +type: Opaque +data: +{{ (.Files.Glob "secrets/*").AsSecrets | indent 2 }} diff --git a/incubator/fluentd-cloudwatch/values.yaml b/incubator/fluentd-cloudwatch/values.yaml new file mode 100644 index 0000000000..36a2e5efdc --- /dev/null +++ b/incubator/fluentd-cloudwatch/values.yaml @@ -0,0 +1,157 @@ +image: fluent/fluentd-kubernetes-daemonset +imageTag: v0.12.33-cloudwatch + +## Specify an imagePullPolicy (Required) +## It's recommended to change this to 'Always' if the image tag is 'latest' +## ref: http://kubernetes.io/docs/user-guide/images/#updating-images +imagePullPolicy: IfNotPresent + + +## Configure resource requests and limits +## ref: http://kubernetes.io/docs/user-guide/compute-resources/ +## +resources: + limits: + cpu: 100m + memory: 200Mi + requests: + cpu: 100m + memory: 200Mi + +#hostNetwork: false + +annotations: {} + +awsRegion: us-east-1 +logGroupName: kubernetes + +fluentdConfig: | + + type null + + + + type tail + path /var/log/containers/*.log + pos_file /var/log/fluentd-containers.log.pos + time_format %Y-%m-%dT%H:%M:%S.%NZ + tag kubernetes.* + format json + read_from_head true + + + + type tail + format /^(?