diff --git a/incubator/mongodb-replicaset/Chart.yaml b/incubator/mongodb-replicaset/Chart.yaml index 31edd2eacc..b25f369610 100644 --- a/incubator/mongodb-replicaset/Chart.yaml +++ b/incubator/mongodb-replicaset/Chart.yaml @@ -1,6 +1,6 @@ name: mongodb-replicaset home: https://github.com/mongodb/mongo -version: 0.1.4 +version: 0.2.0 description: NoSQL document-oriented database that stores JSON-like documents with dynamic schemas, simplifying the integration of data in content-driven applications. icon: https://webassets.mongodb.com/_com_assets/cms/mongodb-logo-rgb-j6w271g1xn.jpg sources: diff --git a/incubator/mongodb-replicaset/README.md b/incubator/mongodb-replicaset/README.md index 45a083cae8..40d10af6f7 100644 --- a/incubator/mongodb-replicaset/README.md +++ b/incubator/mongodb-replicaset/README.md @@ -1,26 +1,23 @@ # MongoDB Helm Chart ## Prerequisites Details -* Kubernetes 1.3 or higher with alpha APIs enabled. -* Petsets are in alpha in 1.3. -* Init containers are alpha in 1.3 and are moving to beta in 1.4. +* Kubernetes 1.5+ with Beta APIs enabled. * PV support on the underlying infrastructure. -## PetSet Details -* http://kubernetes.io/docs/user-guide/petset/ +## StatefulSet Details +* https://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/ -## PetSet Caveats -* http://kubernetes.io/docs/user-guide/petset/#alpha-limitations +## StatefulSet Caveats +* https://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/#limitations ## TODO -* Set up authorization between replicaset peers. -* Add a liveliness check. -* Make the `test.sh` script more robust. +* Set up authorization between replica set peers. +* Add liveness and readiness probes ## Chart Details This chart implements a dynamically scalable [MongoDB replica set](https://docs.mongodb.com/manual/tutorial/deploy-replica-set/) -using Kubernetes PetSets and Init Containers. +using Kubernetes StatefulSets and Init Containers. ## Installing the Chart @@ -35,20 +32,41 @@ $ helm install --name my-release incubator/mongodb-replicaset The following tables lists the configurable parameters of the mongodb chart and their default values. -| Parameter | Description | Default | -| ----------------------- | ---------------------------------- | ---------------------------------------------------------- | -| `Name` | Name of the chart | `mongodb-replicaset` | -| `Image` | Container image name | `mongo` | -| `ImageTag` | Container image tag | `3.2` | -| `ImagePullPolicy` | Container pull policy | `Always` | -| `Replicas` | k8s petset replicas | `3` | -| `Component` | k8s selector key | `mongodb` | -| `ReplicaSet` | MongoDB ReplicaSet name | `rs0` | -| `ClusterRole` | Role of the cluster (Data/Config) | `shardsvr` | -| `Cpu` | container requested cpu | `100m` | -| `Memory` | container requested memory | `512Mi` | -| `PeerPort` | Container listening port | `27017` | -| `Storage` | Persistent volume size | `10Gi` | +| Parameter | Description | Default | +| ------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------- | +| `replicaSet` | Name of the replica set | rs0 | +| `replicas` | Number of replicas in the replica set | 3 | +| `port` | MongoDB port | 27017 | +| `installImage.name` | Image name for the init container that establishes the replica set | gcr.io/google_containers/mongodb-install | +| `installImage.tag` | Image tag for the init container that establishes the replica set | 0.3 | +| `installImage.pullPolicy` | Image pull policy for the init container that establishes the replica set | IfNotPresent | +| `image.name` | MongoDB image name | mongo | +| `image.tag` | MongoDB image tag | 3.2 | +| `image.pullPolicy` | MongoDB image pull policy | IfNotPresent | +| `podAnnotations` | Annotations to be added to MongoDB pods | {} | +| `resources` | Pod resource requests and limits | {} | +| `persistentVolume.enabled` | If `true`, persistent volume claims are created | `true` | +| `persistentVolume.storageClass` | Persistent volume storage class | `volume.alpha.kubernetes.io/storage-class: default` | +| `persistentVolume.accessMode` | Persistent volume access modes | [ReadWriteOnce] | +| `persistentVolume.size` | Persistent volume size | 10Gi | +| `persistentVolume.annotations` | Persistent volume annotations | {} | +| `serviceAnnotations` | Annotations to be added to the service | {} | +| `configmap` | Content of the MongoDB config file | See below | + +*MongoDB config file* + +The MongoDB config file `mongod.conf` is configured via the `configmap` configuration value. The defaults from +`values.yaml` are the following: + +```yaml +configmap: + storage: + dbPath: /data/db + net: + port: 27017 + replication: + replSetName: rs0 +``` Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. @@ -141,7 +159,7 @@ $ kubectl delete pod $RELEASE_NAME-mongodb-0 pod "messy-hydra-mongodb-0" deleted ``` -Delete all pods and let the petset controller bring it up. +Delete all pods and let the statefulset controller bring it up. ```console $ kubectl delete po -l app=mongodb $ kubectl get po --watch-only @@ -191,4 +209,3 @@ connecting to: test ## Scaling Scaling should be managed by `helm upgrade`, which is the recommended way. -You can also scale up by modifying the number of replicas on the PetSet using `kubectl patch` or `kubectl apply`. Deleting the various pods created and the associated resources needs some care and is described in the [petset documentation](http://kubernetes.io/docs/user-guide/petset/#deleting-a-pet-set). diff --git a/incubator/mongodb-replicaset/init/files/on-start.sh b/incubator/mongodb-replicaset/init/files/on-start.sh index 4c95c10d4f..ce917f055e 100644 --- a/incubator/mongodb-replicaset/init/files/on-start.sh +++ b/incubator/mongodb-replicaset/init/files/on-start.sh @@ -19,7 +19,7 @@ function join { } HOSTNAME=$(hostname) -# Parse out cluster name, formatted as: petset_name-index +# Parse out cluster name, formatted as: statefulset_name-index IFS='-' read -ra ADDR <<< "$(hostname)" CLUSTER_NAME="${ADDR[0]}" @@ -58,4 +58,4 @@ if [ $? -eq 0 ]; then fi # Do a clean shutdown of mongod -/usr/bin/mongo --eval "db.getSiblingDB('admin').shutdownServer({force: true})" \ No newline at end of file +/usr/bin/mongo --eval "db.getSiblingDB('admin').shutdownServer({force: true})" diff --git a/incubator/mongodb-replicaset/templates/NOTES.txt b/incubator/mongodb-replicaset/templates/NOTES.txt index dad536ba39..eef8ad61a6 100644 --- a/incubator/mongodb-replicaset/templates/NOTES.txt +++ b/incubator/mongodb-replicaset/templates/NOTES.txt @@ -1,11 +1,13 @@ -1. After the petset is created completely, one can check which instance is primary by running: - $ for i in `seq 0 2`; do kubectl exec {{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}-$i -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"'; done. - This assumes 3 replicas, 0 through 2. It should be modified to reflect the actual number of replicas specified. +1. After the statefulset is created completely, one can check which instance is primary by running: + + $ for ((i = 0; i < {{ .Values.replicas }}; ++i)); do kubectl exec --namespace {{ .Release.Namespace }} {{ template "fullname" . }}-$i -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"'; done 2. One can insert a key into the primary instance of the mongodb replica set by running the following: - $ kubectl exec MASTER_POD_NAME -- /usr/bin/mongo --eval="printjson(db.test.insert({key1: 'value1'}))" - MASTER_POD_NAME must be replaced with the name of the master found from the previous step. + MASTER_POD_NAME must be replaced with the name of the master found from the previous step. + + $ kubectl exec --namespace {{ .Release.Namespace }} MASTER_POD_NAME -- /usr/bin/mongo --eval="printjson(db.test.insert({key1: 'value1'}))" 3. One can fetch the keys stored in the primary or any of the slave nodes in the following manner. - $ kubectl exec POD_NAME -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)" - POD_NAME must be replaced by the name of the pod being queried. + POD_NAME must be replaced by the name of the pod being queried. + + $ kubectl exec --namespace {{ .Release.Namespace }} POD_NAME -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)" diff --git a/incubator/mongodb-replicaset/templates/_helpers.tpl b/incubator/mongodb-replicaset/templates/_helpers.tpl new file mode 100644 index 0000000000..f0d83d2edb --- /dev/null +++ b/incubator/mongodb-replicaset/templates/_helpers.tpl @@ -0,0 +1,16 @@ +{{/* vim: set filetype=mustache: */}} +{{/* +Expand the name of the chart. +*/}} +{{- define "name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +*/}} +{{- define "fullname" -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} diff --git a/incubator/mongodb-replicaset/templates/mongodb-configmap.yaml b/incubator/mongodb-replicaset/templates/mongodb-configmap.yaml new file mode 100644 index 0000000000..7722b4e4b6 --- /dev/null +++ b/incubator/mongodb-replicaset/templates/mongodb-configmap.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + labels: + app: {{ template "name" . }} + chart: {{ .Chart.Name }}-{{ .Chart.Version }} + heritage: {{ .Release.Service }} + release: {{ .Release.Name }} + name: {{ template "fullname" . }} +data: + mongod.conf: | +{{ toYaml .Values.configmap | indent 4 }} diff --git a/incubator/mongodb-replicaset/templates/mongodb-petset.yaml b/incubator/mongodb-replicaset/templates/mongodb-petset.yaml deleted file mode 100644 index 70487271d0..0000000000 --- a/incubator/mongodb-replicaset/templates/mongodb-petset.yaml +++ /dev/null @@ -1,166 +0,0 @@ -# A headless service to create DNS records -apiVersion: v1 -kind: Service -metadata: - name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}" - labels: - heritage: {{.Release.Service | quote }} - release: {{.Release.Name | quote }} - clusterrole: {{.Values.ClusterRole | quote }} - replicaset: {{.Values.ReplicaSet | quote }} - chart: "{{.Chart.Name}}-{{.Chart.Version}}" - component: "{{.Release.Name}}-{{.Values.Component}}" - annotations: - service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" -spec: - ports: - - port: {{.Values.PeerPort}} - name: peer - clusterIP: None - selector: - component: "{{.Release.Name}}-{{.Values.Component}}" ---- -apiVersion: v1 -kind: ConfigMap -metadata: - name: "{{.Release.Name}}-mongo-config" -data: - mongod.conf: | - # mongod.conf - # Where to store the data. - storage: - dbPath: /data/db - net: - port: 27017 - - # Replication Options - - # in replicated mongo databases, specify the replica set name here - sharding: - clusterRole: {{.Values.ClusterRole}} - replication: - replSetName: {{.Values.ReplicaSet}} - ---- -apiVersion: apps/v1beta1 -kind: StatefulSet -metadata: - name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}" - labels: - heritage: {{.Release.Service | quote }} - release: {{.Release.Name | quote }} - clusterrole: {{.Values.ClusterRole | quote }} - replicaset: {{.Values.ReplicaSet | quote }} - chart: "{{.Chart.Name}}-{{.Chart.Version}}" - component: "{{.Release.Name}}-{{.Values.Component}}" -spec: - serviceName: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}" - replicas: {{default 3 .Values.Replicas}} - template: - metadata: - labels: - heritage: {{.Release.Service | quote }} - release: {{.Release.Name | quote }} - clusterrole: {{.Values.ClusterRole | quote }} - replicaset: {{.Values.ReplicaSet | quote }} - chart: "{{.Chart.Name}}-{{.Chart.Version}}" - component: "{{.Release.Name}}-{{.Values.Component}}" - annotations: - pod.alpha.kubernetes.io/init-containers: '[ - { - "name": "install", - "image": "{{.Values.InstallImage}}:{{.Values.InstallImageTag}}", - "args": ["--work-dir=/work-dir"], - "imagePullPolicy": "{{.Values.ImagePullPolicy}}", - "volumeMounts": [ - { - "name": "workdir", - "mountPath": "/work-dir" - }, - { - "name": "config", - "mountPath": "/config" - } - ] - }, - { - "name": "bootstrap", - "image": "{{.Values.Image}}:{{.Values.ImageTag}}", - "command": ["/work-dir/peer-finder"], - "args": ["-on-start=\"/work-dir/on-start.sh\"", "-service={{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}"], - "imagePullPolicy": "{{.Values.ImagePullPolicy}}", - "env": [ - { - "name": "POD_NAMESPACE", - "valueFrom": { - "fieldRef": { - "apiVersion": "v1", - "fieldPath": "metadata.namespace" - } - } - }, - { - "name": "RS", - "value": "{{.Values.ReplicaSet}}" - } - ], - "volumeMounts": [ - { - "name": "workdir", - "mountPath": "/work-dir" - }, - { - "name": "config", - "mountPath": "/config" - }, - { - "name": "datadir", - "mountPath": "/data/db" - } - ] - } - ]' - spec: - containers: - - name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}" - image: "{{.Values.Image}}:{{.Values.ImageTag}}" - imagePullPolicy: "{{.Values.ImagePullPolicy}}" - ports: - - containerPort: {{.Values.PeerPort}} - name: peer - resources: - requests: - cpu: "{{.Values.Cpu}}" - memory: "{{.Values.Memory}}" - command: - - /usr/bin/mongod - - --config=/config/mongod.conf - readinessProbe: - exec: - command: - - sh - - -c - - "/usr/bin/mongo --eval 'printjson(db.serverStatus())'" - initialDelaySeconds: 5 - timeoutSeconds: 5 - volumeMounts: - - name: datadir - mountPath: /data/db - - name: config - mountPath: /config - volumes: - - name: config - configMap: - name: "{{.Release.Name}}-mongo-config" - - name: workdir - emptyDir: {} - volumeClaimTemplates: - - metadata: - name: datadir - annotations: - volume.alpha.kubernetes.io/storage-class: {{.Values.StorageClass}} - spec: - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: {{.Values.Storage}} diff --git a/incubator/mongodb-replicaset/templates/mongodb-service.yaml b/incubator/mongodb-replicaset/templates/mongodb-service.yaml new file mode 100644 index 0000000000..d2903e7488 --- /dev/null +++ b/incubator/mongodb-replicaset/templates/mongodb-service.yaml @@ -0,0 +1,24 @@ +# A headless service to create DNS records +apiVersion: v1 +kind: Service +metadata: + annotations: + service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" + {{- if .Values.serviceAnnotations }} +{{ toYaml .Values.serviceAnnotations | indent 4 }} + {{- end }} + labels: + app: {{ template "name" . }} + chart: {{ .Chart.Name }}-{{ .Chart.Version }} + heritage: {{ .Release.Service }} + release: {{ .Release.Name }} + name: {{ template "fullname" . }} +spec: + type: ClusterIP + clusterIP: None + ports: + - name: peer + port: {{ .Values.port }} + selector: + app: {{ template "name" . }} + release: {{ .Release.Name }} diff --git a/incubator/mongodb-replicaset/templates/mongodb-statefulset.yaml b/incubator/mongodb-replicaset/templates/mongodb-statefulset.yaml new file mode 100644 index 0000000000..e81bc5bdd3 --- /dev/null +++ b/incubator/mongodb-replicaset/templates/mongodb-statefulset.yaml @@ -0,0 +1,132 @@ +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + labels: + app: {{ template "name" . }} + chart: {{ .Chart.Name }}-{{ .Chart.Version }} + heritage: {{ .Release.Service }} + release: {{ .Release.Name }} + name: {{ template "fullname" . }} +spec: + serviceName: {{ template "fullname" . }} + replicas: {{ .Values.replicas }} + template: + metadata: + labels: + app: {{ template "name" . }} + release: {{ .Release.Name }} + annotations: + {{- if .Values.podAnnotations }} +{{ toYaml .Values.podAnnotations | indent 8}} + {{- end }} + pod.alpha.kubernetes.io/init-containers: '[ + { + "name": "install", + "image": "{{ .Values.installImage.name }}:{{ .Values.installImage.tag }}", + "args": ["--work-dir=/work-dir"], + "imagePullPolicy": "{{ .Values.installImage.pullPolicy }}", + "volumeMounts": [ + { + "name": "workdir", + "mountPath": "/work-dir" + }, + { + "name": "config", + "mountPath": "/config" + } + ] + }, + { + "name": "bootstrap", + "image": "{{ .Values.image.name }}:{{ .Values.image.tag }}", + "command": ["/work-dir/peer-finder"], + "args": ["-on-start=/work-dir/on-start.sh", "-service={{ template "fullname" . }}"], + "imagePullPolicy": "{{ .Values.image.pullPolicy }}", + "env": [ + { + "name": "POD_NAMESPACE", + "valueFrom": { + "fieldRef": { + "apiVersion": "v1", + "fieldPath": "metadata.namespace" + } + } + }, + { + "name": "RS", + "value": "{{ .Values.replicaSet }}" + } + ], + "volumeMounts": [ + { + "name": "workdir", + "mountPath": "/work-dir" + }, + { + "name": "config", + "mountPath": "/config" + }, + { + "name": "datadir", + "mountPath": "/data/db" + } + ] + } + ]' + spec: + containers: + - name: {{ template "name" . }} + image: "{{ .Values.image.name }}:{{ .Values.image.tag }}" + imagePullPolicy: "{{ .Values.image.pullPolicy }}" + ports: + - name: peer + containerPort: {{ .Values.port }} + resources: +{{ toYaml .Values.resources | indent 12 }} + command: + - /usr/bin/mongod + - --config=/config/mongod.conf + readinessProbe: + exec: + command: + - sh + - -c + - "/usr/bin/mongo --eval 'printjson(db.serverStatus())'" + initialDelaySeconds: 5 + timeoutSeconds: 5 + volumeMounts: + - name: datadir + mountPath: /data/db + - name: config + mountPath: /config + volumes: + - name: config + configMap: + name: {{ template "fullname" . }} + - name: workdir + emptyDir: {} +{{- if .Values.persistentVolume.enabled }} + volumeClaimTemplates: + - metadata: + name: datadir + annotations: + {{- if .Values.persistentVolume.storageClass | quote }} + volume.beta.kubernetes.io/storage-class: {{ .Values.persistentVolume.storageClass | quote }} + {{- else }} + volume.alpha.kubernetes.io/storage-class: default + {{- end }} + {{- range $key, $value := .Values.persistentVolume.annotations }} + {{ $key }}: {{ $value }} + {{- end }} + spec: + accessModes: + {{- range .Values.persistentVolume.accessModes }} + - {{ . | quote }} + {{- end }} + resources: + requests: + storage: {{ .Values.persistentVolume.size | quote }} +{{- else }} + - name: datadir + emptyDir: {} +{{- end }} diff --git a/incubator/mongodb-replicaset/test.sh b/incubator/mongodb-replicaset/test.sh index ee8d04eb47..1408bd1552 100755 --- a/incubator/mongodb-replicaset/test.sh +++ b/incubator/mongodb-replicaset/test.sh @@ -1,6 +1,6 @@ #! /bin/bash -# Copyright 2016 The Kubernetes Authors All rights reserved. +# Copyright 2016 The Kubernetes Authors. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,11 +14,29 @@ # See the License for the specific language governing permissions and # limitations under the License. -POD_NAME=$RELEASE_NAME-mongodb-replicaset -POD_NAME=${POD_NAME:0:24} +NS="${RELEASE_NAMESPACE:-default}" +POD_NAME="${RELEASE_NAME:-mongo}-mongodb-replicaset" + +for i in `seq 0 2`; do + pod="${POD_NAME}-$i" + kubectl exec --namespace $NS $pod -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"' | grep '"ismaster" : true' + + if [ $? -eq 0 ]; then + echo "Found master: $pod" + MASTER=$pod + break + fi +done + +kubectl exec --namespace $NS $MASTER -- /usr/bin/mongo --eval='printjson(db.test.insert({"status": "success"}))' -kubectl exec "$POD_NAME-0" -- /usr/bin/mongo --eval="printjson(db.test.insert({\"status\": \"success\"}))" # TODO: find maximum duration to wait for slaves to be up-to-date with master. sleep 2 -kubectl exec "$POD_NAME-1" -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)" -kubectl exec "$POD_NAME-2" -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)" + +for i in `seq 0 2`; do + pod="${POD_NAME}-$i" + if [[ $pod != $MASTER ]]; then + echo "Reading from slave: $pod" + kubectl exec --namespace $NS $pod -- /usr/bin/mongo --eval='rs.slaveOk(); db.test.find().forEach(printjson)' + fi +done diff --git a/incubator/mongodb-replicaset/values.yaml b/incubator/mongodb-replicaset/values.yaml index acf4dd9714..f5b8224f50 100644 --- a/incubator/mongodb-replicaset/values.yaml +++ b/incubator/mongodb-replicaset/values.yaml @@ -1,21 +1,49 @@ -# Default values for mongodb. -# This is a YAML-formatted file. -# Declare name/value pairs to be passed into your templates. -# name: value +replicaSet: rs0 +replicas: 3 +port: 27017 -Name: mongodb-replicaset -PeerPort: 27017 -Component: "mongodb" -ReplicaSet: "rs0" -ClusterRole: "shardsvr" -Replicas: 3 -Image: "mongo" -ImageTag: "3.2" -InstallImage: "gcr.io/google_containers/mongodb-install" -InstallImageTag: "0.3" -ImagePullPolicy: "Always" -Cpu: "100m" -Memory: "512Mi" -Storage: "10Gi" -StorageClass: generic +# Specs for the Docker image for the init container that establishes the replica set +installImage: + name: gcr.io/google_containers/mongodb-install + tag: 0.3 + pullPolicy: IfNotPresent +# Specs for the MongoDB image +image: + name: mongo + tag: 3.2 + pullPolicy: IfNotPresent + +# Annotations to be added to MongoDB pods +podAnnotations: {} + +resources: {} + # limits: + # cpu: 100m + # memory: 512Mi + # requests: + # cpu: 100m + # memory: 512Mi + +persistentVolume: + enabled: true + ## If defined, volume.beta.kubernetes.io/storage-class: + ## Default: volume.alpha.kubernetes.io/storage-class: default + ## + # storageClass: fast + accessModes: + - ReadWriteOnce + size: 10Gi + annotations: {} + +# Annotations to be added to the service +serviceAnnotations: {} + +# Entries for the MongoDB config file +configmap: + storage: + dbPath: /data/db + net: + port: 27017 + replication: + replSetName: rs0