[mongodb-replicaset] Restructure and apply best practices (#767)

* [mongodb-replicaset] Restructure and apply best practices

* Split up to one yaml file per resource
* Fix old petset references
* Improve persistence configuration
* Config entries start with a lower-case letter
* Enhance configuration
* Drop component annotation
* Improve test.sh
* No default resoure requests/limits
* Add _helpers.tpl

* Version fix
This commit is contained in:
Reinhard Nägele
2017-03-22 20:33:12 -05:00
committed by Michael Goodness
parent 84c8e43f0f
commit 31b168f6b6
11 changed files with 311 additions and 228 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
name: mongodb-replicaset
home: https://github.com/mongodb/mongo
version: 0.1.4
version: 0.2.0
description: NoSQL document-oriented database that stores JSON-like documents with dynamic schemas, simplifying the integration of data in content-driven applications.
icon: https://webassets.mongodb.com/_com_assets/cms/mongodb-logo-rgb-j6w271g1xn.jpg
sources:
+44 -27
View File
@@ -1,26 +1,23 @@
# MongoDB Helm Chart
## Prerequisites Details
* Kubernetes 1.3 or higher with alpha APIs enabled.
* Petsets are in alpha in 1.3.
* Init containers are alpha in 1.3 and are moving to beta in 1.4.
* Kubernetes 1.5+ with Beta APIs enabled.
* PV support on the underlying infrastructure.
## PetSet Details
* http://kubernetes.io/docs/user-guide/petset/
## StatefulSet Details
* https://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/
## PetSet Caveats
* http://kubernetes.io/docs/user-guide/petset/#alpha-limitations
## StatefulSet Caveats
* https://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/#limitations
## TODO
* Set up authorization between replicaset peers.
* Add a liveliness check.
* Make the `test.sh` script more robust.
* Set up authorization between replica set peers.
* Add liveness and readiness probes
## Chart Details
This chart implements a dynamically scalable [MongoDB replica set](https://docs.mongodb.com/manual/tutorial/deploy-replica-set/)
using Kubernetes PetSets and Init Containers.
using Kubernetes StatefulSets and Init Containers.
## Installing the Chart
@@ -35,20 +32,41 @@ $ helm install --name my-release incubator/mongodb-replicaset
The following tables lists the configurable parameters of the mongodb chart and their default values.
| Parameter | Description | Default |
| ----------------------- | ---------------------------------- | ---------------------------------------------------------- |
| `Name` | Name of the chart | `mongodb-replicaset` |
| `Image` | Container image name | `mongo` |
| `ImageTag` | Container image tag | `3.2` |
| `ImagePullPolicy` | Container pull policy | `Always` |
| `Replicas` | k8s petset replicas | `3` |
| `Component` | k8s selector key | `mongodb` |
| `ReplicaSet` | MongoDB ReplicaSet name | `rs0` |
| `ClusterRole` | Role of the cluster (Data/Config) | `shardsvr` |
| `Cpu` | container requested cpu | `100m` |
| `Memory` | container requested memory | `512Mi` |
| `PeerPort` | Container listening port | `27017` |
| `Storage` | Persistent volume size | `10Gi` |
| Parameter | Description | Default |
| ------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------- |
| `replicaSet` | Name of the replica set | rs0 |
| `replicas` | Number of replicas in the replica set | 3 |
| `port` | MongoDB port | 27017 |
| `installImage.name` | Image name for the init container that establishes the replica set | gcr.io/google_containers/mongodb-install |
| `installImage.tag` | Image tag for the init container that establishes the replica set | 0.3 |
| `installImage.pullPolicy` | Image pull policy for the init container that establishes the replica set | IfNotPresent |
| `image.name` | MongoDB image name | mongo |
| `image.tag` | MongoDB image tag | 3.2 |
| `image.pullPolicy` | MongoDB image pull policy | IfNotPresent |
| `podAnnotations` | Annotations to be added to MongoDB pods | {} |
| `resources` | Pod resource requests and limits | {} |
| `persistentVolume.enabled` | If `true`, persistent volume claims are created | `true` |
| `persistentVolume.storageClass` | Persistent volume storage class | `volume.alpha.kubernetes.io/storage-class: default` |
| `persistentVolume.accessMode` | Persistent volume access modes | [ReadWriteOnce] |
| `persistentVolume.size` | Persistent volume size | 10Gi |
| `persistentVolume.annotations` | Persistent volume annotations | {} |
| `serviceAnnotations` | Annotations to be added to the service | {} |
| `configmap` | Content of the MongoDB config file | See below |
*MongoDB config file*
The MongoDB config file `mongod.conf` is configured via the `configmap` configuration value. The defaults from
`values.yaml` are the following:
```yaml
configmap:
storage:
dbPath: /data/db
net:
port: 27017
replication:
replSetName: rs0
```
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.
@@ -141,7 +159,7 @@ $ kubectl delete pod $RELEASE_NAME-mongodb-0
pod "messy-hydra-mongodb-0" deleted
```
Delete all pods and let the petset controller bring it up.
Delete all pods and let the statefulset controller bring it up.
```console
$ kubectl delete po -l app=mongodb
$ kubectl get po --watch-only
@@ -191,4 +209,3 @@ connecting to: test
## Scaling
Scaling should be managed by `helm upgrade`, which is the recommended way.
You can also scale up by modifying the number of replicas on the PetSet using `kubectl patch` or `kubectl apply`. Deleting the various pods created and the associated resources needs some care and is described in the [petset documentation](http://kubernetes.io/docs/user-guide/petset/#deleting-a-pet-set).
@@ -19,7 +19,7 @@ function join {
}
HOSTNAME=$(hostname)
# Parse out cluster name, formatted as: petset_name-index
# Parse out cluster name, formatted as: statefulset_name-index
IFS='-' read -ra ADDR <<< "$(hostname)"
CLUSTER_NAME="${ADDR[0]}"
@@ -58,4 +58,4 @@ if [ $? -eq 0 ]; then
fi
# Do a clean shutdown of mongod
/usr/bin/mongo --eval "db.getSiblingDB('admin').shutdownServer({force: true})"
/usr/bin/mongo --eval "db.getSiblingDB('admin').shutdownServer({force: true})"
@@ -1,11 +1,13 @@
1. After the petset is created completely, one can check which instance is primary by running:
$ for i in `seq 0 2`; do kubectl exec {{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}-$i -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"'; done.
This assumes 3 replicas, 0 through 2. It should be modified to reflect the actual number of replicas specified.
1. After the statefulset is created completely, one can check which instance is primary by running:
$ for ((i = 0; i < {{ .Values.replicas }}; ++i)); do kubectl exec --namespace {{ .Release.Namespace }} {{ template "fullname" . }}-$i -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"'; done
2. One can insert a key into the primary instance of the mongodb replica set by running the following:
$ kubectl exec MASTER_POD_NAME -- /usr/bin/mongo --eval="printjson(db.test.insert({key1: 'value1'}))"
MASTER_POD_NAME must be replaced with the name of the master found from the previous step.
MASTER_POD_NAME must be replaced with the name of the master found from the previous step.
$ kubectl exec --namespace {{ .Release.Namespace }} MASTER_POD_NAME -- /usr/bin/mongo --eval="printjson(db.test.insert({key1: 'value1'}))"
3. One can fetch the keys stored in the primary or any of the slave nodes in the following manner.
$ kubectl exec POD_NAME -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)"
POD_NAME must be replaced by the name of the pod being queried.
POD_NAME must be replaced by the name of the pod being queried.
$ kubectl exec --namespace {{ .Release.Namespace }} POD_NAME -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)"
@@ -0,0 +1,16 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
@@ -0,0 +1,12 @@
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app: {{ template "name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
name: {{ template "fullname" . }}
data:
mongod.conf: |
{{ toYaml .Values.configmap | indent 4 }}
@@ -1,166 +0,0 @@
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}"
labels:
heritage: {{.Release.Service | quote }}
release: {{.Release.Name | quote }}
clusterrole: {{.Values.ClusterRole | quote }}
replicaset: {{.Values.ReplicaSet | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
component: "{{.Release.Name}}-{{.Values.Component}}"
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
ports:
- port: {{.Values.PeerPort}}
name: peer
clusterIP: None
selector:
component: "{{.Release.Name}}-{{.Values.Component}}"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "{{.Release.Name}}-mongo-config"
data:
mongod.conf: |
# mongod.conf
# Where to store the data.
storage:
dbPath: /data/db
net:
port: 27017
# Replication Options
# in replicated mongo databases, specify the replica set name here
sharding:
clusterRole: {{.Values.ClusterRole}}
replication:
replSetName: {{.Values.ReplicaSet}}
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}"
labels:
heritage: {{.Release.Service | quote }}
release: {{.Release.Name | quote }}
clusterrole: {{.Values.ClusterRole | quote }}
replicaset: {{.Values.ReplicaSet | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
component: "{{.Release.Name}}-{{.Values.Component}}"
spec:
serviceName: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}"
replicas: {{default 3 .Values.Replicas}}
template:
metadata:
labels:
heritage: {{.Release.Service | quote }}
release: {{.Release.Name | quote }}
clusterrole: {{.Values.ClusterRole | quote }}
replicaset: {{.Values.ReplicaSet | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
component: "{{.Release.Name}}-{{.Values.Component}}"
annotations:
pod.alpha.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "{{.Values.InstallImage}}:{{.Values.InstallImageTag}}",
"args": ["--work-dir=/work-dir"],
"imagePullPolicy": "{{.Values.ImagePullPolicy}}",
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
},
{
"name": "config",
"mountPath": "/config"
}
]
},
{
"name": "bootstrap",
"image": "{{.Values.Image}}:{{.Values.ImageTag}}",
"command": ["/work-dir/peer-finder"],
"args": ["-on-start=\"/work-dir/on-start.sh\"", "-service={{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}"],
"imagePullPolicy": "{{.Values.ImagePullPolicy}}",
"env": [
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "RS",
"value": "{{.Values.ReplicaSet}}"
}
],
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
},
{
"name": "config",
"mountPath": "/config"
},
{
"name": "datadir",
"mountPath": "/data/db"
}
]
}
]'
spec:
containers:
- name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}"
image: "{{.Values.Image}}:{{.Values.ImageTag}}"
imagePullPolicy: "{{.Values.ImagePullPolicy}}"
ports:
- containerPort: {{.Values.PeerPort}}
name: peer
resources:
requests:
cpu: "{{.Values.Cpu}}"
memory: "{{.Values.Memory}}"
command:
- /usr/bin/mongod
- --config=/config/mongod.conf
readinessProbe:
exec:
command:
- sh
- -c
- "/usr/bin/mongo --eval 'printjson(db.serverStatus())'"
initialDelaySeconds: 5
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /data/db
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: "{{.Release.Name}}-mongo-config"
- name: workdir
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.alpha.kubernetes.io/storage-class: {{.Values.StorageClass}}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{.Values.Storage}}
@@ -0,0 +1,24 @@
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
{{- if .Values.serviceAnnotations }}
{{ toYaml .Values.serviceAnnotations | indent 4 }}
{{- end }}
labels:
app: {{ template "name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
name: {{ template "fullname" . }}
spec:
type: ClusterIP
clusterIP: None
ports:
- name: peer
port: {{ .Values.port }}
selector:
app: {{ template "name" . }}
release: {{ .Release.Name }}
@@ -0,0 +1,132 @@
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
labels:
app: {{ template "name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
name: {{ template "fullname" . }}
spec:
serviceName: {{ template "fullname" . }}
replicas: {{ .Values.replicas }}
template:
metadata:
labels:
app: {{ template "name" . }}
release: {{ .Release.Name }}
annotations:
{{- if .Values.podAnnotations }}
{{ toYaml .Values.podAnnotations | indent 8}}
{{- end }}
pod.alpha.kubernetes.io/init-containers: '[
{
"name": "install",
"image": "{{ .Values.installImage.name }}:{{ .Values.installImage.tag }}",
"args": ["--work-dir=/work-dir"],
"imagePullPolicy": "{{ .Values.installImage.pullPolicy }}",
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
},
{
"name": "config",
"mountPath": "/config"
}
]
},
{
"name": "bootstrap",
"image": "{{ .Values.image.name }}:{{ .Values.image.tag }}",
"command": ["/work-dir/peer-finder"],
"args": ["-on-start=/work-dir/on-start.sh", "-service={{ template "fullname" . }}"],
"imagePullPolicy": "{{ .Values.image.pullPolicy }}",
"env": [
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "RS",
"value": "{{ .Values.replicaSet }}"
}
],
"volumeMounts": [
{
"name": "workdir",
"mountPath": "/work-dir"
},
{
"name": "config",
"mountPath": "/config"
},
{
"name": "datadir",
"mountPath": "/data/db"
}
]
}
]'
spec:
containers:
- name: {{ template "name" . }}
image: "{{ .Values.image.name }}:{{ .Values.image.tag }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
ports:
- name: peer
containerPort: {{ .Values.port }}
resources:
{{ toYaml .Values.resources | indent 12 }}
command:
- /usr/bin/mongod
- --config=/config/mongod.conf
readinessProbe:
exec:
command:
- sh
- -c
- "/usr/bin/mongo --eval 'printjson(db.serverStatus())'"
initialDelaySeconds: 5
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /data/db
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: {{ template "fullname" . }}
- name: workdir
emptyDir: {}
{{- if .Values.persistentVolume.enabled }}
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
{{- if .Values.persistentVolume.storageClass | quote }}
volume.beta.kubernetes.io/storage-class: {{ .Values.persistentVolume.storageClass | quote }}
{{- else }}
volume.alpha.kubernetes.io/storage-class: default
{{- end }}
{{- range $key, $value := .Values.persistentVolume.annotations }}
{{ $key }}: {{ $value }}
{{- end }}
spec:
accessModes:
{{- range .Values.persistentVolume.accessModes }}
- {{ . | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.persistentVolume.size | quote }}
{{- else }}
- name: datadir
emptyDir: {}
{{- end }}
+24 -6
View File
@@ -1,6 +1,6 @@
#! /bin/bash
# Copyright 2016 The Kubernetes Authors All rights reserved.
# Copyright 2016 The Kubernetes Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -14,11 +14,29 @@
# See the License for the specific language governing permissions and
# limitations under the License.
POD_NAME=$RELEASE_NAME-mongodb-replicaset
POD_NAME=${POD_NAME:0:24}
NS="${RELEASE_NAMESPACE:-default}"
POD_NAME="${RELEASE_NAME:-mongo}-mongodb-replicaset"
for i in `seq 0 2`; do
pod="${POD_NAME}-$i"
kubectl exec --namespace $NS $pod -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"' | grep '"ismaster" : true'
if [ $? -eq 0 ]; then
echo "Found master: $pod"
MASTER=$pod
break
fi
done
kubectl exec --namespace $NS $MASTER -- /usr/bin/mongo --eval='printjson(db.test.insert({"status": "success"}))'
kubectl exec "$POD_NAME-0" -- /usr/bin/mongo --eval="printjson(db.test.insert({\"status\": \"success\"}))"
# TODO: find maximum duration to wait for slaves to be up-to-date with master.
sleep 2
kubectl exec "$POD_NAME-1" -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)"
kubectl exec "$POD_NAME-2" -- /usr/bin/mongo --eval="rs.slaveOk(); db.test.find().forEach(printjson)"
for i in `seq 0 2`; do
pod="${POD_NAME}-$i"
if [[ $pod != $MASTER ]]; then
echo "Reading from slave: $pod"
kubectl exec --namespace $NS $pod -- /usr/bin/mongo --eval='rs.slaveOk(); db.test.find().forEach(printjson)'
fi
done
+47 -19
View File
@@ -1,21 +1,49 @@
# Default values for mongodb.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value
replicaSet: rs0
replicas: 3
port: 27017
Name: mongodb-replicaset
PeerPort: 27017
Component: "mongodb"
ReplicaSet: "rs0"
ClusterRole: "shardsvr"
Replicas: 3
Image: "mongo"
ImageTag: "3.2"
InstallImage: "gcr.io/google_containers/mongodb-install"
InstallImageTag: "0.3"
ImagePullPolicy: "Always"
Cpu: "100m"
Memory: "512Mi"
Storage: "10Gi"
StorageClass: generic
# Specs for the Docker image for the init container that establishes the replica set
installImage:
name: gcr.io/google_containers/mongodb-install
tag: 0.3
pullPolicy: IfNotPresent
# Specs for the MongoDB image
image:
name: mongo
tag: 3.2
pullPolicy: IfNotPresent
# Annotations to be added to MongoDB pods
podAnnotations: {}
resources: {}
# limits:
# cpu: 100m
# memory: 512Mi
# requests:
# cpu: 100m
# memory: 512Mi
persistentVolume:
enabled: true
## If defined, volume.beta.kubernetes.io/storage-class: <storageClass>
## Default: volume.alpha.kubernetes.io/storage-class: default
##
# storageClass: fast
accessModes:
- ReadWriteOnce
size: 10Gi
annotations: {}
# Annotations to be added to the service
serviceAnnotations: {}
# Entries for the MongoDB config file
configmap:
storage:
dbPath: /data/db
net:
port: 27017
replication:
replSetName: rs0