github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-18 20:39:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Stateful Sets

Browse Source 
- explain the reason why we have stateful sets
- explain the relationship between volumes, persistent volumes,
  persistent volume claims, volume claim templates
- show how to run a Consul cluster with a stateful set
This commit is contained in:
Jerome Petazzoni
2018-09-02 08:51:03 -05:00
parent 59b7386b91
commit b158babb7f
3 changed files with 424 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
k8s/consul.yaml Normal file
View File

@@ -0,0 +1,62 @@
apiVersion: v1
kind: Service
metadata:
name: consul
spec:
ports:
- port: 8500
name: http
selector:
app: consul
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: consul
spec:
serviceName: consul
replicas: 3
selector:
matchLabels:
app: consul
template:
metadata:
labels:
app: consul
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- consul
topologyKey: kubernetes.io/hostname
terminationGracePeriodSeconds: 10
containers:
- name: consul
image: "consul:1.2.2"
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- "agent"
- "-bootstrap-expect=3"
- "-retry-join=consul-0.consul.$(NAMESPACE).svc.cluster.local"
- "-retry-join=consul-1.consul.$(NAMESPACE).svc.cluster.local"
- "-retry-join=consul-2.consul.$(NAMESPACE).svc.cluster.local"
- "-client=0.0.0.0"
- "-data-dir=/consul/data"
- "-server"
- "-ui"
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- consul leave

361
slides/k8s/statefulsets.md Normal file
View File

@@ -0,0 +1,361 @@
# Stateful sets
- Stateful sets are a type of resource in the Kubernetes API
(like pods, deployments, services...)
- They offer mechanisms to deploy scaled stateful applications
- At a first glance, they look like *deployments*:
- a stateful set defines a pod spec and a number of replicas *R*
- it will make sure that *R* copies of the pod are running
- that number can be changed while the stateful set is running
- updating the pod spec will cause a rolling update to happen
- But they also have some significant differences
---
## Stateful sets unique features
- Pods in a stateful set are numbered (from 0 to *R-1*) and ordered
- They are started and updated in order (from 0 to *R-1*)
- A pod is started (or updated) only when the previous one is ready
- They are stopped in reverse order (from *R-1* to 0)
- Each pod know its identity (i.e. which number it is in the set)
- Each pod can discover the IP address of the others easily
- The pods can have persistent volumes attached to them
🤔 Wait a minute ... Can't we already attach volumes to pods and deployments?
---
## Volumes and Persistent Volumes
- [Volumes](https://kubernetes.io/docs/concepts/storage/volumes/) are used for many purposes:
- sharing data between containers in a pod
- exposing configuration information and secrets to containers
- accessing storage systems
- The last type of volumes is known as a "Persistent Volume"
---
## Persistent Volumes types
- There are many [types of Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes) available:
- public cloud storage (GCEPersistentDisk, AWSElasticBlockStore, AzureDisk...)
- private cloud storage (Cinder, VsphereVolume...)
- traditional storage systems (NFS, iSCSI, FC...)
- distributed storage (Ceph, Clusterfs, Portworx...)
- Using a persistent volume requires:
- creating the volume out-of-band (outside of the Kubernetes API)
- referencing the volume in the pod description, with all its parameters
---
## Using a Persistent Volume
Here is a pod definition using an AWS EBS volume (that has to be created first):
```yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-using-my-ebs-volume
spec:
containers:
- image: ...
name: container-using-my-ebs-volume
volumeMounts:
- mountPath: /my-ebs
name: my-ebs-volume
volumes:
- name: my-ebs-volume
awsElasticBlockStore:
volumeID: vol-049df61146c4d7901
fsType: ext4
```
---
## Shortcomings of Persistent Volumes
- Their lifecycle (creation, deletion...) is managed outside of the Kubernetes API
(we can't just use `kubectl apply/create/delete/...` to manage them)
- If a Deployment uses a volume, all replicas end up using the same volume
- That volume must then support concurrent access
- some volumes do (e.g. NFS servers support multiple read/write access)
- some volumes support concurrent reads
- some volumes support concurrent access for colocated pods
- What we really need is a way for each replica to have its own volume
---
## Persistent Volume Claims
- To abstract the different types of storage, a pod can use a special volume type
- This type is a *Persistent Volume Claim*
- Using a Persistent Volume Claim is a two-step process:
- creating the claim
- using the claim in a pod (as if it were any other kind of volume)
- Between these two steps, something will happen behind the scenes:
- Kubernetes will associate an existing volume with the claim
- ... or dynamically create a volume if possible and necessary
---
## What's in a Persistent Volume Claim?
- At the very least, the claim should indicate:
- the size of the volume (e.g. "5 GiB")
- the access mode (e.g. "read-write by a single pod")
- In can also give extra details, like:
- which storage system to use (e.g. Portworx, EBS...)
- extra parameters for that storage system
e.g.: "replicate the data 3 times, and use SSD media"
---
## Using a Persistent Volume Claim
Here is the same definition as earlier, but using a PVC:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-using-a-claim
spec:
containers:
- image: ...
name: container-using-a-claim
volumeMounts:
- mountPath: /my-ebs
name: my-volume
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-claim
```
---
## Persistent Volume Claims and Stateful sets
- The pods in a stateful set can define a `volumeClaimTemplate`
- A `volumeClaimTemplate` will dynamically create one Persistent Volume Claim per pod
- Each pod will therefore have its own volume
- These volumes are numbered (like the pods)
- When updating the stateful set (e.g. image upgrade), each pod keeps its volume
- When pods get rescheduled (e.g. node failure), they keep their volume
(this requires a storage system that is not node-local)
- These volumes are not automatically deleted
(when the stateful set is scaled down or deleted)
---
## Stateful set recap
- A Stateful sets manages a number of identical pods
(like a Deployment)
- These pods are numbered, and started/upgraded/stopped in a specific order
- These pods are aware of their number
(e.g., #0 can decide to be the primary, and #1 can be secondary)
- These pods can find the IP addresses of the other pods in the set
(through a *headless service*)
- These pods can each have their own persistent storage
(Deployments cannot do that)
---
## Stateful sets in action
- We are going to deploy a Consul cluster with 3 nodes
- Consul is a highly-available key/value store
(like etcd or Zookeeper)
- One easy way to bootstrap a cluster is to tell each node:
- the addresses of other nodes
- how many nodes are expected (to know when quorum is reached)
---
## Bootstrapping a Consul cluster
*After reading the Consul documentation carefully (and/or asking around),
we figure out the minimal command-line to run our Consul cluster.*
```
consul agent -data=dir=/consul/data -client=0.0.0.0 -server -ui \
-bootstrap-expect=3 \
-retry-join=`X.X.X.X` \
-retry-join=`Y.Y.Y.Y`
```
- We need to replace X.X.X.X and Y.Y.Y.Y with the addresses of other nodes
- We can specify DNS names, but then they have to be FQDN
- It's OK to specify ourselves in the list as well
- We can therefore use the same command-line on all nodes (easier!)
---
## Discovering the addresses of other pods
- When a service is created for a stateful set, individual DNS entries are created
- These entries are constructed like this:
`<name-of-stateful-set>-<n>.<name-of-service>.<namespace>.svc.cluster.local`
- `<n>` is the number of the pod in the set (starting at zero)
- If we deploy Consul in the default namespace, the names could be:
- `consul-0.consul.default.svc.cluster.local`
- `consul-1.consul.default.svc.cluster.local`
- `consul-2.consul.default.svc.cluster.local`
---
## Putting it all together
- The file `k8s/consul.yaml` defines a service and a stateful set
- It has a few extra touches:
- the name of the namespace is injected through an environment variable
- a `podAntiAffinity` prevents two pods from running on the same node
- a `preStop` hook makes the pod leave the cluster when shutdown gracefully
This was inspired by this [excellent tutorial](https://github.com/kelseyhightower/consul-on-kubernetes) by Kelsey Hightower.
Some features from the original tutorial (TLS authentication between
nodes and encryption of gossip traffic) were removed for simplicity.
---
## Running our Consul cluster
- We'll use the provided YAML file
.exercise[
- Create the stateful set and associated service:
```bash
kubectl apply -f ~/container.training/k8s/consul.yaml
```
- Check the logs as the pods come up one after another:
```bash
stern consul
```
- Check the health of the cluster:
```bash
kubectl exec consul-0 consul members
```
]
---
## Caveats
- We haven't used a `volumeClaimTemplate` here
- That's because we don't have a storage provider yet
(except if you're running this on your own on a full-featured cluster)
- What happens if we lose a pod?
- a new pod gets rescheduled (with an empty state)
- the new pod tries to connect to the two others
- it will be accepted (after 1-2 minutes of instability)
- and it will retrieve the data from the other pods
---
## Failure modes
- What happens if we lose two pods?
- manual repair will be required
- we will need to instruct the remaining one to act solo
- then rejoin new pods
- What happens if we lose three pods? (aka all of them)
- we lose all the data (ouch)
- If we run Consul without persistent storage, backups are a good idea!

5
slides/new-content.yml
View File

@@ -12,10 +12,7 @@ chapters:
- k8s/build-with-docker.md
- k8s/build-with-kaniko.md
- k8s/configuration.md
- |
# Stateful sets
setting up consul with stateful sets
- k8s/statefulsets.md
- |
# Persistence