github/container.training

Fork 0

mirror of https://github.com/jpetazzo/container.training.git synced 2026-05-17 14:27:11 +00:00

Files

Jerome Petazzoni 524d6e4fc1 Minor updates to load balancing example

2018-09-28 18:31:39 -05:00

12 KiB

Raw Blame History

Managing configuration

Some applications need to be configured (obviously!)
There are many ways for our code to pick up configuration:
- command-line arguments
- environment variables
- configuration files
- configuration servers (getting configuration from a database, an API...)
- ... and more (because programmers can be very creative!)
How can we do these things with containers and Kubernetes?

Passing configuration to containers

There are many ways to pass configuration to code running in a container:
- baking it in a custom image
- command-line arguments
- environment variables
- injecting configuration files
- exposing it over the Kubernetes API
- configuration servers
Let's review these different strategies!

Baking custom images

Put the configuration in the image

(it can be in a configuration file, but also ENV or CMD actions)
It's easy! It's simple!
Unfortunately, it also has downsides:
- multiplication of images
- different images for dev, staging, prod ...
- minor reconfigurations require a whole build/push/pull cycle
Avoid doing it unless you don't have the time to figure out other options

Command-line arguments

Pass options to args array in the container specification

Example (source):

    args: 
      - "--data-dir=/var/lib/etcd"
      - "--advertise-client-urls=http://127.0.0.1:2379"
      - "--listen-client-urls=http://127.0.0.1:2379"
      - "--listen-peer-urls=http://127.0.0.1:2380"
      - "--name=etcd"

The options can be passed directly to the program that we run ...

... or to a wrapper script that will use them to e.g. generate a config file

Command-line arguments, pros & cons

Works great when options are passed directly to the running program

(otherwise, a wrapper script can work around the issue)
Works great when there aren't too many parameters

(to avoid a 20-lines args array)
Requires documentation and/or understanding of the underlying program

("which parameters and flags do I need, again?")
Well-suited for mandatory parameters (without default values)
Not ideal when we need to pass a real configuration file anyway

Environment variables

Pass options through the env map in the container specification

Example:

    env:
    - name: ADMIN_PORT
      value: "8080"
    - name: ADMIN_AUTH
      value: Basic
    - name: ADMIN_CRED
      value: "admin:0pensesame!"

.warning[value must be a string! Make sure that numbers and fancy strings are quoted.]

🤔 Why this weird {name: xxx, value: yyy} scheme? It will be revealed soon!

The downward API

In the previous example, environment variables have fixed values
We can also use a mechanism called the downward API
The downward API allows to expose pod or container information
- either through special files (we won't show that for now)
- or through environment variables
The value of these environment variables is computed when the container is started
Remember: environment variables won't (can't) change after container start
Let's see a few concrete examples!

Exposing the pod's namespace

    - name: MY_POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace

Useful to generate FQDN of services

(in some contexts, a short name is not enough)

For instance, the two commands should be equivalent:

curl api-backend
curl api-backend.$MY_POD_NAMESPACE.svc.cluster.local

Exposing the pod's IP address

    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

Useful if we need to know our IP address

(we could also read it from eth0, but this is more solid)

Exposing the container's resource limits

    - name: MY_MEM_LIMIT
      valueFrom:
        resourceFieldRef:
          containerName: test-container
          resource: limits.memory

Useful for runtimes where memory is garbage collected
Example: the JVM

(the memory available to the JVM should be set with the -Xmx flag)
Best practice: set a memory limit, and pass it to the runtime

(see this blog post for a detailed example)

More about the downward API

This documentation page tells more about these environment variables
And this one explains the other way to use the downward API

(through files that get created in the container filesystem)

Environment variables, pros and cons

Works great when the running program expects these variables
Works great for optional parameters with reasonable defaults

(since the container image can provide these defaults)
Sort of auto-documented

(we can see which environment variables are defined in the image, and their values)
Can be (ab)used with longer values ...
... You can put an entire Tomcat configuration file in an environment ...
... But should you?

(Do it if you really need to, we're not judging! But we'll see better ways.)

Injecting configuration files

Sometimes, there is no way around it: we need to inject a full config file
Kubernetes provides a mechanism for that purpose: configmaps
A configmap is a Kubernetes resource that exists in a namespace
Conceptually, it's a key/value map

(values are arbitrary strings)
We can think about them in (at least) two different ways:
- as holding entire configuration file(s)
- as holding individual configuration parameters

Note: to hold sensitive information, we can use "Secrets", which are another type of resource behaving very much like configmaps. We'll cover them just after!

Configmaps storing entire files

In this case, each key/value pair corresponds to a configuration file
Key = name of the file
Value = content of the file
There can be one key/value pair, or as many as necessary

(for complex apps with multiple configuration files)

Examples:

# Create a configmap with a single key, "app.conf"
kubectl create configmap my-app-config --from-file=app.conf
# Create a configmap with a single key, "app.conf" but another file
kubectl create configmap my-app-config --from-file=app.conf=app-prod.conf
# Create a configmap with multiple keys (one per file in the config.d directory)
kubectl create configmap my-app-config --from-file=config.d/

Configmaps storing individual parameters

In this case, each key/value pair corresponds to a parameter
Key = name of the parameter
Value = value of the parameter

Examples:

# Create a configmap with two keys
kubectl create cm my-app-config \
    --from-literal=foreground=red \
    --from-literal=background=blue

# Create a configmap from a file containing key=val pairs
kubectl create cm my-app-config \
    --from-env-file=app.conf

Exposing configmaps to containers

Configmaps can be exposed as plain files in the filesystem of a container
- this is achieved by declaring a volume and mounting it in the container
- this is particularly effective for configmaps containing whole files
Configmaps can be exposed as environment variables in the container
- this is achieved with the downward API
- this is particularly effective for configmaps containing individual parameters
Let's see how to do both!

Passing a configuration file with a configmap

We will start a load balancer powered by HAProxy
We will use the official haproxy image
It expects to find its configuration in /usr/local/etc/haproxy/haproxy.cfg
We will provide a simple HAproxy configuration, k8s/haproxy.cfg
It listens on port 80, and load balances connections between IBM and Google

Creating the configmap

.exercise[

Go to the k8s directory in the repository:
```
cd ~/container.training/k8s
```
Create a configmap named haproxy and holding the configuration file:
```
kubectl create configmap haproxy --from-file=haproxy.cfg
```
Check what our configmap looks like:
```
kubectl get configmap haproxy -o yaml
```

]

Using the configmap

We are going to use the following pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: haproxy
spec:
  volumes:
  - name: config
    configMap:
      name: haproxy
  containers:
  - name: haproxy
    image: haproxy
    volumeMounts:
    - name: config
      mountPath: /usr/local/etc/haproxy/

Using the configmap

The resource definition from the previous slide is in k8s/haproxy.yaml

.exercise[

Create the HAProxy pod:

kubectl apply -f ~/container.training/k8s/haproxy.yaml

Check the IP address allocated to the pod:

kubectl get pod haproxy -o wide
IP=$(kubectl get pod haproxy -o json | jq -r .status.podIP)

]

Testing our load balancer

The load balancer will send:
- half of the connections to Google
- the other half to IBM

.exercise[

Access the load balancer a few times:
```
curl $IP
curl $IP
curl $IP
```

]

We should see connections served by Google, and others served by IBM.
(Each server sends us a redirect page. Look at the URL that they send us to!)

Exposing configmaps with the downward API

We are going to run a Docker registry on a custom port
By default, the registry listens on port 5000
This can be changed by setting environment variable REGISTRY_HTTP_ADDR
We are going to store the port number in a configmap
Then we will expose that configmap to a container environment variable

Creating the configmap

.exercise[

Our configmap will have a single key, http.addr:

kubectl create configmap registry --from-literal=http.addr=0.0.0.0:80

Check our configmap:
```
kubectl get configmap registry -o yaml
```

]

Using the configmap

We are going to use the following pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: registry
spec:
  containers:
  - name: registry
    image: registry
    env:
    - name: REGISTRY_HTTP_ADDR
      valueFrom:
        configMapKeyRef:
          name: registry
          key: http.addr

Using the configmap

The resource definition from the previous slide is in k8s/registry.yaml

.exercise[

Create the registry pod:

kubectl apply -f ~/container.training/k8s/registry.yaml

Check the IP address allocated to the pod:

kubectl get pod registry -o wide
IP=$(kubectl get pod registry -o json | jq -r .status.podIP)

Confirm that the registry is available on port 80:
```
curl $IP/v2/_catalog
```

]

Passwords, tokens, sensitive information

For sensitive information, there is another special resource: Secrets
Secrets and Configmaps work almost the same way

(we'll expose the differences on the next slide)
The intent is different, though:

"You should use secrets for things which are actually secret like API keys, credentials, etc., and use config map for not-secret configuration data."

"In the future there will likely be some differentiators for secrets like rotation or support for backing the secret API w/ HSMs, etc."

(Source: the author of both features)

Differences between configmaps and secrets

Secrets are base64-encoded when shown with kubectl get secrets -o yaml
- keep in mind that this is just encoding, not encryption
- it is very easy to automatically extract and decode secrets
Secrets can be encrypted at rest
With RBAC, we can authorize a user to access configmaps, but not secrets

(since they are two different kinds of resources)

12 KiB Raw Blame History

Managing configuration

Passing configuration to containers

Baking custom images

Command-line arguments

Command-line arguments, pros & cons

Environment variables

The downward API

Exposing the pod's namespace

Exposing the pod's IP address

Exposing the container's resource limits

More about the downward API

Environment variables, pros and cons

Injecting configuration files

Configmaps storing entire files

Configmaps storing individual parameters

Exposing configmaps to containers

Passing a configuration file with a configmap

Creating the configmap

Using the configmap

Using the configmap

Testing our load balancer

Exposing configmaps with the downward API

Creating the configmap

Using the configmap

Using the configmap

Passwords, tokens, sensitive information

Differences between configmaps and secrets

12 KiB

Raw Blame History