github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-02-14 17:49:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
01c374d0a47686f6813e86b35721044464444771
container.training/slides/k8s/M6-START-a-company-scenario.md
Ludovic Piot bc6100301e 📝 Add monitoring stack install
2025-06-12 20:05:14 +02:00

9.7 KiB
Raw Blame History

Kubernetes in production —
an end-to-end example

  • Previous training modules focused on individual topics

    (e.g. RBAC, network policies, CRDs, Helm...)

  • We will now show how to put everything together to deploy apps in production

    (dealing with typical challenges like: multiple apps, multiple teams, multiple clusters...)

  • Our first challenge will be to pick and choose which components to use

    (among the vast Cloud Native Landscape)

  • We'll start with a basic Kubernetes cluster (on cloud or on premises)

  • We'll and enhance it by adding features one at a time

The cast

There are 3 teams in our company:

  • ⚙️OPS is the platform engineering team

    • they're responsible for building and configuring Kubernetes clusters

  • the 🎸ROCKY team develops and manages the 🎸ROCKY app

    • that app manages a collection of rock & pop albums

    • it's deployed with plain YAML manifests

  • the 🎬MOVY team develops and manages the 🎬MOVY app

    • that app manages a collection of movie soundtrack albums

    • it's deployed with Helm charts

Code and team organization

  • 🎸ROCKY and 🎬MOVY reside in separate git repositories

  • Each team can write code, build package, and deploy their applications:

    • independently
      (= without having to worry about what's happening in the other repo)

    • autonomously
      (= without having to synchronize or obtain privileges from another team)

Cluster organization

The ⚙️OPS team manages 2 Kubernetes clusters:

  • ☁️CLOUDY: managed cluster from a public cloud provider

  • 🤘METAL: custom-built cluster installed on bare Linux servers

Let's see the differences between these clusters.

☁️CLOUDY cluster

  • Managed cluster from a public cloud provider ("Kubernetes-as-a-Service")

  • HA control plane deployed and managed by the cloud provider

  • Two worker nodes (potentially with cluster autoscaling)

  • Usually comes pre-installed with some basic features

    (e.g. metrics-server, CNI, CSI, sometimes an ingress controller)

  • Requires extra components to be production-ready

    (e.g. Flux or other gitops pipeline, observability...)

  • Example: Scaleway Kapsule (but many other KaaS options are available)

🤘METAL cluster

  • Custom-built cluster installed on bare Linux servers

  • HA control plane deployed and managed by the ⚙️OPS team

  • 3 nodes

    • in our example, the nodes will run both the control plane and our apps

    • it is more typical to use dedicated control plane nodes
      (example: 3 control plane nodes + at least 3 worker nodes)

  • Comes with even less pre-installed components than ☁️CLOUDY

    (requiring more work from our ⚙️OPS team)

  • Example: we'll use k0s (but many other distros are available)

⚗️TEST and 🏭PROD

  • The ⚙️OPS team creates 2 environments for each dev team

    (⚗️TEST and 🏭PROD)

  • These environments exist on both clusters

    (meaning 2 apps × 2 clusters × 2 envs = 8 envs total)

  • The setup for each env and cluster should follow DRY principles

    (to ensure configurations are consistent and minimize maintenance)

  • Each cluster and each env has its own lifecycle

    (= it should be possible to deploy, add an extra components/feature...
    on one env without impacting the other)

Multi-tenancy

Both 🎸ROCKY and 🎬MOVY teams should use dedicated "tenants" on each cluster/env

  • the 🎸ROCKY team should be able to deploy, upgrade and configure its app within its dedicated namespace without anybody else involved

  • and the same for 🎬MOVY

  • neither team's deployments might interfere with the other, maintaining a clean and conflict-free environment

Application overview

  • Both dev teams are working on an app to manage music albums

  • This app is mostly based on a Spring framework demo called spring-music

  • This lab uses a dedicated fork container.training-spring-music:

    • with 2 branches dedicated to the 🎸ROCKY and 🎬MOVY teams

  • The app architecture consists of 2 tiers:

    • a Java/Spring Web app
    • a PostgreSQL database

📂 specific file: application.yaml

This is where we configure the application to connect to the PostgreSQL database.

.lab[

🔍 Location: /src/main/resources/application.yml

]

PROFILE=postgres env var is set in docker-compose.yaml file, for example…

📂 specific file: AlbumRepositoryPopulator.java

This is where the album collection is initially loaded from the file album.json

.lab[

🔍 Location: /src/main/java/org/cloudfoundry/samples/music/repositories/AlbumRepositoryPopulator.java

]

🚚 How to deploy?

The ⚙️OPS team offers 2 deployment strategies that dev teams can use autonomously:

  • 🎸ROCKY uses a Flux GitOps workflow based on regular Kubernetes YAML resources

  • 🎬MOVY uses a Flux GitOps workflow based on Helm charts

🍱 What features?

The ⚙️OPS team aims to provide clusters offering the following features to its users:

  • a network stack with efficient workload isolation

  • ingress and load-balancing capabilites

  • an enterprise-grade monitoring solution for real-time insights

  • automated policy rule enforcement to control Kubernetes resources requested by dev teams

🌰 In a nutshell

  • 3 teams: ⚙️OPS, 🎸ROCKY, 🎬MOVY

  • 2 clusters: ☁️CLOUDY, 🤘METAL

  • 2 envs per cluster and per dev team: ⚗️TEST, 🏭PROD

  • 2 Web apps Java/Spring + PostgreSQL: one for pop and rock albums, another for movie soundtrack albums

  • 2 deployment strategies: regular YAML resources + Kustomize, Helm charts

💻 Flux is used both

  • to operate the clusters
  • and to manage the GitOps deployment workflows

What our scenario might look like…

%%{init:
    {
      "theme": "default",
      "gitGraph": {
        "mainBranchName": "OPS",
        "mainBranchOrder": 0
      }
    }
}%%
gitGraph
    commit id:"0" tag:"start"
    branch ROCKY order:4
    branch MOVY order:5
    branch YouRHere order:6

    checkout YouRHere
    commit id:'x'
    checkout OPS
    merge YouRHere id:'YOU ARE HERE'

    checkout OPS
    commit id:'Flux install on CLOUDY cluster' tag:'T01'
    branch TEST-env order:1
    commit id:'FLUX install on TEST' tag:'T02' type: HIGHLIGHT

    checkout OPS
    commit id:'Flux config. for TEST tenant' tag:'T03'
    commit id:'namespace isolation by RBAC'
    checkout TEST-env
    merge OPS id:'ROCKY tenant creation' tag:'T04'

    checkout OPS
    commit id:'ROCKY deploy. config.' tag:'R01'

    checkout TEST-env
    merge OPS id:'TEST ready to deploy ROCKY' type: HIGHLIGHT tag:'R02'

    checkout ROCKY
    commit id:'ROCKY' tag:'v1.0.0'

    checkout TEST-env
    merge ROCKY tag:'ROCKY v1.0.0'

    checkout OPS
    commit id:'Ingress-controller config.' tag:'T05'
    checkout TEST-env
    merge OPS id:'Ingress-controller install' type: HIGHLIGHT tag:'T06'

    checkout OPS
    commit id:'ROCKY patch for ingress config.' tag:'R03'
    checkout TEST-env
    merge OPS id:'ingress config. for ROCKY app'

    checkout ROCKY
    commit id:'blue color' tag:'v1.0.1'
    checkout TEST-env
    merge ROCKY tag:'ROCKY v1.0.1'

    checkout ROCKY
    commit id:'pink color' tag:'v1.0.2'
    checkout TEST-env
    merge ROCKY tag:'ROCKY v1.0.2'

    checkout OPS
    commit id:'FLUX config for MOVY deployment' tag:'M01'
    checkout TEST-env
    merge OPS id:'FLUX ready to deploy MOVY' type: HIGHLIGHT tag:'M02'

    checkout MOVY
    commit id:'MOVY' tag:'v1.0.3'
    checkout TEST-env
    merge MOVY tag:'MOVY v1.0.3' type: REVERSE

    checkout OPS
    commit id:'Network policies'
    checkout TEST-env
    merge OPS type: HIGHLIGHT tag:'T07'

    checkout OPS
    commit id:'k0s install on METAL cluster' tag:'K01'
    commit id:'Flux config. for METAL cluster' tag:'K02'
    branch METAL_TEST-PROD order:3
    commit id:'ROCKY/MOVY tenants on METAL' type: HIGHLIGHT
    checkout OPS
    commit id:'Flux config. for OpenEBS' tag:'K03'
    checkout METAL_TEST-PROD
    merge OPS id:'openEBS on METAL' type: HIGHLIGHT

    checkout OPS
    commit id:'Prometheus install'
    checkout TEST-env
    merge OPS type: HIGHLIGHT

    checkout OPS
    commit id:'Kyverno install'
    commit id:'Kyverno rules'
    checkout TEST-env
    merge OPS type: HIGHLIGHT

    checkout OPS
    commit id:'Flux config. for PROD tenant' tag:'P01'
    branch PROD-env order:2
    commit id:'ROCKY tenant on PROD'
    checkout OPS
    commit id:'ROCKY patch for PROD' tag:'R04'
    checkout PROD-env
    merge OPS id:'PROD ready to deploy ROCKY' type: HIGHLIGHT
    checkout PROD-env
    merge ROCKY tag:'ROCKY v1.0.2'

    checkout MOVY
    commit id:'MOVY HELM chart' tag:'M03'
    checkout TEST-env
    merge MOVY tag:'MOVY v1.0'
Reference in New Issue View Git Blame Copy Permalink