mirror of
https://github.com/jpetazzo/container.training.git
synced 2026-07-22 06:16:31 +00:00
a47c51618b
GCP quotas are fairly limited (on my account, I can only use 8 public IP addresses per zone, which means that I cannot deploy many public clusters in a single zone). I tried to use private clusters, but that causes other problems. This refactoring makes it possible to spread clusters across multiple zones. Since I have access to 20+ zones in Europe and 20+ zones in the US, this lets me create a lot of public clusters and simplifies the module quite a bit.
163 lines
4.3 KiB
Markdown
163 lines
4.3 KiB
Markdown
⚠️ This is work in progress. The UX needs to be improved,
|
|
and the docs could be better.
|
|
|
|
This directory contains a Terraform configuration to deploy
|
|
a bunch of Kubernetes clusters on various cloud providers,
|
|
using their respective managed Kubernetes products.
|
|
|
|
## With shell wrapper
|
|
|
|
This is the recommended use. It makes it easy to start N clusters
|
|
on any provider. It will create a directory with a name like
|
|
`tag-YYYY-MM-DD-HH-MM-SS-SEED-PROVIDER`, copy the Terraform configuration
|
|
to that directory, then create the clusters using that configuration.
|
|
|
|
1. One-time setup: configure provider authentication for the provider(s) that you wish to use.
|
|
|
|
- Digital Ocean:
|
|
```bash
|
|
doctl auth init
|
|
```
|
|
|
|
- Google Cloud Platform: you will need to create a project named `prepare-tf`
|
|
and enable the relevant APIs for this project (sorry, if you're new to GCP,
|
|
this sounds vague; but if you're familiar with it you know what to do; if you
|
|
want to change the project name you can edit the Terraform configuration)
|
|
|
|
- Linode:
|
|
```bash
|
|
linode-cli configure
|
|
```
|
|
|
|
- Oracle Cloud: FIXME
|
|
(set up `oci` through the `oci-cli` Python package)
|
|
|
|
- Scaleway: run `scw init`
|
|
|
|
2. Optional: set number of clusters, cluster size, and region.
|
|
|
|
By default, 1 cluster will be configured, with 2 nodes, and auto-scaling up to 5 nodes.
|
|
|
|
If you want, you can override these parameters, with the following variables.
|
|
|
|
```bash
|
|
export TF_VAR_how_many_clusters=5
|
|
export TF_VAR_min_nodes_per_pool=2
|
|
export TF_VAR_max_nodes_per_pool=4
|
|
export TF_VAR_location=xxx
|
|
```
|
|
|
|
The `location` variable is optional. Each provider should have a default value.
|
|
The value of the `location` variable is provider-specific. Examples:
|
|
|
|
| Provider | Example value | How to see possible values
|
|
|---------------|-------------------|---------------------------
|
|
| Digital Ocean | `ams3` | `doctl compute region list`
|
|
| Google Cloud | `europe-north1-a` | `gcloud compute zones list`
|
|
| Linode | `eu-central` | `linode-cli regions list`
|
|
| Oracle Cloud | `eu-stockholm-1` | `oci iam region list`
|
|
|
|
You can also specify multiple locations, and then they will be
|
|
used in round-robin fashion.
|
|
|
|
For example, with Google Cloud, since the default quotas are very
|
|
low (my account is limited to 8 public IP addresses per zone, and
|
|
my requests to increase that quota were denied) you can do the
|
|
following:
|
|
|
|
```bash
|
|
export TF_VAR_location=$(gcloud compute zones list --format=json | jq -r .[].name | grep ^europe)
|
|
```
|
|
|
|
Then when you apply, clusters will be created across all available
|
|
zones in Europe. (When I write this, there are 20+ zones in Europe,
|
|
so even with my quota, I can create 40 clusters.)
|
|
|
|
3. Run!
|
|
|
|
```bash
|
|
./run.sh <providername>
|
|
```
|
|
|
|
(If you don't specify a provider name, it will list available providers.)
|
|
|
|
4. Shutting down
|
|
|
|
Go to the directory that was created by the previous step (`tag-YYYY-MM...`)
|
|
and run `terraform destroy`.
|
|
|
|
You can also run `./clean.sh` which will destroy ALL clusters deployed by the previous run script.
|
|
|
|
## Without shell wrapper
|
|
|
|
Expert mode.
|
|
|
|
Useful to run steps sperarately, and/or when working on the Terraform configurations.
|
|
|
|
1. Select the provider you wish to use.
|
|
|
|
Go to the `source` directory and edit `main.tf`.
|
|
|
|
Change the `source` attribute of the `module "clusters"` section.
|
|
|
|
Check the content of the `modules` directory to see available choices.
|
|
|
|
2. Initialize the provider.
|
|
|
|
```bash
|
|
terraform init
|
|
```
|
|
|
|
3. Configure provider authentication.
|
|
|
|
See steps above, and add the following extra steps:
|
|
|
|
- Digital Coean:
|
|
```bash
|
|
export DIGITALOCEAN_ACCESS_TOKEN=$(grep ^access-token ~/.config/doctl/config.yaml | cut -d: -f2 | tr -d " ")
|
|
```
|
|
|
|
- Linode:
|
|
```bash
|
|
export LINODE_TOKEN=$(grep ^token ~/.config/linode-cli | cut -d= -f2 | tr -d " ")
|
|
```
|
|
|
|
4. Decide how many clusters and how many nodes per clusters you want.
|
|
|
|
5. Provision clusters.
|
|
|
|
```bash
|
|
terraform apply
|
|
```
|
|
|
|
6. Perform second stage provisioning.
|
|
|
|
This will install an SSH server on the clusters.
|
|
|
|
```bash
|
|
cd stage2
|
|
terraform init
|
|
terraform apply
|
|
```
|
|
|
|
7. Obtain cluster connection information.
|
|
|
|
The following command shows connection information, one cluster per line, ready to copy-paste in a shared document or spreadsheet.
|
|
|
|
```bash
|
|
terraform output -json | jq -r 'to_entries[].value.value'
|
|
```
|
|
|
|
8. Destroy clusters.
|
|
|
|
```bash
|
|
cd ..
|
|
terraform destroy
|
|
```
|
|
|
|
9. Clean up stage2.
|
|
|
|
```bash
|
|
rm stage2/terraform.tfstate*
|
|
```
|