github/container.training
1
0
Fork 0
mirror of https://github.com/jpetazzo/container.training.git synced 2026-07-18 20:39:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'BretFisher-readme-updates'

Browse Source
This commit is contained in:
Jérôme Petazzoni
2016-09-29 00:25:46 +02:00
parent c8c745459c 1d02ddf271
commit bcd2433fa4
4 changed files with 240 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files

155
README.md
View File

@@ -1,4 +1,4 @@
# Orchestration at scale(s)
# Orchestration at Scale with Docker
This is the material for the "Docker orchestration workshop"
written and delivered by Jérôme Petazzoni (and possibly others)
@@ -14,18 +14,43 @@ at multiple conferences and events like:
- Zenika, Paris (2016, February)
- Container Solutions, Amsterdam (2016, February)
## Content
## Slides
- Chapter 1: Getting Started: running apps with docker-compose
- Chapter 2: Scaling out with Swarm Mode
- Chapter 3: Operating the Swarm (networks, updates, logging)
- Chapter 4: Deeper in Swarm (stateful services, scripting, DAB's)
The slides are in the `docs` directory.
## How This Repo is Organized
To view them locally open `docs/index.html` in your browser.
- **dockercoins**
- Sample App: compose files and source code for the dockercoins sample apps
used throughout the workshop
- **docs**
- Slide Deck: presentation slide deck, works out-of-box with GitHub Pages,
uses https://remarkjs.com
- **prepare-local**
- untested scripts for automating the creation of local virtualbox VM's
(could use your help validating)
- **prepare-vms**
- scripts for automating the creation of AWS instances for students
To view them online open https://jpetazzo.github.io/orchestration-workshop/ in your browser.
## Slide Deck
## Sample code
- The slides are in the `docs` directory.
- To view them locally open `docs/index.html` in your browser. It works
offline too.
- To view them online open https://jpetazzo.github.io/orchestration-workshop/
in your browser.
- When you fork this repo, be sure GitHub Pages is enabled in repo Settings
for "master branch /docs folder" and you'll have your own website for them.
- They use https://remarkjs.com to allow simple markdown in a html file that
remark will transform into a presentation in the browser.
The sample app is in the `dockercoins` directory.
## Sample App: Dockercoins!
The sample app is in the `dockercoins` directory. It's used during all chapters
for explaining different concepts of orchestration.
To see it in action:
@@ -33,55 +58,97 @@ To see it in action:
- this will build and start all the services
- the web UI will be available on port 8000
## Running the Workshop
## Running the workshop
### General timeline of planning a workshop
WARNING: those instructions are incomplete. Consider
them as notes quickly drafted on a napkin rather than
proper documentation!
- Fork repo and run through slides, doing the hands-on to be sure you
understand the different `dockercoins` repo's and the steps we go through to
get to a full Swarm Mode cluster of many containers. You'll update the first
few slides and last slide at a minimum, with your info.
- Your docs directory can use GitHub Pages.
- This workshop expects 5 servers per student. You can get away with as little
as 2 servers per student, but you'll need to change the slide deck to
accommodate. More servers = more fun.
- If you have more then ~20 students, try to get an assistant (TA) to help
people with issues, so you don't have to stop the workshop to help someone
with ssh etc.
- AWS is our most tested process for generating student machines. In
`prepare-vms` you'll find scripts to create EC2 instances, install docker,
pre-pull images, and even print "cards" to place at each students seat with
IP's and username/password.
- Test AWS Scripts: Be sure to test creating *all* your needed servers a week
before workshop (just for a few minutes). You'll likely hit AWS limits in the
region closest to your class, and it sometimes takes days to get AWS to raise
those limits with a support ticket.
- Create a https://gitter.im chat room for your workshop and update slides
with url. Also useful for TA to monitor this during workshop. You can use it
before/after to answer questions, and generally works as a better answer then
"email me that question".
- If you can send an email to students ahead of time, mention how they should
get SSH, and test that SSH works. If they can `ssh github.com` and get
`permission denied (publickey)` then they know it worked, and SSH is properly
installed and they don't have anything blocking it. SSH and a browser are all
they need for class.
- Typically you create the servers the day before or morning of workshop, and
leave them up the rest of day after workshop. If creating hundreds of servers,
you'll likely want to run all these `trainer` commands from a dedicated
instance you have in same region as instances you want to create. Much faster
this way if you're on poor internet. Also, create 2 sets of servers for
yourself, and use one during workshop and the 2nd is a backup.
- Remember you'll need to print the "cards" for students, so you'll need to
create instances while you have a way to print them.
### Things That Could Go Wrong
- Creating AWS instances ahead of time, and you hit its limits in region and
didn't plan enough time to wait on support to increase your limits. :(
- Students have technical issues during workshop. Can't get ssh working,
locked-down computer, host firewall, etc.
- Horrible wifi, or ssh port TCP/22 not open on network! If wifi sucks you
can try using MOSH https://mosh.org which handles SSH over UDP. TMUX can also
prevent you from loosing your place if you get disconnected from servers.
https://tmux.github.io
- Forget to print "cards" and cut them up for handing out IP's.
- Forget to have fun and focus on your students!
### Creating the VMs
I use the `trainctl` script from the `docker-fundamentals`
repository. Sorry if you don't have that!
`prepare-vms/trainer` is the script that gets you most of what you need for
setting up instances. See
[prepare-vms/README.md](/jpetazzo/orchestration-workshop/tree/master/prepare-vms)
for all the info on tools and scripts.
After starting the VMs, use the `trainctl ips` command
to dump the list of IP addresses into a file named `ips.txt`.
### Content for Different Workshop Durations
With all the slides, this workshop is a full day long. If you need to deliver
it in shorter timelines, here's some recommendations on what to cut out. You
can replace `---` with `???` which will hide slides. Or leave them there and
add something like `(EXTRA CREDIT)` to title so students can still view the
content but you also know to skip during presentation.
### Generating the printed cards
#### 3 Hour Version
- Put `ips.txt` file in `prepare-vms` directory.
- Generate HTML file.
- Open it in Chrome.
- Transform to PDF.
- Print it.
- Limit time on debug tools, maybe skip a few. *"Chapter 1:
Identifying bottlenecks"*
- Limit time on Compose, try to have them building the Swarm Mode by 30
minutes in
- Skip most of Chapter 3, Centralized Logging and ELK
- Skip most of Chapter 4, but keep stateful services and DAB's if possible
- Mention what DAB's are, but make this part optional in case you run out
of time
#### 2 Hour Version
### Deploying your SSH key to all the machines
- Make sure that you have SSH keys loaded (`ssh-add -l`).
- Source `rc`.
- Run `pcopykey`.
### Installing extra packages
- Source `postprep.rc`.
(This will install a few extra packages, add entries to
/etc/hosts, generate SSH keys, and deploy them on all hosts.)
### Final touches
- Set two groups of machines for instructor's use.
- You will use the first group during the workshop.
- The second group will run a web server with the slides.
- Log into the first machine of the second group.
- Git clone this repo.
- Put up the web server as instructed above.
- Use cli53 to add an A record for e.g. `view.dckr.info`.
- Skip all the above, and:
- Skip the story arc of debugging dockercoins all together, skipping the
troubleshooting tools. Just focus on getting them from single-host to
multi-host and multi-container.
- Goal is first 30min on intro and Docker Compose and what dockercoins is,
and getting it up on one node in docker-compose.
- Next 60-75 minutes is getting dockercoins in Swarm Mode services across
servers. Big Win.
- Last 15-30 minutes is for stateful services, DAB files, and questions.
# Problems? Bugs? Questions?

140
prepare-vms/README.md
View File

@@ -1,67 +1,113 @@
# Trainer tools to prepare VMs for Docker workshops
# Trainer tools to create and prepare VMs for Docker workshops on AWS
## 1. Prerequisites
## Prerequisites
* [Docker](https://docs.docker.com/engine/installation/)
* [Docker Compose](https://docs.docker.com/compose/install/)
- [Docker](https://docs.docker.com/engine/installation/)
- [Docker Compose](https://docs.docker.com/compose/install/)
## 2. Clone the repo
## General Workflow
- fork/clone repo
- set required environment variables for AWS
- create your own setting file from `settings/example.yaml`
- run `./trainer` commands to create instances, install docker, setup each users environment in node1, other management tasks
- run `./trainer cards` command to generate PDF for printing handouts of each users host IP's and login info
## Clone/Fork the Repo, and Build the Tools Image
The Docker Compose file here is used to build a image with all the dependencies to run the `./trainer` commands and optional tools. Each run of the script will check if you have those dependencies locally on your host, and will only use the container if you're [missing a dependency](jpetazzo/orchestration-workshop/blob/master/prepare-vms/trainer#L5).
$ git clone https://github.com/jpetazzo/orchestration-workshop.git
$ cd orchestration-workshop/prepare-vms
$ docker-compose build
$ ./trainer <commands> # See "Summary of commands" section below
## 3. Preparing the environment
## Preparing to Run `./trainer`
Required environment variables:
### Required AWS Permissions/Info
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_DEFAULT_REGION`
- Initial assumptions are you're using a root account. If you'd like to use a IAM user, it will need `AmazonEC2FullAccess` and `IAMReadOnlyAccess`.
- Using a non-default VPC or Security Group isn't supported out of box yet, but until then you can [customize the `trainer-cli` script](jpetazzo/orchestration-workshop/blob/master/prepare-vms/scripts/trainer-cli#L396-L401).
- These instances will assign the default VPC Security Group, which does not open any ports from Internet by default. So you'll need to add Inbound rules for `SSH | TCP | 22 | 0.0.0.0/0` and `Custom TCP Rule | TCP | 8000 - 8002 | 0.0.0.0/0`, or run `./trainer opensg` which opens up all ports.
### 4. Update settings.yaml
### Required Environment Variables
Then pass `settings/YOUR_WORKSHOP_NAME-settings.yaml` as an argument to `deploy`, `cards`, etc.
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_DEFAULT_REGION`
## Usage
### Update/copy `settings/example.yaml`
### Summary of commands
Then pass `settings/YOUR_WORKSHOP_NAME-settings.yaml` as an argument to `trainer deploy`, `trainer cards`, etc.
The `trainer` script can be executed directly.
./trainer cards 2016-09-28-00-33-bret settings/orchestration.yaml
Summary of steps to launch a batch of instances for a workshop:
## `./trainer` Usage
* Export the environment variables needed by the AWS CLI (see **2. Preparing the environment** above)
* `./trainer start N` (where `N` is the number of AWS instances to create)
* `./trainer list` to view the list of tags
* `./trainer list TAG` to view the instances with a given `TAG`
* `./trainer deploy TAG settings/somefile.yaml` to run `scripts/postprep.rc` via parallel-ssh
* `./trainer pull-images TAG` to pre-pull a bunch of Docker images to the instances
* `./trainer test TAG`
* `./trainer cards TAG settings/somefile.yaml` to generate a PDF and an HTML file you can print and cut to hand out cards with connection information to attendees
```
./trainer <command> [n-instances|tag] [settings/file.yaml]
`./trainer` will run locally if all its dependencies are fulfilled; otherwise it will run in a Docker container.
Core commands:
start n Start n instances
list [TAG] If a tag is provided, list its VMs. Otherwise, list tags.
deploy TAG Deploy all instances with a given tag
pull-images TAG Pre-pull docker images. Run only after deploying.
stop TAG Stop and delete instances tagged TAG
It will check for the necessary environment variables. Then, if all its dependencies are installed
locally, it will execute `trainer-cli`. If not, it will look for a local Docker image
tagged `preparevms_prepare-vms` (created automatically when you run `docker-compose build`).
If found, it will run in a container. If not found, the user will be prompted to
either install the missing dependencies or run `docker-compose build`.
Extras:
ips TAG List all IPs of instances with a given tag (updates ips.txt)
ids TAG/TOKEN List all instance IDs with a given tag
shell Get a shell in the trainer container
status TAG Print information about this tag and its VMs
tags List all tags (per-region)
retag TAG/TOKEN TAG Retag instances with a new tag
## Detailed usage
Beta:
ami Look up Amazon Machine Images
cards FILE Generate cards
opensg Modify AWS security groups
```
### Start some VMs
### Summary of What `./trainer` Does For You
$ ./trainer start 10
- Used to manage bulk AWS instances for you without needing to use AWS cli or gui.
- Can manage multiple "tags" or groups of instances, which are tracked in `prepare-vms/tags/`
- Can also create PDF/HTML for printing student info for instance IP's and login.
- The `./trainer` script can be executed directly.
- It will run locally if all its dependencies are fulfilled; otherwise it will run in the Docker container you created with `docker-compose build` (preparevms_prepare-vms).
- During `start` it will add your default local SSH key to all instances under the `ubuntu` user.
- During `deploy` it will create the `docker` user with password `training`, which is printing on the cards for students. For now, this is hard coded.
A few things will happen:
### Example Steps to Launch a Batch of Instances for a Workshop
* Your local SSH key will be synced
* AWS instances will be created and tagged
* A directory will be created
- Export the environment variables needed by the AWS CLI (see **Required Environment Variables** above)
- Run `./trainer start N` Creates `N` EC2 instances
- Your local SSH key will be synced to instances under `ubuntu` user
- AWS instances will be created and tagged based on date, and IP's stored in `prepare-vms/tags/`
- Run `./trainer deploy TAG settings/somefile.yaml` to run `scripts/postprep.rc` via parallel-ssh
- If it errors or times out, you should be able to rerun
- Requires good connection to run all the parallel SSH connections, up to 100 parallel (ProTip: create dedicated management instance in same AWS region where you run all these utils from)
- Run `./trainer pull-images TAG` to pre-pull a bunch of Docker images to the instances
- Run `./trainer cards TAG settings/somefile.yaml` generates PDF/HTML files to print and cut and hand out to students
- *Have a great workshop*
- Run `./trainer stop TAG` to terminate instances.
Details below.
## Other Tools
### Deploying your SSH key to all the machines
- Make sure that you have SSH keys loaded (`ssh-add -l`).
- Source `rc`.
- Run `pcopykey`.
### Installing extra packages
- Source `postprep.rc`.
(This will install a few extra packages, add entries to
/etc/hosts, generate SSH keys, and deploy them on all hosts.)
## Even More Details
#### Sync of SSH keys
@@ -83,7 +129,7 @@ This ips.txt file will be created in the $TAG/ directory and a symlink will be p
If you create new VMs, the symlinked file will be overwritten.
## Deployment
#### Deployment
Instances can be deployed manually using the `deploy` command:
@@ -91,29 +137,29 @@ Instances can be deployed manually using the `deploy` command:
The `postprep.rc` file will be copied via parallel-ssh to all of the VMs and executed.
### Pre-pull images
#### Pre-pull images
$ ./trainer pull-images TAG
### Generate cards
#### Generate cards
$ ./trainer cards TAG settings/somefile.yaml
### List tags
#### List tags
$ ./trainer list
### List VMs
#### List VMs
$ ./trainer list TAG
This will print a human-friendly list containing some information about each instance.
### Stop and destroy VMs
#### Stop and destroy VMs
$ ./trainer stop TAG
## ToDo
* Don't write to bash history in system() in postprep
* compose, etc version inconsistent (int vs str)
- Don't write to bash history in system() in postprep
- compose, etc version inconsistent (int vs str)

BIN
prepare-vms/media/swarm.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 457 KiB

36
prepare-vms/settings/example.yaml Normal file
View File

@@ -0,0 +1,36 @@
# This file is passed by trainer-cli to scripts/ips-txt-to-html.py
workshop_name: Docker Orchestration
workshop_short_name: orchestration
repo: https://github.com/jpetazzo/orchestration-workshop
url: http://container.training/ # moreinfo link printed on cards
#engine_version: experimental.docker.com #extra features that may change/runaway
#engine_version: test.docker.com
engine_version: get.docker.com #prod release
compose_version: 1.8.1
machine_version: 0.8.2
swarm_version: 1.2.5
# for now these are hard coded in script, and only used for printing cards
instance_login: docker
instance_password: training
# 12 per page works well, but is quite small text
clustersize: 5 # Number of VMs per cluster
pagesize: 12 # Number of cards to print per page
background_image: https://raw.githubusercontent.com/jpetazzo/orchestration-workshop/master/prepare-vms/media/swarm.png
# To be printed on the cards:
blurb: >
Here is the connection information to your very own
{cluster_or_machine} for this {workshop_name} workshop. You can connect
to {this_or_each} VM with any SSH client.
Your {machine_is_or_machines_are}:
# {url} will be replaced by the script
footer: >
<p>For slides, chat and other useful links, see: </p>
<center>{url}</center>