diff --git a/slides/intro/Copy_On_Write.md b/slides/intro/Copy_On_Write.md index 687d08ba..a5b84b60 100644 --- a/slides/intro/Copy_On_Write.md +++ b/slides/intro/Copy_On_Write.md @@ -1,3 +1,339 @@ # Copy-on-write filesystems -FIXME +Container engines rely on copy-on-write to be able +to start containers quickly, regardless of their size. + +We will explain how that works, and review some of +the copy-on-write storage systems available on Linux. + +--- + +## What is copy-on-write? + +- Copy-on-write is a mechanism allowing to share data. + +- The data appears to be a copy, but is only + a link (or reference) to the original data. + +- The actual copy happens only when someone + tries to change the shared data. + +- Whoever changes the shared data ends up + using their own copy instead of the shared data. + +--- + +## A few metaphors + +-- + +- First metaphor: +
white board and tracing paper + +-- + +- Second metaphor: +
magic books with shadowy pages + +-- + +- Third metaphor: +
just-in-time house building + +--- + +## Copy-on-write is *everywhere* + +- Process creation with `fork()`. + +- Consistent disk snapshots. + +- Efficient VM provisioning. + +- And, of course, containers. + +--- + +## Copy-on-write and containers + +Copy-on-write is essential to give us "convenient" containers. + +- Creating a new container (from an existing image) is "free". + + (Otherwise, we would have to copy the image first.) + +- Customizing a container (by tweaking a few files) is cheap. + + (Adding a 1 KB configuration file to a 1 GB container takes 1 KB, not 1 GB.) + +- We can take snapshots, i.e. have "checkpoints" or "save points" + when building images. + +--- + +## AUFS overview + +- The original (legacy) copy-on-write filesystem used by first versions of Docker. + +- Combine multiple *branches* in a specific order. + +- Each branch is just a normal directory. + +- You generally have: + + - at least one read-only branch (at the bottom), + + - exactly one read-write branch (at the top). + + (But other fun combinations are possible too!) + +--- + +## AUFS operations: opening a file + +- With `O_RDONLY` - read-only access: + + - look it up in each branch, starting from the top + + - open the first one we find + +- With `O_WRONLY` or `O_RDWR` - write access: + + - if the file exists on the top branch: open it + + - if the file exists on another branch: "copy up" +
+ (i.e. copy the file to the top branch and open the copy) + + - if the file doesn't exist on any branch: create it on the top branch + +That "copy-up" operation can take a while if the file is big! + +--- + +## AUFS operations: deleting a file + +- A *whiteout* file is created. + +- This is similar to the concept of "tombstones" used in some data systems. + +``` + # docker run ubuntu rm /etc/shadow + + # ls -la /var/lib/docker/aufs/diff/$(docker ps --no-trunc -lq)/etc + total 8 + drwxr-xr-x 2 root root 4096 Jan 27 15:36 . + drwxr-xr-x 5 root root 4096 Jan 27 15:36 .. + -r--r--r-- 2 root root 0 Jan 27 15:36 .wh.shadow +``` + +--- + +## AUFS performance + +- AUFS `mount()` is fast, so creation of containers is quick. + +- Read/write access has native speeds. + +- But initial `open()` is expensive in two scenarios: + + - when writing big files (log files, databases ...), + + - when searching many directories (PATH, classpath, etc.) over many layers. + +- Protip: when we built dotCloud, we ended up putting + all important data on *volumes*. + +- When starting the same container multiple times: + + - the data is loaded only once from disk, and cached only once in memory; + + - but `dentries` will be duplicated. + +--- + +## Device Mapper + +Device Mapper is a rich subsystem with many features. + +It can be used for: RAID, encrypted devices, snapshots, and more. + +In the context of containers (and Docker in particular), "Device Mapper" +means: + +"the Device Mapper system + its *thin provisioning target*" + +If you see the abbreviation "thinp" it stands for "thin provisioning". + +--- + +## Device Mapper principles + +- Copy-on-write happens on the *block* level + (instead of the *file* level). + +- Each container and each image get their own block device. + +- At any given time, it is possible to take a snapshot: + + - of an existing container (to create a frozen image), + + - of an existing image (to create a container from it). + +- If a block has never been written to: + + - it's assumed to be all zeros, + + - it's not allocated on disk. + +(That last property is the reason for the name "thin" provisioning.) + +--- + +## Device Mapper operational details + +- Two storage areas are needed: + one for *data*, another for *metadata*. + +- "data" is also called the "pool"; it's just a big pool of blocks. + + (Docker uses the smallest possible block size, 64 KB.) + +- "metadata" contains the mappings between virtual offsets (in the + snapshots) and physical offsets (in the pool). + +- Each time a new block (or a copy-on-write block) is written, + a block is allocated from the pool. + +- When there are no more blocks in the pool, attempts to write + will stall until the pool is increased (or the write operation + aborted). + +- In other words: when running out of space, containers are + frozen, but operations will resume as soon as space is available. + +--- + +## Device Mapper performance + +- By default, Docker puts data and metadata on a loop device + backed by a sparse file. + +- This is great from a usability point of view, + since zero configuration is needed. + +- But it is terrible from a performance point of view: + + - each time a container writes to a new block, + - a block has to be allocated from the pool, + - and when it's written to, + - a block has to be allocated from the sparse file, + - and sparse file performance isn't great anyway. + +- If you use Device Mapper, make sure to put data (and metadata) + on devices! + +--- + +## BTRFS principles + +- BTRFS is a filesystem (like EXT4, XFS, NTFS...) with built-in snapshots. + +- The "copy-on-write" happens at the filesystem level. + +- BTRFS integrates the snapshot and block pool management features + at the filesystem level. + + (Instead of the block level for Device Mapper.) + +- In practice, we create a "subvolume" and + later take a "snapshot" of that subvolume. + + Imagine: `mkdir` with Super Powers and `cp -a` with Super Powers. + +- These operations can be executed with the `btrfs` CLI tool. + +--- + +## BTRFS in practice with Docker + +- Docker can use BTRFS and its snapshotting features to store container images. + +- The only requirement is that `/var/lib/docker` is on a BTRFS filesystem. + + (Or, the directory specified with the `--data-root` flag when starting the engine.) + +--- + +class: extra-details + +## BTRFS quirks + +- BTRFS works by dividing its storage in *chunks*. + +- A chunk can contain data or metadata. + +- You can run out of chunks (and get `No space left on device`) + even though `df` shows space available. + + (Because chunks are only partially allocated.) + +- Quick fix: + +``` + # btrfs filesys balance start -dusage=1 /var/lib/docker +``` + +--- + +## Overlay2 + +- Overlay2 is very similar to AUFS. + +- However, it has been merged in "upstream" kernel. + +- It is therefore available on all modern kernels. + + (AUFS was available on Debian and Ubuntu, but required custom kernels on other distros.) + +- It is simpler than AUFS (it can only have two branches, called "layers"). + +- The container engine abstracts this detail, so this is not a concern. + +- Overlay2 storage drivers generally use hard links between layers. + +- This improves `stat()` and `open()` performance, at the expense of inode usage. + +--- + +## ZFS + +- ZFS is similar to BTRFS (at least from a container user's perspective). + +- Pros: + + - high performance + - high reliability (with e.g. data checksums) + - optional data compression and deduplication + +- Cons: + + - high memory usage + - not in upstream kernel + +- It is available as a kernel module or through FUSE. + +--- + +## Which one is the best? + +- According to Michael Crosby (core Docker maintainer), overlay2! + +- Overlay2 is available on all modern systems. + +- Its memory usage is better than Device Mapper, BTRFS, or ZFS. + +- The remarks about *write performance* shouldn't bother you: +
+ data should always be stored in volumes anyway! +