container.training/docs/kubenet.md

Kubernetes network model

• TL,DR:

  Our cluster (nodes and pods) is one big flat IP network.

--

• In detail:

• all nodes must be able to reach each other, without NAT

• all pods must be able to reach each other, without NAT

• pods and nodes must be able to reach each other, without NAT

• each pod is aware of its IP address (no NAT)

• Kubernetes doesn't mandate any particular implementation

---

Kubernetes network model: the good

• Everything can reach everything

• No address translation

• No port translation

• No new protocol

• Pods cannot move from a node to another and keep their IP address

• IP addresses don't have to be "portable" from a node to another

  (We can use e.g. a subnet per node and use a simple routed topology)

• The specification is simple enough to allow many various implementations

---

Kubernetes network model: the bad and the ugly

• Everything can reach everything

  • if you want security, you need to add network policies

  • the network implementation that you use needs to support them

• There are literally dozens of implementations out there

  (15 are listed in the Kubernetes documentation)

• It looks like you have a level 3 network, but it's only level 4

  (The spec requires UDP and TCP, but not port ranges or arbitrary IP packets)

• kube-proxy is on the data path when connecting to a pod or container,
  and it's not particularly fast (relies on userland proxying or iptables)

---

Kubernetes network model: in practice

• The nodes that we are using have been setup to use Weave

• We don't endorse Weave in a particular way, it just Works For Us

• Don't worry about the warning about kube-proxy performance

• Unless you:

  • routinely saturate 10G network interfaces

  • count packet rates in millions per second

  • run high-traffic VOIP or gaming platforms

  • do weird things that involve millions of simultaneous connections
    (in which case you're already familiar with kernel tuning)