diff --git a/.gitignore b/.gitignore index 00e48a445..9c9c40ea5 100644 --- a/.gitignore +++ b/.gitignore @@ -30,32 +30,19 @@ coverage.html *~ # Project specific -weaver/weaver -weavedns/weavedns -sigproxy/sigproxy -tools/bin -tools/build -releases -weave.tar -weavedns.tar -weaveexec.tar -weaveexec/weave -weaveexec/sigproxy -weaveexec/docker*.tgz -Vagrantfile.local scope.tar -scope/probe/probe -scope/bridge/bridge -scope/app/app -scope/demoprobe/demoprobe -scope/fixprobe/fixprobe -scope/genreport/genreport -scope/graphviz/graphviz -scope/oneshot/oneshot -scope/web/assets/videos/*.mp4 -scope/release/*.tar.gz -scope/web/docs/*.html -scope/*sublime-project -scope/*sublime-workspace -scope/*npm-debug.log +probe/probe +bridge/bridge +app/app +demoprobe/demoprobe +fixprobe/fixprobe +genreport/genreport +graphviz/graphviz +oneshot/oneshot +web/assets/videos/*.mp4 +release/*.tar.gz +web/docs/*.html +*sublime-project +*sublime-workspace +*npm-debug.log diff --git a/ENHANCE.md b/ENHANCE.md deleted file mode 100644 index b531396f4..000000000 --- a/ENHANCE.md +++ /dev/null @@ -1,137 +0,0 @@ -# Enhance - -## 2014 10 05 - -Informed by a new context, we began sketching out a simplified data model. -Initial requirements: - -- Must support order-of-thousands nodes in the topology. -- Must support a scrapable API. -- Packet counts, etc. are a value-add and should be composed in to the topology as needed. - -Desires based on experience so far: - -- Too much effort was spent wrangling deeply nested hierarchies in the report structs. They should be flattened. -- It would be nice if the probe API were directly consumable by the app. - - This enables much simpler composition of "bridge" components, including none at all. - - This composes nicely with the scrapable API requirement. -- The simplest use case of no metadata at all could (should) be satisfied by scraping the /proc FS, with no packet sniffing. - -Peter's initial thoughts was to key everything based on the atom of host + socket_descriptor. -This represents the most specific (application-level) topology. -Other more general topologies (network, transport) could be derived from that. - -Harmen noted that we want to capture data that may not be associated with a socket_descriptor. -Therefore we should build multiple, independent topologies: application, network, and transport. -Each one is a complete topology for that layer, with lower (OSI) layers always containing the higher ones. - -We think we can get away with a report structure that looks like this - -```json -{ - "application": { - "adjacency": { - "host:src": ["dst1", "dst2", "dst3"] - }, - "metadata": { - "host:src:dst1": {} - } - } - - "network": { - "adjacency": {}, - "metadata": {} - } - - "transport": { - "adjacency": {}, - "metadata": {} - } -} -``` - -Where the meaning of the keys and key-parts depends on the section of the report. -Specifically, in the application section, src is an encoding of the socket_descriptor. - -Note that we still need another section to map application keys to processes, either physical (PIDs) or logical (process names, service names, etc.). -That mapping can take the form of a function: - -```go -type mapper func(socket_descriptor) string -``` - -Where the returned string will be the aggregation key of the logical (application) topology. -At the lowest level, it could be the PID that bound the socket_descriptor. -At a middle layer, it could be the name of the process associated with the PID. -At a higher layer, the function could query a service discovery component, and return a service name. -It would probably be good to support multiple simultaneous mappers, so the application-layer topology could have multiple levels of detail. - -### Example implementation - -Two probes, A and B, with a single TCP connection, both agree there is a connection. - -```json -{ - "proc": { - "mapping": { - "hostA:192.168.1.1:12345": "curl", - "hostB:192.168.1.2:80": "apache" - }, - "adjacency": { - "hostA:192.168.1.1:12345": ["192.168.1.2:80"], - "hostB:192.168.1.2:80": ["192.168.1.1:12345"] - }, - "metadata": { - "hostA:192.168.1.1:12345:192.168.1.2:80": {"egress": "12bytes", "ingress": "0", "d": "15s"}, - "hostB:192.168.1.2:80:192.168.1.1:12345": {"egress": "0", "ingress": "12bytes", "d": "15s"} - } - }, - - "transport": { - "adjacency": { - "hostA:192.168.1.1:12345": ["192.168.1.2:80"], - "hostB:192.168.1.2:80": ["192.168.1.1:12345"] - }, - "metadata": { - "hostA:192.168.1.1:12345:192.168.1.2:80": {"egress": "12bytes", "ingress": "0", "d": "15s"}, - "hostB:192.168.1.2:80:192.168.1.1:12345": {"egress": "0", "ingress": "12bytes", "d": "15s"} - } - }, - - "network": { - "adjacency": { - "hostA:192.168.1.1": ["192.168.1.2"], - "hostB:192.168.1.2": ["192.168.1.1"] - }, - "metadata": { - "hostA:192.168.1.1:12345:192.168.1.2:80": {"egress": "12bytes", "ingress": "0", "d": "15s"}, - "hostB:192.168.1.2:80:192.168.1.1:12345": {"egress": "0", "ingress": "12bytes", "d": "15s"} - } - }, - - "nodes": { - "hostA": { - "addresses": ["192.168.1.1"], - "hostname": "..." - }, - "hostB": { - "addresses": ["192.168.1.2"], - "hostname": "...", - }, - } -} -``` - -Note that the proc.mapping can only be filled by the procfs-probe. -But {proc,transport}.{adjacency,metadata} could in theory be filled both procfs-probe and sniffing-probe. -That's a consequence of the fact that, in the case of UDP and TCP, the socket_descriptor is fully reconstructable from a network packet's source and destination fields. -For simplicity, and for now, we'd like to restrict that freedom, and state that the proc section should be filled only by the procfs-probe. -This draws a clear line of ownership and demarcation. -We can revisit this decision in the future, if it proves too restrictive. - -For now we will assert that the metadata type is the same structure for each section. -Adjacency list values don't necessarily need to exist as nodes. Cases are non-probe machines and broadcast addresses. - -Merging reports erquires a time component because rates require a window, and we can't make assumptions about fixed window sizes (either because the edge isn't present for the entire interval, or the whole probe isn't). -Reports are by definition non-overlapping for a single e.g. socket_descriptor, so that's a safe assumption to make. Therefore, all we need are durations, which may be semantically merged with simple addition. - diff --git a/Makefile b/Makefile deleted file mode 100644 index 8bca0541f..000000000 --- a/Makefile +++ /dev/null @@ -1,45 +0,0 @@ -.PHONY: default get all build release clean - -default: all - -get: - cd app && go get - -all: - $(MAKE) -C report - $(MAKE) -C xfer - $(MAKE) -C app - $(MAKE) -C bridge - $(MAKE) -C demoprobe - $(MAKE) -C oneshot - $(MAKE) -C fixprobe - $(MAKE) -C genreport - $(MAKE) -C probe - $(MAKE) -C integration - -build: - $(MAKE) -C report build - $(MAKE) -C xfer build - $(MAKE) -C app build - $(MAKE) -C bridge build - $(MAKE) -C demoprobe build - $(MAKE) -C oneshot build - $(MAKE) -C fixprobe build - $(MAKE) -C probe build - $(MAKE) -C genreport build - $(MAKE) -C integration build - -release: - $(MAKE) -C release - -clean: - $(MAKE) -C report clean - $(MAKE) -C xfer clean - $(MAKE) -C app clean - $(MAKE) -C bridge clean - $(MAKE) -C demoprobe clean - $(MAKE) -C oneshot clean - $(MAKE) -C fixprobe clean - $(MAKE) -C genreport clean - $(MAKE) -C probe clean - $(MAKE) -C integration clean diff --git a/NOTICE b/NOTICE index 2bd5ec81b..0883dc250 100644 --- a/NOTICE +++ b/NOTICE @@ -1,3 +1,3 @@ Weave -Copyright 2014-2015 Weaveworks Ltd. +Copyright 2015 Weaveworks Ltd. This product includes software developed at Weaveworks Ltd. diff --git a/README.md b/README.md index eb7093836..f1e1c2a6d 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,3 @@ -# Cello +# Scope -Network harmony for the 21st century. - -## Component diagram - -``` -+-----------------------------------+ -| client | -+-----------------------------------+ - ^ - | 4040, report.RenderableNode - | -+---------------------------------------------------+ -| app | -+---------------------------------------------------+ - ^ ^ ^ ^ - | | | | 4030, report.Report - | | | | -+--------+ +-------+ +----------+ +-----------+ -| bridge | .-| probe | .-| fixprobe | .-| demoprobe | -+--------+ | +-------+ | +----------+ | +-----------+ - ^^^ | | | - ||| | | | - ||'-----' | | - |'------------------' | - '----------------------------------' -``` \ No newline at end of file +TODO diff --git a/REST.md b/REST.md deleted file mode 100644 index a07386dec..000000000 --- a/REST.md +++ /dev/null @@ -1,158 +0,0 @@ -# Rest structure - -The various REST calls to retrieve topology information. - -# Topologies list - -``` -GET /api/topology -``` - -Returns a list of topology descriptions. Each description has the fields: - -- `name` - string; friendly name of the topology -- `url` - string; path of the topology -- `grouped_url` - string or not existing; path to the 'grouped' version on the - topology, if there is one for this topology -- `stats` - object; basic statistics about the topology, with these fields: - - node_count - int - - nonpseudo_node_count - int - - edge_count - int - - -# Full topologies - -``` -GET /api/topology/:kind -``` - -Where `:kind` is one of - -- `processpid` - processes by PID (most granular) -- `processname` - processes by name -- `networkip` - hosts (interfaces) by IP -- `networkhost` - hosts by hostname (least granular) - -A GET on a topology URL gives: - -```json -{ - "nodes": { - "nodeID": {} - } -} -``` - -`nodeID` is an opaque string. The node object is a map with the fields: - -- `id` - string; same as nodeID -- `adjacency` - array of strings; list of node IDs that this node is connected to -- `label_major` - string; primary label for this node -- `label_minor` - string; extra label information for this node -- `pseudo` - bool; true if the node is "deduced" from conditions and has no actual probe; false if this node comes from a probe -- `origin` - optional array of strings; origin IDs that the node information comes from - -All node IDs listed in `adjacency` have a corresponding node entry. In other -words, the graph is complete. - -If A has B in its `adjacency` list, that means A communicates to B. B may or -may not have A in its adjacency list. In other words, the graph is directed. - -Pseudonodes are nodes for which we don't have direct measurements, but we know -that other nodes communicate with them. These nodes will always have empty -adjacency and origin lists. - - -# Websocket - -There is also a websocket version of the full topology, which provides -continuous updates. - -``` -GET ws://hostname/api/topology/:kind/ws?t=5s -``` - -The URL is the same as for an HTTP topology, but with `/ws` appended. The -argument `t` can be used to specify the update interval. - -The topology is send as a stream of updates, starting with the difference -between the empty topology and the current state. - -```json -{ - "add": [], - "update": [], - "remove": [] -} -``` - -The object has these fields: - -- add - array of objects; a list of node objects which are new since the last update -- update - array of objects; a list of node objects which have changed -- remove - array of strings; a list of node IDs which are gone - - -# Nodes - -Given any base topology URL, there is also: - -``` -GET /api/topology/:kind/:id -``` - -This returns a single node object. It has fields in addition to those from the -corresponding entry from the full topology. - -```json -{ - "node": {} -} -``` - -Extra fields: - - aggregate - object - sum of all edge metadata, see 'metadata' below for - the fields. - -# Edges - -Details for an edge: - -``` -GET /api/topology/:kind/:id/:adjacentid -``` - -This returns the metadata for the undirected edge between :id and :adjacentid, -from the point of view of :id. If the edge is not found it'll still return a -statuscode 200, but without values. - -```json -{ - "metadata": {} -} -``` - -metadata can have these keys: - - - max_conn_count_tcp - int; maximum number of concurrent TCP connections. - - egress_bytes - int; total number of bytes. - - ingress_bytes - int; total number of bytes. - -Keys will only be set if the statistic is measured by the probes used. If a -statistic has the value 0 the key will still be present. - -# Origin - -Origin calls give information about the machine on which the measurements are -made. - -``` -GET /api/origin/:id -``` - -Origin is a reference to a physical machine that runs a probe. The origin ID -comes from the `origin` field in a node object. The returned object is a map -with the fields: - -- `hostname` - string; fully qualified domain name of the host -- `os` - string; operating system of the host, e.g. "linux"