github/weave-scope
1
0
Fork 0
mirror of https://github.com/weaveworks/scope.git synced 2026-07-28 01:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

rm some toplevel docs, minor changes

Browse Source
This commit is contained in:
Peter Bourgon
2015-05-12 18:21:37 +02:00
committed by Tom Wilkie
parent d50e827fd5
commit ac71b262c3
6 changed files with 17 additions and 395 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
.gitignore vendored
View File

@@ -30,32 +30,19 @@ coverage.html
*~
# Project specific
weaver/weaver
weavedns/weavedns
sigproxy/sigproxy
tools/bin
tools/build
releases
weave.tar
weavedns.tar
weaveexec.tar
weaveexec/weave
weaveexec/sigproxy
weaveexec/docker*.tgz
Vagrantfile.local
scope.tar
scope/probe/probe
scope/bridge/bridge
scope/app/app
scope/demoprobe/demoprobe
scope/fixprobe/fixprobe
scope/genreport/genreport
scope/graphviz/graphviz
scope/oneshot/oneshot
scope/web/assets/videos/*.mp4
scope/release/*.tar.gz
scope/web/docs/*.html
scope/*sublime-project
scope/*sublime-workspace
scope/*npm-debug.log
probe/probe
bridge/bridge
app/app
demoprobe/demoprobe
fixprobe/fixprobe
genreport/genreport
graphviz/graphviz
oneshot/oneshot
web/assets/videos/*.mp4
release/*.tar.gz
web/docs/*.html
*sublime-project
*sublime-workspace
*npm-debug.log

137
ENHANCE.md
View File

@@ -1,137 +0,0 @@
# Enhance
## 2014 10 05
Informed by a new context, we began sketching out a simplified data model.
Initial requirements:
- Must support order-of-thousands nodes in the topology.
- Must support a scrapable API.
- Packet counts, etc. are a value-add and should be composed in to the topology as needed.
Desires based on experience so far:
- Too much effort was spent wrangling deeply nested hierarchies in the report structs. They should be flattened.
- It would be nice if the probe API were directly consumable by the app.
- This enables much simpler composition of "bridge" components, including none at all.
- This composes nicely with the scrapable API requirement.
- The simplest use case of no metadata at all could (should) be satisfied by scraping the /proc FS, with no packet sniffing.
Peter's initial thoughts was to key everything based on the atom of host + socket_descriptor.
This represents the most specific (application-level) topology.
Other more general topologies (network, transport) could be derived from that.
Harmen noted that we want to capture data that may not be associated with a socket_descriptor.
Therefore we should build multiple, independent topologies: application, network, and transport.
Each one is a complete topology for that layer, with lower (OSI) layers always containing the higher ones.
We think we can get away with a report structure that looks like this
```json
{
"application": {
"adjacency": {
"host:src": ["dst1", "dst2", "dst3"]
},
"metadata": {
"host:src:dst1": {}
}
}
"network": {
"adjacency": {},
"metadata": {}
}
"transport": {
"adjacency": {},
"metadata": {}
}
}
```
Where the meaning of the keys and key-parts depends on the section of the report.
Specifically, in the application section, src is an encoding of the socket_descriptor.
Note that we still need another section to map application keys to processes, either physical (PIDs) or logical (process names, service names, etc.).
That mapping can take the form of a function:
```go
type mapper func(socket_descriptor) string
```
Where the returned string will be the aggregation key of the logical (application) topology.
At the lowest level, it could be the PID that bound the socket_descriptor.
At a middle layer, it could be the name of the process associated with the PID.
At a higher layer, the function could query a service discovery component, and return a service name.
It would probably be good to support multiple simultaneous mappers, so the application-layer topology could have multiple levels of detail.
### Example implementation
Two probes, A and B, with a single TCP connection, both agree there is a connection.
```json
{
"proc": {
"mapping": {
"hostA:192.168.1.1:12345": "curl",
"hostB:192.168.1.2:80": "apache"
},
"adjacency": {
"hostA:192.168.1.1:12345": ["192.168.1.2:80"],
"hostB:192.168.1.2:80": ["192.168.1.1:12345"]
},
"metadata": {
"hostA:192.168.1.1:12345:192.168.1.2:80": {"egress": "12bytes", "ingress": "0", "d": "15s"},
"hostB:192.168.1.2:80:192.168.1.1:12345": {"egress": "0", "ingress": "12bytes", "d": "15s"}
}
},
"transport": {
"adjacency": {
"hostA:192.168.1.1:12345": ["192.168.1.2:80"],
"hostB:192.168.1.2:80": ["192.168.1.1:12345"]
},
"metadata": {
"hostA:192.168.1.1:12345:192.168.1.2:80": {"egress": "12bytes", "ingress": "0", "d": "15s"},
"hostB:192.168.1.2:80:192.168.1.1:12345": {"egress": "0", "ingress": "12bytes", "d": "15s"}
}
},
"network": {
"adjacency": {
"hostA:192.168.1.1": ["192.168.1.2"],
"hostB:192.168.1.2": ["192.168.1.1"]
},
"metadata": {
"hostA:192.168.1.1:12345:192.168.1.2:80": {"egress": "12bytes", "ingress": "0", "d": "15s"},
"hostB:192.168.1.2:80:192.168.1.1:12345": {"egress": "0", "ingress": "12bytes", "d": "15s"}
}
},
"nodes": {
"hostA": {
"addresses": ["192.168.1.1"],
"hostname": "..."
},
"hostB": {
"addresses": ["192.168.1.2"],
"hostname": "...",
},
}
}
```
Note that the proc.mapping can only be filled by the procfs-probe.
But {proc,transport}.{adjacency,metadata} could in theory be filled both procfs-probe and sniffing-probe.
That's a consequence of the fact that, in the case of UDP and TCP, the socket_descriptor is fully reconstructable from a network packet's source and destination fields.
For simplicity, and for now, we'd like to restrict that freedom, and state that the proc section should be filled only by the procfs-probe.
This draws a clear line of ownership and demarcation.
We can revisit this decision in the future, if it proves too restrictive.
For now we will assert that the metadata type is the same structure for each section.
Adjacency list values don't necessarily need to exist as nodes. Cases are non-probe machines and broadcast addresses.
Merging reports erquires a time component because rates require a window, and we can't make assumptions about fixed window sizes (either because the edge isn't present for the entire interval, or the whole probe isn't).
Reports are by definition non-overlapping for a single e.g. socket_descriptor, so that's a safe assumption to make. Therefore, all we need are durations, which may be semantically merged with simple addition.

45
Makefile
View File

@@ -1,45 +0,0 @@
.PHONY: default get all build release clean
default: all
get:
cd app && go get
all:
$(MAKE) -C report
$(MAKE) -C xfer
$(MAKE) -C app
$(MAKE) -C bridge
$(MAKE) -C demoprobe
$(MAKE) -C oneshot
$(MAKE) -C fixprobe
$(MAKE) -C genreport
$(MAKE) -C probe
$(MAKE) -C integration
build:
$(MAKE) -C report build
$(MAKE) -C xfer build
$(MAKE) -C app build
$(MAKE) -C bridge build
$(MAKE) -C demoprobe build
$(MAKE) -C oneshot build
$(MAKE) -C fixprobe build
$(MAKE) -C probe build
$(MAKE) -C genreport build
$(MAKE) -C integration build
release:
$(MAKE) -C release
clean:
$(MAKE) -C report clean
$(MAKE) -C xfer clean
$(MAKE) -C app clean
$(MAKE) -C bridge clean
$(MAKE) -C demoprobe clean
$(MAKE) -C oneshot clean
$(MAKE) -C fixprobe clean
$(MAKE) -C genreport clean
$(MAKE) -C probe clean
$(MAKE) -C integration clean

2
NOTICE
View File

@@ -1,3 +1,3 @@
Weave
Copyright 2014-2015 Weaveworks Ltd.
Copyright 2015 Weaveworks Ltd.
This product includes software developed at Weaveworks Ltd.

29
README.md
View File

@@ -1,28 +1,3 @@
# Cello
# Scope
Network harmony for the 21st century.
## Component diagram
```
+-----------------------------------+
| client |
+-----------------------------------+
^
| 4040, report.RenderableNode
|
+---------------------------------------------------+
| app |
+---------------------------------------------------+
^ ^ ^ ^
| | | | 4030, report.Report
| | | |
+--------+ +-------+ +----------+ +-----------+
| bridge | .-| probe | .-| fixprobe | .-| demoprobe |
+--------+ | +-------+ | +----------+ | +-----------+
^^^ | | |
||| | | |
||'-----' | |
|'------------------' |
'----------------------------------'
```
TODO

158
REST.md
View File

@@ -1,158 +0,0 @@
# Rest structure
The various REST calls to retrieve topology information.
# Topologies list
```
GET /api/topology
```
Returns a list of topology descriptions. Each description has the fields:
- `name` - string; friendly name of the topology
- `url` - string; path of the topology
- `grouped_url` - string or not existing; path to the 'grouped' version on the
topology, if there is one for this topology
- `stats` - object; basic statistics about the topology, with these fields:
- node_count - int
- nonpseudo_node_count - int
- edge_count - int
# Full topologies
```
GET /api/topology/:kind
```
Where `:kind` is one of
- `processpid` - processes by PID (most granular)
- `processname` - processes by name
- `networkip` - hosts (interfaces) by IP
- `networkhost` - hosts by hostname (least granular)
A GET on a topology URL gives:
```json
{
"nodes": {
"nodeID": {}
}
}
```
`nodeID` is an opaque string. The node object is a map with the fields:
- `id` - string; same as nodeID
- `adjacency` - array of strings; list of node IDs that this node is connected to
- `label_major` - string; primary label for this node
- `label_minor` - string; extra label information for this node
- `pseudo` - bool; true if the node is "deduced" from conditions and has no actual probe; false if this node comes from a probe
- `origin` - optional array of strings; origin IDs that the node information comes from
All node IDs listed in `adjacency` have a corresponding node entry. In other
words, the graph is complete.
If A has B in its `adjacency` list, that means A communicates to B. B may or
may not have A in its adjacency list. In other words, the graph is directed.
Pseudonodes are nodes for which we don't have direct measurements, but we know
that other nodes communicate with them. These nodes will always have empty
adjacency and origin lists.
# Websocket
There is also a websocket version of the full topology, which provides
continuous updates.
```
GET ws://hostname/api/topology/:kind/ws?t=5s
```
The URL is the same as for an HTTP topology, but with `/ws` appended. The
argument `t` can be used to specify the update interval.
The topology is send as a stream of updates, starting with the difference
between the empty topology and the current state.
```json
{
"add": [],
"update": [],
"remove": []
}
```
The object has these fields:
- add - array of objects; a list of node objects which are new since the last update
- update - array of objects; a list of node objects which have changed
- remove - array of strings; a list of node IDs which are gone
# Nodes
Given any base topology URL, there is also:
```
GET /api/topology/:kind/:id
```
This returns a single node object. It has fields in addition to those from the
corresponding entry from the full topology.
```json
{
"node": {}
}
```
Extra fields:
- aggregate - object - sum of all edge metadata, see 'metadata' below for
the fields.
# Edges
Details for an edge:
```
GET /api/topology/:kind/:id/:adjacentid
```
This returns the metadata for the undirected edge between :id and :adjacentid,
from the point of view of :id. If the edge is not found it'll still return a
statuscode 200, but without values.
```json
{
"metadata": {}
}
```
metadata can have these keys:
- max_conn_count_tcp - int; maximum number of concurrent TCP connections.
- egress_bytes - int; total number of bytes.
- ingress_bytes - int; total number of bytes.
Keys will only be set if the statistic is measured by the probes used. If a
statistic has the value 0 the key will still be present.
# Origin
Origin calls give information about the machine on which the measurements are
made.
```
GET /api/origin/:id
```
Origin is a reference to a physical machine that runs a probe. The origin ID
comes from the `origin` field in a node object. The returned object is a map
with the fields:
- `hostname` - string; fully qualified domain name of the host
- `os` - string; operating system of the host, e.g. "linux"