Compare commits

...

24 Commits

Author SHA1 Message Date
M. Mert Yıldıran
c1b2cda468 Change the log level from ERROR to INFO for cases; not a Go binary or symbol table is stripped (#1164) 2022-06-26 15:25:26 +03:00
M. Mert Yıldıran
cf91f3dab4 Fix Go crypto/tls eBPF tracer on ARM64 (#1162)
* Handle the Go `ABIInternal` differences on ARM64

* Log the Capstone version, arch and mode

* Upgrade Capstone from `4.0.2` to `5.0-rc2` to have ARM64 instructions fix

* Use the correct register on ARM64 for reading buffer length

* Fix the addresses on ARM64

* #run_acceptance_tests

* Update `x86.o`

* Update `arm64.o`

* #run_acceptance_tests
2022-06-26 14:11:04 +03:00
RoyUP9
1ebc51b45e Remove OAS contract monitoring feature (#1163) 2022-06-26 12:22:25 +03:00
gadotroee
a96072623f Fix problems that golang inspection reported (#1158) 2022-06-23 16:49:56 +03:00
gadotroee
8c187179b0 Adding tests to the bucket statistics (#1160) 2022-06-23 11:04:12 +03:00
lirazyehezkel
d7d802830f TRA-4274 fix ui common linter issues (#1161)
* fix lint issues

* fix all ui-common line issues

* max warnings 0
2022-06-22 09:56:27 +03:00
AmitUp9
4d64dd4b04 TRA-4602_timeline bar charts to traffic statistics (#1159)
* pie chart for protocols and methods by requests and volume

* protocols legend

* timeline bar chart component created

* timeline can view requests and volume

* sorting the bra charts by timestemp

* disable view of <1% pieces in pie

* space added to the end of the file

* package.json update

* cr fixes

* remove spave

* remove unnecessary react fragment

Co-authored-by: Liraz Yehezkel <lirazy@up9.com>
2022-06-21 14:54:56 +03:00
gadotroee
9a40895e9c Add timing endpoint (timeline data) for stats (#1157) 2022-06-21 14:33:04 +03:00
M. Mert Yıldıran
f9a9c05f48 Add ARM64 support to eBPF (#1151)
* Define and use `BPF_CFLAGS` environment variable

* Add eBPF dependencies to `builder-from-amd64-to-arm64v8` and `builder-native-base`

* Add eBPF dependencies to `builder-from-arm64v8-to-amd64`

* Only compile x86 arch of Capstone for x86 target

* Build and install `libbpf` from source

* Fix `builder-from-arm64v8-to-amd64`

* Add `BPF_TARGET` environment variable

* Fix the eBPF verifier error on ARM64

* Fix `go_crypto_tls_ex_uprobe`

* Fix the check

* #run_acceptance_tests

* Fix the build script

* Include ARM64 files

* Bring back `x86.o`

* Generate both endianness

* Fix Dockerfile

* #run_acceptance_tests

* Determine the endianness on runtime if it's possible in Go (default little-endian) #run_acceptance_tests

* Revert "Determine the endianness on runtime if it's possible in Go (default little-endian) #run_acceptance_tests"

This reverts commit a2c83c6040.

* Remove big-endian files #run_acceptance_tests

* Fix Dockerfile #run_acceptance_tests

Co-authored-by: Ubuntu <ubuntu@ip-172-31-33-233.eu-central-1.compute.internal>
2022-06-20 16:16:56 +03:00
lirazyehezkel
c9d4f88de8 TRA-4602 Traffic statistics - Pie Chart (#1153)
* pie chart for protocols and methods by requests and volume

* protocols legend

* methods legend

* pie chart component

* code refactor

* no message

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2022-06-19 17:19:06 +03:00
gadotroee
6f117d0a84 Accumulative stats for protocol and methods (#1144)
only data for pie chart currently
2022-06-19 16:59:35 +03:00
David Levanon
99cb0b4f44 Close gopacket connection immediately, basic throttling and assembler mutex removal
* close gopacket conn immediately

* increase last ack threshold to 3 seconds

* remove empty condition

* add periodic stats

* protect connections from concurrent updates

* implement basic throttling base on live streams count

* remove assembler mutex

* pr fixes

* change max conns default to 500

* create connectionId type

* fix linter
2022-06-19 16:47:03 +03:00
M. Mert Yıldıran
167bbe3741 Fix a slice bounds out of range panic in Go offsets calculation (#1155)
* Fix a `slice bounds out of range` error in Go offsets calculation

* Change `Errorf` to `Warningf`
2022-06-19 12:19:32 +03:00
Andrey Pokhilko
7a823e89f1 Add option to limit example len (#1145)
Do not save OAS example response if it is over the limit.
SpecGen.MaxExampleLen is measured in bytes.
Limit=0 means "don't record any examples", while limit=-1 means "record all".

Limit is configured via the config file / --set: --set oas.max-example-len=100.
The default is 10KB.

This PR breaks the previous config. Moved oas bool (enable/disable OAS) to oas.enable.
Users who wish to disable OAS (or to explicitly enable it) should now to so with --set oas.enable=false instead of --set oas=false.

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
2022-06-16 14:18:42 +03:00
gadotroee
be98d85cb8 Recreate package-lock to remove "old" packages (#1152) 2022-06-16 13:04:38 +03:00
AmitUp9
e743ab7f7a TRA_4579 - fix acceptance test of right panel body checkboxes (#1150)
* #run_acceptance_tests

* added data-cy to checkboxes in body section, removed unnecessary tests and fix decoded text testing

* #run_acceptance_tests
2022-06-16 10:27:31 +03:00
AmitUp9
78081a4a16 Node sass npm package migrate to sass npm package (#1143)
* remove node-sass from project and install sass instead

* fix ui-common path
2022-06-14 16:53:34 +03:00
lirazyehezkel
0ef6a7d2c4 Fix images imports (#1139)
* Fix images imports

* fix image name
2022-06-13 16:19:50 +03:00
Igor Gov
ba361df7e7 Mizu install should be namespace aware (#1141) 2022-06-13 15:26:00 +03:00
M. Mert Yıldıran
1f70eab0a4 Update Basenine hashes (#1138) 2022-06-12 16:36:33 +03:00
M. Mert Yıldıran
ea4991905b Upgrade Basenine to v0.8.3 (#1137)
* Upgrade Basenine to `v0.8.3`

* #run_acceptance_tests
2022-06-12 16:15:33 +03:00
M. Mert Yıldıran
2ad4838cf5 Add Go crypto/tls eBPF tracer for TLS connections (#1120)
* Run `go generate tls_tapper.go`

* Add `golang_uprobes.c`

* Add Golang hooks and offsets

* Add `golangConnection` struct and implement `pollGolangReadWrite` method

* Upgrade `github.com/cilium/ebpf` version to `v0.8.1`

* Fix the linter error

* Move map related stuff to `maps.h` and run `go generate tls_tapper.go`

* Remove unused parameter

* Add an environment variable to test Golang locally

* Replace `Libssl` occurrences with `Ssllib` for consistency

* Fix exe path finding

* Temporarily disable OpenSSL

* Fix the mixed offsets and dissection preparation

* Change the read symbol from `net/http.(*persistConn).Read` to `crypto/tls.(*Conn).Read`

* Remove `len` and `cap` fields

* Fix the indent

* Fix the read data address

* Make `golang_dial_writes` key `__u64` and include the PID

* Fix the read data address one more time

* Temporarily disable the PCAP capture

* Add a uprobe for `net/http.(*gzipReader).Read` to read chunked HTTP response body

* Cancel `golang_crypto_tls_read_uprobe` if it's a gzip read

* Make hash map names more meaningful

* Pass the connection address from `write` to `gzip` through a common address between `gzip` and `dial`

* Fix the probed line number links

* Add `golangReader` struct and implement its `Read` method

* Have a single counter pair and request response matcher per Golang connection

* Add `MIZU_GLOBAL_GOLANG_PATH` environment variable

* `NULL` terminate the bytes with `unix.ByteSliceToString`

* Temporarily reject the gzip chunks

* Add malformed TODOs

* Revert "`NULL` terminate the bytes with `unix.ByteSliceToString`"

This reverts commit 7ee7ef7e44.

* Bring back `len` and `cap` fields

* Set `len` and `cap` in `golang_net_http_gzipreader_read_uprobe` as well

* Remove two `TODO`s

* Fix the `key_gzip` offsets

* Compress if it's gzip chunk (probably wrong!)

* Revert "Compress if it's gzip chunk (probably wrong!)"

This reverts commit 094a7c3da4.

* Remove `golang_net_http_gzipreader_read_uprobe`

* Read constant 4KiB

* Use constant read length

* Get the correct len of bytes (saw the second entry)

* Set all buffer sizes to `CHUNK_SIZE`

* Remove a `TODO`

* Revert "Temporarily disable the PCAP capture"

This reverts commit a2da15ef2d.

* Update `golang_crypto_tls_read_uprobe`

* Set the `reader` field of `tlsStream` to fix a `nil pointer dereference` error

* Don't export any fields of `golangConnection`

* Close the reader when we drop the connection

* Add a tracepoint for `sys_enter_close` to detect socket closes

* Rename `socket` struct to `golang_socket`

* Call `should_tap` in Golang uprobes

* Add `log_error` calls

* Revert "Temporarily disable OpenSSL"

This reverts commit f54d9a453f.

* Fix linter

* Revert "Revert "Temporarily disable OpenSSL""

This reverts commit 2433d867af.

* Change `golang_read_writes` map type from `BPF_RINGBUF` to `BPF_PERF_OUTPUT`

* Rename `golang_read_write` to `golang_event`

* Define an error

* Add comments

* Revert "Revert "Revert "Temporarily disable OpenSSL"""

This reverts commit e5a1de9c71.

* Fix `pollGolang`

* Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""

This reverts commit 6e1bd5d4f3.

* Fix `panic: send on closed channel`

* Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""

This reverts commit 57d0584655.

* Use `findLibraryByPid`

* Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""""

This reverts commit 46f3d290b0.

* Revert "Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""""

This reverts commit 775c833c06.

* Log tapping Golang

* Fix `Poll`

* Refactor `golang_net_http_dialconn_uprobe`

* Remove an excess error check

* Fix `can only use path@version syntax with 'go get' and 'go install' in module-aware mode` error in `tap/tlstapper/bpf-builder/build.sh`

* Unify Golang and OpenSSL under a single perf event buffer and `tls_chunk` struct

* Generate `tlsTapperChunkType` type (enum) as well

* Use kernel page size for the `sys_closes` perf buffer

* Fix the linter error

* Fix `MIZU_GLOBAL_GOLANG_PID` environment variable's functionality

* Rely on tracepoints for file descriptor retrieval in Golang implementation

* Remove the unnecessary changes

* Move common functions into `common.c`

* Declare `lookup_ssl_info` function to reduce duplication

* Fix linter

* Add comments and TODOs

* Remove `MIZU_GLOBAL_GOLANG_PATH` environment variable

* Update the object files

* Fix indentation

* Update object files

* Add `go_abi_internal.h`

* Fix `lookup_ssl_info`

* Convert indentation to spaces

* Add header guard comment

* Add more comments

* Find the `ret` instructions using Capstone Engine and `uprobe` the `return` statements

* Implement `get_fd_from_tcp_conn` function

* Separate SSL contexts to OpenSSL and Go

* Move `get_count_bytes` from `common.c` to `openssl_uprobes.c`

* Rename everything contains Golang to Go

* Reduce duplication in `go_uprobes.c`

* Update the comments

* Install Capstone in CI and Docker native builds

* Update `devops/install-capstone.sh`

* Add Capstone to AArch64 cross-compilation target

* Fix some of the issues on ARM64

* Delete the map element in `_ex_urpobe`

* Remove an unsued `LOG_` macro

* Rename `aquynh` to `capstone-engine`

* Add comment

* Revert "Fix some of the issues on ARM64"

This reverts commit 0b3eceddf4.

* Revert "Revert "Fix some of the issues on ARM64""

This reverts commit 681534ada1.

* Update object files

* Remove unnecessary return

* Increase timeout

* #run_acceptance_tests

* #run_acceptance_tests

* Fix the `arm64v8` sourced builds

* #run_acceptance_tests
2022-06-12 14:14:41 +03:00
lirazyehezkel
e41488ef3e TRA-4276 Folder structure refactor (#1131)
* mizu-common folder structure refactor

* no message

* change images import from absolute to relative

* relative sass imports

* no message

* remove tgz

* spinner style
2022-06-12 11:20:47 +03:00
AmitUp9
533fb71bf4 TRA-4596_Update material UI to v5 (#1132)
* install @mui v5 and change imports

* upgrade MUI v5 in mizu community

* package-lock of mizu community and ui-common

* fix ui-common path

* #run_acceptance_tests

* remove comment

* cr fixes
2022-06-08 15:32:34 +03:00
235 changed files with 12025 additions and 10659 deletions

View File

@@ -26,6 +26,7 @@ jobs:
run: |
sudo apt update
sudo apt install -y libpcap-dev
./devops/install-capstone.sh
- name: Check Agent modified files
id: agent_modified_files
@@ -37,7 +38,7 @@ jobs:
with:
version: latest
working-directory: agent
args: --timeout=3m
args: --timeout=10m
- name: Check shared modified files
id: shared_modified_files
@@ -49,7 +50,7 @@ jobs:
with:
version: latest
working-directory: shared
args: --timeout=3m
args: --timeout=10m
- name: Check tap modified files
id: tap_modified_files
@@ -61,7 +62,7 @@ jobs:
with:
version: latest
working-directory: tap
args: --timeout=3m
args: --timeout=10m
- name: Check cli modified files
id: cli_modified_files
@@ -73,7 +74,7 @@ jobs:
with:
version: latest
working-directory: cli
args: --timeout=3m
args: --timeout=10m
- name: Check acceptanceTests modified files
id: acceptanceTests_modified_files
@@ -85,7 +86,7 @@ jobs:
with:
version: latest
working-directory: acceptanceTests
args: --timeout=3m
args: --timeout=10m
- name: Check tap/api modified files
id: tap_api_modified_files

View File

@@ -35,6 +35,11 @@ jobs:
run: |
sudo apt-get install libpcap-dev
- name: Install Capstone
shell: bash
run: |
./devops/install-capstone.sh
- name: Check CLI modified files
id: cli_modified_files
run: devops/check_modified_files.sh cli/

View File

@@ -25,29 +25,46 @@ RUN npm run build
### Base builder image for native builds architecture
FROM golang:1.17-alpine AS builder-native-base
ENV CGO_ENABLED=1 GOOS=linux
RUN apk add --no-cache libpcap-dev g++ perl-utils
RUN apk add --no-cache \
libpcap-dev \
g++ \
perl-utils \
curl \
build-base \
binutils-gold \
bash \
clang \
llvm \
libbpf-dev \
linux-headers
COPY devops/install-capstone.sh .
RUN ./install-capstone.sh
### Intermediate builder image for x86-64 to x86-64 native builds
FROM builder-native-base AS builder-from-amd64-to-amd64
ENV GOARCH=amd64
ENV BPF_TARGET=amd64 BPF_CFLAGS="-O2 -g -D__TARGET_ARCH_x86"
### Intermediate builder image for AArch64 to AArch64 native builds
FROM builder-native-base AS builder-from-arm64v8-to-arm64v8
ENV GOARCH=arm64
ENV BPF_TARGET=arm64 BPF_CFLAGS="-O2 -g -D__TARGET_ARCH_arm64"
### Builder image for x86-64 to AArch64 cross-compilation
FROM up9inc/linux-arm64-musl-go-libpcap AS builder-from-amd64-to-arm64v8
FROM up9inc/linux-arm64-musl-go-libpcap-capstone-bpf:capstone-5.0-rc2 AS builder-from-amd64-to-arm64v8
ENV CGO_ENABLED=1 GOOS=linux
ENV GOARCH=arm64 CGO_CFLAGS="-I/work/libpcap"
ENV GOARCH=arm64 CGO_CFLAGS="-I/work/libpcap -I/work/capstone/include"
ENV BPF_TARGET=arm64 BPF_CFLAGS="-O2 -g -D__TARGET_ARCH_arm64 -I/usr/xcc/aarch64-linux-musl-cross/aarch64-linux-musl/include/"
### Builder image for AArch64 to x86-64 cross-compilation
FROM up9inc/linux-x86_64-musl-go-libpcap AS builder-from-arm64v8-to-amd64
FROM up9inc/linux-x86_64-musl-go-libpcap-capstone-bpf:capstone-5.0-rc2 AS builder-from-arm64v8-to-amd64
ENV CGO_ENABLED=1 GOOS=linux
ENV GOARCH=amd64 CGO_CFLAGS="-I/libpcap"
ENV GOARCH=amd64 CGO_CFLAGS="-I/libpcap -I/capstone/include"
ENV BPF_TARGET=amd64 BPF_CFLAGS="-O2 -g -D__TARGET_ARCH_x86 -I/usr/local/musl/x86_64-unknown-linux-musl/include/"
### Final builder image where the build happens
@@ -86,6 +103,11 @@ ARG GIT_BRANCH
ARG BUILD_TIMESTAMP
ARG VER=0.0
WORKDIR /app/tap/tlstapper
RUN rm tlstapper_bpf*
RUN GOARCH=${BUILDARCH} go generate tls_tapper.go
WORKDIR /app/agent-build
RUN go build -ldflags="-extldflags=-static -s -w \
@@ -95,8 +117,8 @@ RUN go build -ldflags="-extldflags=-static -s -w \
-X 'github.com/up9inc/mizu/agent/pkg/version.Ver=${VER}'" -o mizuagent .
# Download Basenine executable, verify the sha1sum
ADD https://github.com/up9inc/basenine/releases/download/v0.8.2/basenine_linux_${GOARCH} ./basenine_linux_${GOARCH}
ADD https://github.com/up9inc/basenine/releases/download/v0.8.2/basenine_linux_${GOARCH}.sha256 ./basenine_linux_${GOARCH}.sha256
ADD https://github.com/up9inc/basenine/releases/download/v0.8.3/basenine_linux_${GOARCH} ./basenine_linux_${GOARCH}
ADD https://github.com/up9inc/basenine/releases/download/v0.8.3/basenine_linux_${GOARCH}.sha256 ./basenine_linux_${GOARCH}.sha256
RUN shasum -a 256 -c basenine_linux_"${GOARCH}".sha256 && \
chmod +x ./basenine_linux_"${GOARCH}" && \

View File

@@ -269,7 +269,7 @@ function checkRightSideResponseBody() {
const responseBody = JSON.parse(decodedBody);
const expectdJsonBody = {
const expectedJsonBody = {
args: RegExp({}),
url: RegExp('http://.*/get'),
headers: {
@@ -279,27 +279,24 @@ function checkRightSideResponseBody() {
}
};
expect(responseBody.args).to.match(expectdJsonBody.args);
expect(responseBody.url).to.match(expectdJsonBody.url);
expect(responseBody.headers['User-Agent']).to.match(expectdJsonBody.headers['User-Agent']);
expect(responseBody.headers['Accept-Encoding']).to.match(expectdJsonBody.headers['Accept-Encoding']);
expect(responseBody.headers['X-Forwarded-Uri']).to.match(expectdJsonBody.headers['X-Forwarded-Uri']);
const expectedStringInJsonBody = RegExp('/api/v1/namespaces/.*/services/.*/proxy/get');
expect(responseBody.args).to.match(expectedJsonBody.args);
expect(responseBody.url).to.match(expectedJsonBody.url);
expect(responseBody.headers['User-Agent']).to.match(expectedJsonBody.headers['User-Agent']);
expect(responseBody.headers['Accept-Encoding']).to.match(expectedJsonBody.headers['Accept-Encoding']);
expect(responseBody.headers['X-Forwarded-Uri']).to.match(expectedJsonBody.headers['X-Forwarded-Uri']);
cy.get(`${Cypress.env('bodyJsonClass')}`).should('have.text', encodedBody);
cy.get(`[data-cy="lineNumbersCheckBoxInput"]`).should('be.disabled');
clickCheckbox('Decode Base64');
cy.get(`[data-cy="lineNumbersCheckBoxInput"]`).should('not.be.disabled');
cy.get(`${Cypress.env('bodyJsonClass')} > `).its('length').should('be.gt', 1).then(linesNum => {
cy.get(`${Cypress.env('bodyJsonClass')} > >`).its('length').should('be.gt', linesNum).then(jsonItemsNum => {
// checkPrettyAndLineNums(decodedBody);
//clickCheckbox('Line numbers');
//checkPrettyOrNothing(jsonItemsNum, decodedBody);
// clickCheckbox('Pretty');
// checkPrettyOrNothing(jsonItemsNum, decodedBody);
//
// clickCheckbox('Line numbers');
// checkOnlyLineNumberes(jsonItemsNum, decodedBody);
checkOnlyLineNumberes(jsonItemsNum, expectedStringInJsonBody);
});
});
});
@@ -309,37 +306,9 @@ function clickCheckbox(type) {
cy.contains(`${type}`).prev().children().click();
}
function checkPrettyAndLineNums(decodedBody) {
decodedBody = decodedBody.replaceAll(' ', '');
cy.get(`${Cypress.env('bodyJsonClass')} >`).then(elements => {
const lines = Object.values(elements);
lines.forEach((line, index) => {
if (line.getAttribute) {
const cleanLine = getCleanLine(line);
const currentLineFromDecodedText = decodedBody.substring(0, cleanLine.length);
expect(cleanLine).to.equal(currentLineFromDecodedText, `expected the text in line number ${index + 1} to match the text that generated by the base64 decoding`)
decodedBody = decodedBody.substring(cleanLine.length);
}
});
});
}
function getCleanLine(lineElement) {
return (lineElement.innerText.substring(0, lineElement.innerText.length - 1)).replaceAll(' ', '');
}
function checkPrettyOrNothing(jsonItems, decodedBody) {
cy.get(`${Cypress.env('bodyJsonClass')} > `).should('have.length', jsonItems).then(text => {
const json = text.text();
expect(json).to.equal(decodedBody);
});
}
function checkOnlyLineNumberes(jsonItems, decodedText) {
cy.get(`${Cypress.env('bodyJsonClass')} >`).should('have.length', 1).and('have.text', decodedText);
cy.get(`${Cypress.env('bodyJsonClass')} > >`).should('have.length', jsonItems)
cy.get(`${Cypress.env('bodyJsonClass')} > >`).should('have.length', jsonItems);
cy.get(`${Cypress.env('bodyJsonClass')} >`).contains(decodedText);
}
function serviceMapCheck() {

View File

@@ -109,6 +109,9 @@ func TestRedis(t *testing.T) {
}
func TestAmqp(t *testing.T) {
t.Skip("ignoredd for now because those tests are not stable")
if testing.Short() {
t.Skip("ignored acceptance test")
}

View File

@@ -57,7 +57,6 @@ kubectl expose deployment redis --type=LoadBalancer --port=6379 -n mizu-tests --
echo "Creating rabbitmq service"
kubectl expose deployment rabbitmq --type=LoadBalancer --port=5672 -n mizu-tests --dry-run=client -o yaml | kubectl apply -f -
# TODO: need to understand how to fail if address already in use
echo "Starting proxy"
rm -f ${PROXY_LOG}
kubectl proxy --port=8080 > ${PROXY_LOG} &
@@ -78,7 +77,6 @@ minikube image load "${MIZU_CI_IMAGE}"
echo "Build cli"
cd cli && make build GIT_BRANCH=ci SUFFIX=ci
# TODO: need to understand how to fail if password is asked (sudo)
echo "Starting tunnel"
rm -f ${TUNNEL_LOG}
minikube tunnel > ${TUNNEL_LOG} &

View File

@@ -6,10 +6,6 @@ import (
"encoding/json"
"fmt"
"io/ioutil"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/util/homedir"
"net/http"
"os"
"os/exec"
@@ -20,6 +16,11 @@ import (
"testing"
"time"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/util/homedir"
"github.com/up9inc/mizu/shared"
)
@@ -175,7 +176,7 @@ func ApplyKubeFilesForTest(t *testing.T, kubeContext string, namespace string, f
return nil
}
func ApplyKubeFile(kubeContext string, namespace string, filename string) (error) {
func ApplyKubeFile(kubeContext string, namespace string, filename string) error {
cmdArgs := []string{
"apply",
"--context", kubeContext,

View File

@@ -6,7 +6,6 @@ require (
github.com/antelman107/net-wait-go v0.0.0-20210623112055-cf684aebda7b
github.com/chanced/openapi v0.0.8
github.com/djherbis/atime v1.1.0
github.com/getkin/kin-openapi v0.89.0
github.com/gin-contrib/pprof v1.3.0
github.com/gin-contrib/static v0.0.1
github.com/gin-gonic/gin v1.7.7
@@ -20,7 +19,7 @@ require (
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/orcaman/concurrent-map v1.0.0
github.com/stretchr/testify v1.7.0
github.com/up9inc/basenine/client/go v0.0.0-20220509204026-c37adfc587f4
github.com/up9inc/basenine/client/go v0.0.0-20220612112747-3b28eeac9c51
github.com/up9inc/mizu/logger v0.0.0
github.com/up9inc/mizu/shared v0.0.0
github.com/up9inc/mizu/tap v0.0.0
@@ -47,18 +46,18 @@ require (
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/MakeNowJust/heredoc v1.0.0 // indirect
github.com/Masterminds/semver v1.5.0 // indirect
github.com/PuerkitoBio/purell v1.1.1 // indirect
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/beevik/etree v1.1.0 // indirect
github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5 // indirect
github.com/chanced/dynamic v0.0.0-20211210164248-f8fadb1d735b // indirect
github.com/cilium/ebpf v0.8.0 // indirect
github.com/cilium/ebpf v0.8.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
github.com/fatih/camelcase v1.0.0 // indirect
github.com/fvbommel/sortorder v1.0.2 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-errors/errors v1.4.2 // indirect
github.com/go-logr/logr v1.2.2 // indirect
@@ -84,6 +83,7 @@ require (
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.14.2 // indirect
github.com/knightsc/gapstone v0.0.0-20191231144527-6fa5afaf11a9 // indirect
github.com/leodido/go-urn v1.2.1 // indirect
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
github.com/mailru/easyjson v0.7.7 // indirect

View File

@@ -77,6 +77,8 @@ github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3
github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd/go.mod h1:64YHyfSL2R96J44Nlwm39UHepQbyR5q10x7iYa1ks2E=
github.com/MakeNowJust/heredoc v1.0.0 h1:cXCdzVdstXyiTqTvfqk9SDHpKNjxuom+DOlyEeQ4pzQ=
github.com/MakeNowJust/heredoc v1.0.0/go.mod h1:mG5amYoWBHf8vpLOuehzbGGw0EHxpZZ6lCpQ4fNJ8LE=
github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=
@@ -126,8 +128,8 @@ github.com/chanced/openapi v0.0.8/go.mod h1:SxE2VMLPw+T7Vq8nwbVVhDF2PigvRF4n5Xyq
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/cilium/ebpf v0.8.0 h1:2V6KSg3FRADVU2BMIRemZ0hV+9OM+aAHhZDjQyjJTAs=
github.com/cilium/ebpf v0.8.0/go.mod h1:f5zLIM0FSNuAkSyLAN7X+Hy6yznlF1mNiWUMfxMtrgk=
github.com/cilium/ebpf v0.8.1 h1:bLSSEbBLqGPXxls55pGr5qWZaTqcmfDJHhou7t254ao=
github.com/cilium/ebpf v0.8.1/go.mod h1:f5zLIM0FSNuAkSyLAN7X+Hy6yznlF1mNiWUMfxMtrgk=
github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag=
github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
@@ -209,9 +211,6 @@ github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui72
github.com/fvbommel/sortorder v1.0.2 h1:mV4o8B2hKboCdkJm+a7uX/SIpZob4JzUpc5GGnM45eo=
github.com/fvbommel/sortorder v1.0.2/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
github.com/getkin/kin-openapi v0.89.0 h1:p4nagHchUKGn85z/f+pse4aSh50nIBOYjOhMIku2hiA=
github.com/getkin/kin-openapi v0.89.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/pprof v1.3.0 h1:G9eK6HnbkSqDZBYbzG4wrjCsA4e+cvYAHUZw6W+W9K0=
github.com/gin-contrib/pprof v1.3.0/go.mod h1:waMjT1H9b179t3CxuG1cV3DHpga6ybizwfBaM5OXaB0=
@@ -371,7 +370,6 @@ github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2c
github.com/googleapis/gnostic v0.5.5 h1:9fHAtK0uDfpveeqqo1hkEZJcFvYXAiCN3UutL8F9xHw=
github.com/googleapis/gnostic v0.5.5/go.mod h1:7+EbHbldMins07ALC74bsA81Ovc97DwqyJO1AENw9kA=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
@@ -455,6 +453,8 @@ github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+o
github.com/klauspost/compress v1.9.8/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.14.2 h1:S0OHlFk/Gbon/yauFJ4FfJJF5V0fc5HbBTJazi28pRw=
github.com/klauspost/compress v1.14.2/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
github.com/knightsc/gapstone v0.0.0-20191231144527-6fa5afaf11a9 h1:1KszOoXSFt0aRQ6wxxcKm7QKgfLPI0TWO47UcY/f+vA=
github.com/knightsc/gapstone v0.0.0-20191231144527-6fa5afaf11a9/go.mod h1:1K5hEzsMBLTPdRJKEHqBFJ8Zt2VRqDhomcQ11KH0WW4=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
@@ -694,8 +694,8 @@ github.com/ugorji/go v1.2.6/go.mod h1:anCg0y61KIhDlPZmnH+so+RQbysYVyDko0IMgJv0Nn
github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
github.com/ugorji/go/codec v1.2.6 h1:7kbGefxLoDBuYXOms4yD7223OpNMMPNPZxXk5TvFcyQ=
github.com/ugorji/go/codec v1.2.6/go.mod h1:V6TCNZ4PHqoHGFZuSG1W8nrCzzdgA2DozYxWFFpvxTw=
github.com/up9inc/basenine/client/go v0.0.0-20220509204026-c37adfc587f4 h1:nNOrU1HVH0fnaG7GNhxCc8kNPVL035Iix7ihUF6lZT8=
github.com/up9inc/basenine/client/go v0.0.0-20220509204026-c37adfc587f4/go.mod h1:SvJGPoa/6erhUQV7kvHBwM/0x5LyO6XaG2lUaCaKiUI=
github.com/up9inc/basenine/client/go v0.0.0-20220612112747-3b28eeac9c51 h1:6op+PUYmTlxze3V3f30lWKix3sWqv1M9rvRhyaxbsdQ=
github.com/up9inc/basenine/client/go v0.0.0-20220612112747-3b28eeac9c51/go.mod h1:SvJGPoa/6erhUQV7kvHBwM/0x5LyO6XaG2lUaCaKiUI=
github.com/vishvananda/netns v0.0.0-20211101163701-50045581ed74 h1:gga7acRE695APm9hlsSMoOoE65U4/TcqNj90mc69Rlg=
github.com/vishvananda/netns v0.0.0-20211101163701-50045581ed74/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0=
github.com/wI2L/jsondiff v0.1.1 h1:r2TkoEet7E4JMO5+s1RCY2R0LrNPNHY6hbDeow2hRHw=

View File

@@ -118,7 +118,7 @@ func hostApi(socketHarOutputChannel chan<- *tapApi.OutputChannelItem) *gin.Engin
api.WebSocketRoutes(ginApp, &eventHandlers)
if config.Config.OAS {
if config.Config.OAS.Enable {
routes.OASRoutes(ginApp)
}
@@ -200,7 +200,7 @@ func runInHarReaderMode() {
}
func enableExpFeatureIfNeeded() {
if config.Config.OAS {
if config.Config.OAS.Enable {
oasGenerator := dependency.GetInstance(dependency.OasGeneratorDependency).(oas.OasGenerator)
oasGenerator.Start()
}
@@ -227,7 +227,7 @@ func setUIFlags(uiIndexPath string) error {
return err
}
replacedContent := strings.Replace(string(read), "__IS_OAS_ENABLED__", strconv.FormatBool(config.Config.OAS), 1)
replacedContent := strings.Replace(string(read), "__IS_OAS_ENABLED__", strconv.FormatBool(config.Config.OAS.Enable), 1)
replacedContent = strings.Replace(replacedContent, "__IS_SERVICE_MAP_ENABLED__", strconv.FormatBool(config.Config.ServiceMap), 1)
err = ioutil.WriteFile(uiIndexPath, []byte(replacedContent), 0)
@@ -363,7 +363,7 @@ func handleIncomingMessageAsTapper(socketConnection *websocket.Conn) {
func initializeDependencies() {
dependency.RegisterGenerator(dependency.ServiceMapGeneratorDependency, func() interface{} { return servicemap.GetDefaultServiceMapInstance() })
dependency.RegisterGenerator(dependency.OasGeneratorDependency, func() interface{} { return oas.GetDefaultOasGeneratorInstance() })
dependency.RegisterGenerator(dependency.OasGeneratorDependency, func() interface{} { return oas.GetDefaultOasGeneratorInstance(config.Config.OAS.MaxExampleLen) })
dependency.RegisterGenerator(dependency.EntriesInserter, func() interface{} { return api.GetBasenineEntryInserterInstance() })
dependency.RegisterGenerator(dependency.EntriesProvider, func() interface{} { return &entries.BasenineEntriesProvider{} })
dependency.RegisterGenerator(dependency.EntriesSocketStreamer, func() interface{} { return &api.BasenineEntryStreamer{} })

View File

@@ -1,107 +0,0 @@
package api
import (
"bytes"
"context"
"fmt"
"io/ioutil"
"net/http"
"github.com/getkin/kin-openapi/openapi3"
"github.com/getkin/kin-openapi/openapi3filter"
"github.com/getkin/kin-openapi/routers"
legacyrouter "github.com/getkin/kin-openapi/routers/legacy"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/tap/api"
)
const (
ContractNotApplicable api.ContractStatus = 0
ContractPassed api.ContractStatus = 1
ContractFailed api.ContractStatus = 2
)
func loadOAS(ctx context.Context) (doc *openapi3.T, contractContent string, router routers.Router, err error) {
path := fmt.Sprintf("%s%s", shared.ConfigDirPath, shared.ContractFileName)
bytes, err := ioutil.ReadFile(path)
if err != nil {
return
}
contractContent = string(bytes)
loader := &openapi3.Loader{Context: ctx}
doc, _ = loader.LoadFromData(bytes)
err = doc.Validate(ctx)
if err != nil {
return
}
router, _ = legacyrouter.NewRouter(doc)
return
}
func validateOAS(ctx context.Context, doc *openapi3.T, router routers.Router, req *http.Request, res *http.Response) (isValid bool, reqErr error, resErr error) {
isValid = true
reqErr = nil
resErr = nil
// Find route
route, pathParams, err := router.FindRoute(req)
if err != nil {
return
}
// Validate request
requestValidationInput := &openapi3filter.RequestValidationInput{
Request: req,
PathParams: pathParams,
Route: route,
}
if reqErr = openapi3filter.ValidateRequest(ctx, requestValidationInput); reqErr != nil {
isValid = false
}
responseValidationInput := &openapi3filter.ResponseValidationInput{
RequestValidationInput: requestValidationInput,
Status: res.StatusCode,
Header: res.Header,
}
if res.Body != nil {
body, _ := ioutil.ReadAll(res.Body)
res.Body = ioutil.NopCloser(bytes.NewBuffer(body))
responseValidationInput.SetBodyBytes(body)
}
// Validate response.
if resErr = openapi3filter.ValidateResponse(ctx, responseValidationInput); resErr != nil {
isValid = false
}
return
}
func handleOAS(ctx context.Context, doc *openapi3.T, router routers.Router, req *http.Request, res *http.Response, contractContent string) (contract api.Contract) {
contract = api.Contract{
Content: contractContent,
Status: ContractNotApplicable,
}
isValid, reqErr, resErr := validateOAS(ctx, doc, router, req, res)
if isValid {
contract.Status = ContractPassed
} else {
contract.Status = ContractFailed
if reqErr != nil {
contract.RequestReason = reqErr.Error()
} else {
contract.RequestReason = ""
}
if resErr != nil {
contract.ResponseReason = resErr.Error()
} else {
contract.ResponseReason = ""
}
}
return
}

View File

@@ -11,16 +11,15 @@ import (
"strings"
"time"
"github.com/up9inc/mizu/agent/pkg/models"
"github.com/up9inc/mizu/agent/pkg/dependency"
"github.com/up9inc/mizu/agent/pkg/models"
"github.com/up9inc/mizu/agent/pkg/oas"
"github.com/up9inc/mizu/agent/pkg/servicemap"
"github.com/up9inc/mizu/agent/pkg/har"
"github.com/up9inc/mizu/agent/pkg/holder"
"github.com/up9inc/mizu/agent/pkg/providers"
"github.com/up9inc/mizu/agent/pkg/oas"
"github.com/up9inc/mizu/agent/pkg/servicemap"
"github.com/up9inc/mizu/agent/pkg/resolver"
"github.com/up9inc/mizu/agent/pkg/utils"
@@ -100,37 +99,16 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
panic("Channel of captured messages is nil")
}
disableOASValidation := false
ctx := context.Background()
doc, contractContent, router, err := loadOAS(ctx)
if err != nil {
logger.Log.Infof("Disabled OAS validation: %s", err.Error())
disableOASValidation = true
}
for item := range outputItems {
extension := extensionsMap[item.Protocol.Name]
resolvedSource, resolvedDestionation, namespace := resolveIP(item.ConnectionInfo)
if namespace == "" && item.Namespace != tapApi.UNKNOWN_NAMESPACE {
if namespace == "" && item.Namespace != tapApi.UnknownNamespace {
namespace = item.Namespace
}
mizuEntry := extension.Dissector.Analyze(item, resolvedSource, resolvedDestionation, namespace)
if extension.Protocol.Name == "http" {
if !disableOASValidation {
var httpPair tapApi.HTTPRequestResponsePair
if err := json.Unmarshal([]byte(mizuEntry.HTTPPair), &httpPair); err != nil {
logger.Log.Error(err)
} else {
contract := handleOAS(ctx, doc, router, httpPair.Request.Payload.RawRequest, httpPair.Response.Payload.RawResponse, contractContent)
mizuEntry.ContractStatus = contract.Status
mizuEntry.ContractRequestReason = contract.RequestReason
mizuEntry.ContractResponseReason = contract.ResponseReason
mizuEntry.ContractContent = contract.Content
}
}
harEntry, err := har.NewEntry(mizuEntry.Request, mizuEntry.Response, mizuEntry.StartTime, mizuEntry.ElapsedTime)
if err == nil {
rules, _, _ := models.RunValidationRulesState(*harEntry, mizuEntry.Destination.Name)
@@ -144,13 +122,14 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
continue
}
providers.EntryAdded(len(data))
entryInserter := dependency.GetInstance(dependency.EntriesInserter).(EntryInserter)
if err := entryInserter.Insert(mizuEntry); err != nil {
logger.Log.Errorf("Error inserting entry, err: %v", err)
}
summary := extension.Dissector.Summarize(mizuEntry)
providers.EntryAdded(len(data), summary)
serviceMapGenerator := dependency.GetInstance(dependency.ServiceMapGeneratorDependency).(servicemap.ServiceMapSink)
serviceMapGenerator.NewTCPEntry(mizuEntry.Source, mizuEntry.Destination, &item.Protocol)

View File

@@ -44,10 +44,10 @@ func GetEntries(c *gin.Context) {
}
entriesProvider := dependency.GetInstance(dependency.EntriesProvider).(entries.EntriesProvider)
entries, metadata, err := entriesProvider.GetEntries(entriesRequest)
entryWrappers, metadata, err := entriesProvider.GetEntries(entriesRequest)
if !HandleEntriesError(c, err) {
baseEntries := make([]interface{}, 0)
for _, entry := range entries {
for _, entry := range entryWrappers {
baseEntries = append(baseEntries, entry.Base)
}
c.JSON(http.StatusOK, models.EntriesResponse{

View File

@@ -34,12 +34,12 @@ func TestGetOASSpec(t *testing.T) {
func getRecorderAndContext() (*httptest.ResponseRecorder, *gin.Context) {
dependency.RegisterGenerator(dependency.OasGeneratorDependency, func() interface{} {
return oas.GetDefaultOasGeneratorInstance()
return oas.GetDefaultOasGeneratorInstance(-1)
})
recorder := httptest.NewRecorder()
c, _ := gin.CreateTestContext(recorder)
oas.GetDefaultOasGeneratorInstance().Start()
oas.GetDefaultOasGeneratorInstance().GetServiceSpecs().Store("some", oas.NewGen("some"))
oas.GetDefaultOasGeneratorInstance(-1).Start()
oas.GetDefaultOasGeneratorInstance(-1).GetServiceSpecs().Store("some", oas.NewGen("some"))
return recorder, c
}

View File

@@ -79,6 +79,15 @@ func GetGeneralStats(c *gin.Context) {
c.JSON(http.StatusOK, providers.GetGeneralStats())
}
func GetAccumulativeStats(c *gin.Context) {
c.JSON(http.StatusOK, providers.GetAccumulativeStats())
}
func GetAccumulativeStatsTiming(c *gin.Context) {
// for now hardcoded 10 bars of 5 minutes interval
c.JSON(http.StatusOK, providers.GetAccumulativeStatsTiming(300, 10))
}
func GetCurrentResolvingInformation(c *gin.Context) {
c.JSON(http.StatusOK, holder.GetResolver().GetMap())
}

View File

@@ -1,11 +1,11 @@
package dependency
var typeIntializerMap = make(map[DependencyContainerType]func() interface{}, 0)
var typeInitializerMap = make(map[ContainerType]func() interface{}, 0)
func RegisterGenerator(name DependencyContainerType, fn func() interface{}) {
typeIntializerMap[name] = fn
func RegisterGenerator(name ContainerType, fn func() interface{}) {
typeInitializerMap[name] = fn
}
func GetInstance(name DependencyContainerType) interface{} {
return typeIntializerMap[name]()
func GetInstance(name ContainerType) interface{} {
return typeInitializerMap[name]()
}

View File

@@ -1,12 +1,12 @@
package dependency
type DependencyContainerType string
type ContainerType string
const (
ServiceMapGeneratorDependency = "ServiceMapGeneratorDependency"
OasGeneratorDependency = "OasGeneratorDependency"
EntriesInserter = "EntriesInserter"
EntriesProvider = "EntriesProvider"
EntriesSocketStreamer = "EntriesSocketStreamer"
EntryStreamerSocketConnector = "EntryStreamerSocketConnector"
ServiceMapGeneratorDependency ContainerType = "ServiceMapGeneratorDependency"
OasGeneratorDependency ContainerType = "OasGeneratorDependency"
EntriesInserter ContainerType = "EntriesInserter"
EntriesProvider ContainerType = "EntriesProvider"
EntriesSocketStreamer ContainerType = "EntriesSocketStreamer"
EntryStreamerSocketConnector ContainerType = "EntryStreamerSocketConnector"
)

View File

@@ -9,7 +9,7 @@ var ignoredCtypes = []string{"application/javascript", "application/x-javascript
var ignoredHeaders = []string{
"a-im", "accept",
"authorization", "cache-control", "connection", "content-encoding", "content-length", "content-type", "cookie",
"authorization", "cache-control", "connection", "content-encoding", "content-length", "content-range", "content-type", "cookie",
"date", "dnt", "expect", "forwarded", "from", "front-end-https", "host", "http2-settings",
"max-forwards", "origin", "pragma", "proxy-authorization", "proxy-connection", "range", "referer",
"save-data", "te", "trailer", "transfer-encoding", "upgrade", "upgrade-insecure-requests", "x-download-options",

View File

@@ -28,13 +28,14 @@ type OasGenerator interface {
}
type defaultOasGenerator struct {
started bool
serviceSpecs *sync.Map
started bool
serviceSpecs *sync.Map
maxExampleLen int
}
func GetDefaultOasGeneratorInstance() *defaultOasGenerator {
func GetDefaultOasGeneratorInstance(maxExampleLen int) *defaultOasGenerator {
syncOnce.Do(func() {
instance = NewDefaultOasGenerator()
instance = NewDefaultOasGenerator(maxExampleLen)
logger.Log.Debug("OAS Generator Initialized")
})
return instance
@@ -117,6 +118,7 @@ func (g *defaultOasGenerator) getGen(dest string, urlStr string) *SpecGen {
var gen *SpecGen
if !found {
gen = NewGen(u.Scheme + "://" + dest)
gen.MaxExampleLen = g.maxExampleLen
g.serviceSpecs.Store(dest, gen)
} else {
gen = val.(*SpecGen)
@@ -132,9 +134,10 @@ func (g *defaultOasGenerator) GetServiceSpecs() *sync.Map {
return g.serviceSpecs
}
func NewDefaultOasGenerator() *defaultOasGenerator {
func NewDefaultOasGenerator(maxExampleLen int) *defaultOasGenerator {
return &defaultOasGenerator{
started: false,
serviceSpecs: &sync.Map{},
started: false,
serviceSpecs: &sync.Map{},
maxExampleLen: maxExampleLen,
}
}

View File

@@ -8,7 +8,7 @@ import (
)
func TestOASGen(t *testing.T) {
gen := GetDefaultOasGeneratorInstance()
gen := GetDefaultOasGeneratorInstance(-1)
e := new(har.Entry)
err := json.Unmarshal([]byte(`{"startedDateTime": "20000101","request": {"url": "https://host/path", "method": "GET"}, "response": {"status": 200}}`), e)

View File

@@ -42,6 +42,8 @@ type reqResp struct { // hello, generics in Go
}
type SpecGen struct {
MaxExampleLen int // -1 unlimited, 0 and above sets limit
oas *openapi.OpenAPI
tree *Node
lock sync.Mutex
@@ -59,7 +61,11 @@ func NewGen(server string) *SpecGen {
spec.Servers = make([]*openapi.Server, 0)
spec.Servers = append(spec.Servers, &openapi.Server{URL: server})
gen := SpecGen{oas: spec, tree: new(Node)}
gen := SpecGen{
oas: spec,
tree: new(Node),
MaxExampleLen: -1,
}
return &gen
}
@@ -228,7 +234,7 @@ func (g *SpecGen) handlePathObj(entryWithSource *EntryWithSource) (string, error
split = strings.Split(urlParsed.Path, "/")
}
node := g.tree.getOrSet(split, new(openapi.PathObj), entryWithSource.Id)
opObj, err := handleOpObj(entryWithSource, node.pathObj)
opObj, err := handleOpObj(entryWithSource, node.pathObj, g.MaxExampleLen)
if opObj != nil {
return opObj.OperationID, err
@@ -237,7 +243,7 @@ func (g *SpecGen) handlePathObj(entryWithSource *EntryWithSource) (string, error
return "", err
}
func handleOpObj(entryWithSource *EntryWithSource, pathObj *openapi.PathObj) (*openapi.Operation, error) {
func handleOpObj(entryWithSource *EntryWithSource, pathObj *openapi.PathObj, limit int) (*openapi.Operation, error) {
entry := entryWithSource.Entry
isSuccess := 100 <= entry.Response.Status && entry.Response.Status < 400
opObj, wasMissing, err := getOpObj(pathObj, entry.Request.Method, isSuccess)
@@ -250,12 +256,12 @@ func handleOpObj(entryWithSource *EntryWithSource, pathObj *openapi.PathObj) (*o
return nil, nil
}
err = handleRequest(&entry.Request, opObj, isSuccess, entryWithSource.Id)
err = handleRequest(&entry.Request, opObj, isSuccess, entryWithSource.Id, limit)
if err != nil {
return nil, err
}
err = handleResponse(&entry.Response, opObj, isSuccess, entryWithSource.Id)
err = handleResponse(&entry.Response, opObj, isSuccess, entryWithSource.Id, limit)
if err != nil {
return nil, err
}
@@ -342,7 +348,7 @@ func handleCounters(opObj *openapi.Operation, success bool, entryWithSource *Ent
return nil
}
func handleRequest(req *har.Request, opObj *openapi.Operation, isSuccess bool, sampleId string) error {
func handleRequest(req *har.Request, opObj *openapi.Operation, isSuccess bool, sampleId string, limit int) error {
// TODO: we don't handle the situation when header/qstr param can be defined on pathObj level. Also the path param defined on opObj
urlParsed, err := url.Parse(req.URL)
if err != nil {
@@ -390,7 +396,7 @@ func handleRequest(req *har.Request, opObj *openapi.Operation, isSuccess bool, s
} else {
reqCtype, _ := getReqCtype(req)
reqMedia, err := fillContent(reqResp{Req: req}, reqBody.Content, reqCtype, sampleId)
reqMedia, err := fillContent(reqResp{Req: req}, reqBody.Content, reqCtype, sampleId, limit)
if err != nil {
return err
}
@@ -402,7 +408,7 @@ func handleRequest(req *har.Request, opObj *openapi.Operation, isSuccess bool, s
return nil
}
func handleResponse(resp *har.Response, opObj *openapi.Operation, isSuccess bool, sampleId string) error {
func handleResponse(resp *har.Response, opObj *openapi.Operation, isSuccess bool, sampleId string, limit int) error {
// TODO: we don't support "default" response
respObj, err := getResponseObj(resp, opObj, isSuccess)
if err != nil {
@@ -415,7 +421,7 @@ func handleResponse(resp *har.Response, opObj *openapi.Operation, isSuccess bool
respCtype := getRespCtype(resp)
respContent := respObj.Content
respMedia, err := fillContent(reqResp{Resp: resp}, respContent, respCtype, sampleId)
respMedia, err := fillContent(reqResp{Resp: resp}, respContent, respCtype, sampleId, limit)
if err != nil {
return err
}
@@ -467,7 +473,7 @@ func handleRespHeaders(reqHeaders []har.Header, respObj *openapi.ResponseObj, sa
}
}
func fillContent(reqResp reqResp, respContent openapi.Content, ctype string, sampleId string) (*openapi.MediaType, error) {
func fillContent(reqResp reqResp, respContent openapi.Content, ctype string, sampleId string, limit int) (*openapi.MediaType, error) {
content, found := respContent[ctype]
if !found {
respContent[ctype] = &openapi.MediaType{}
@@ -510,7 +516,7 @@ func fillContent(reqResp reqResp, respContent openapi.Content, ctype string, sam
handleFormDataMultipart(text, content, params)
}
if content.Example == nil && len(exampleMsg) > len(content.Example) {
if len(exampleMsg) > len(content.Example) && (limit < 0 || len(exampleMsg) <= limit) {
content.Example = exampleMsg
}
}

View File

@@ -48,7 +48,7 @@ func TestEntries(t *testing.T) {
t.FailNow()
}
gen := NewDefaultOasGenerator()
gen := NewDefaultOasGenerator(-1)
gen.serviceSpecs = new(sync.Map)
loadStartingOAS("test_artifacts/catalogue.json", "catalogue", gen.serviceSpecs)
loadStartingOAS("test_artifacts/trcc.json", "trcc-api-service", gen.serviceSpecs)
@@ -122,7 +122,7 @@ func TestEntries(t *testing.T) {
}
func TestFileSingle(t *testing.T) {
gen := NewDefaultOasGenerator()
gen := NewDefaultOasGenerator(-1)
gen.serviceSpecs = new(sync.Map)
// loadStartingOAS()
file := "test_artifacts/params.har"
@@ -212,7 +212,7 @@ func loadStartingOAS(file string, label string, specs *sync.Map) {
}
func TestEntriesNegative(t *testing.T) {
gen := NewDefaultOasGenerator()
gen := NewDefaultOasGenerator(-1)
gen.serviceSpecs = new(sync.Map)
files := []string{"invalid"}
_, err := feedEntries(files, false, gen)
@@ -223,7 +223,7 @@ func TestEntriesNegative(t *testing.T) {
}
func TestEntriesPositive(t *testing.T) {
gen := NewDefaultOasGenerator()
gen := NewDefaultOasGenerator(-1)
gen.serviceSpecs = new(sync.Map)
files := []string{"test_artifacts/params.har"}
_, err := feedEntries(files, false, gen)

View File

@@ -333,7 +333,7 @@
}
}
},
"example": "agent-id=ade\u0026callback-url=\u0026token=sometoken",
"example": "agent-id=ade\u0026callback-url=\u0026token=sometoken-second-val\u0026optional=another",
"x-sample-entry": "000000000000000000000008"
}
},

View File

@@ -2,7 +2,12 @@ package providers
import (
"reflect"
"sync"
"time"
"github.com/jinzhu/copier"
"github.com/up9inc/mizu/logger"
"github.com/up9inc/mizu/tap/api"
)
type GeneralStats struct {
@@ -12,7 +17,49 @@ type GeneralStats struct {
LastEntryTimestamp int
}
var generalStats = GeneralStats{}
type BucketStats []*TimeFrameStatsValue
type TimeFrameStatsValue struct {
BucketTime time.Time `json:"timestamp"`
ProtocolStats map[string]ProtocolStats `json:"protocols"`
}
type ProtocolStats struct {
MethodsStats map[string]*SizeAndEntriesCount `json:"methods"`
Color string `json:"color"`
}
type SizeAndEntriesCount struct {
EntriesCount int `json:"entriesCount"`
VolumeInBytes int `json:"volumeInBytes"`
}
type AccumulativeStatsCounter struct {
Name string `json:"name"`
EntriesCount int `json:"entriesCount"`
VolumeSizeBytes int `json:"volumeSizeBytes"`
}
type AccumulativeStatsProtocol struct {
AccumulativeStatsCounter
Color string `json:"color"`
Methods []*AccumulativeStatsCounter `json:"methods"`
}
type AccumulativeStatsProtocolTime struct {
ProtocolsData []*AccumulativeStatsProtocol `json:"protocols"`
Time int64 `json:"timestamp"`
}
var (
generalStats = GeneralStats{}
bucketsStats = BucketStats{}
bucketStatsLocker = sync.Mutex{}
)
const (
InternalBucketThreshold = time.Minute * 1
)
func ResetGeneralStats() {
generalStats = GeneralStats{}
@@ -22,7 +69,31 @@ func GetGeneralStats() GeneralStats {
return generalStats
}
func EntryAdded(size int) {
func GetAccumulativeStats() []*AccumulativeStatsProtocol {
bucketStatsCopy := getBucketStatsCopy()
if len(bucketStatsCopy) == 0 {
return make([]*AccumulativeStatsProtocol, 0)
}
methodsPerProtocolAggregated, protocolToColor := getAggregatedStatsAllTime(bucketStatsCopy)
return convertAccumulativeStatsDictToArray(methodsPerProtocolAggregated, protocolToColor)
}
func GetAccumulativeStatsTiming(intervalSeconds int, numberOfBars int) []*AccumulativeStatsProtocolTime {
bucketStatsCopy := getBucketStatsCopy()
if len(bucketStatsCopy) == 0 {
return make([]*AccumulativeStatsProtocolTime, 0)
}
firstBucketTime := getFirstBucketTime(time.Now().UTC(), intervalSeconds, numberOfBars)
methodsPerProtocolPerTimeAggregated, protocolToColor := getAggregatedResultTimingFromSpecificTime(intervalSeconds, bucketStatsCopy, firstBucketTime)
return convertAccumulativeStatsTimelineDictToArray(methodsPerProtocolPerTimeAggregated, protocolToColor)
}
func EntryAdded(size int, summery *api.BaseEntry) {
generalStats.EntriesCount++
generalStats.EntriesVolumeInGB += float64(size) / (1 << 30)
@@ -32,5 +103,186 @@ func EntryAdded(size int) {
generalStats.FirstEntryTimestamp = currentTimestamp
}
addToBucketStats(size, summery)
generalStats.LastEntryTimestamp = currentTimestamp
}
func addToBucketStats(size int, summery *api.BaseEntry) {
entryTimeBucketRounded := getBucketFromTimeStamp(summery.Timestamp)
if len(bucketsStats) == 0 {
bucketsStats = append(bucketsStats, &TimeFrameStatsValue{
BucketTime: entryTimeBucketRounded,
ProtocolStats: map[string]ProtocolStats{},
})
}
bucketOfEntry := bucketsStats[len(bucketsStats)-1]
if bucketOfEntry.BucketTime != entryTimeBucketRounded {
bucketOfEntry = &TimeFrameStatsValue{
BucketTime: entryTimeBucketRounded,
ProtocolStats: map[string]ProtocolStats{},
}
bucketsStats = append(bucketsStats, bucketOfEntry)
}
if _, found := bucketOfEntry.ProtocolStats[summery.Protocol.Abbreviation]; !found {
bucketOfEntry.ProtocolStats[summery.Protocol.Abbreviation] = ProtocolStats{
MethodsStats: map[string]*SizeAndEntriesCount{},
Color: summery.Protocol.BackgroundColor,
}
}
if _, found := bucketOfEntry.ProtocolStats[summery.Protocol.Abbreviation].MethodsStats[summery.Method]; !found {
bucketOfEntry.ProtocolStats[summery.Protocol.Abbreviation].MethodsStats[summery.Method] = &SizeAndEntriesCount{
VolumeInBytes: 0,
EntriesCount: 0,
}
}
bucketOfEntry.ProtocolStats[summery.Protocol.Abbreviation].MethodsStats[summery.Method].EntriesCount += 1
bucketOfEntry.ProtocolStats[summery.Protocol.Abbreviation].MethodsStats[summery.Method].VolumeInBytes += size
}
func getBucketFromTimeStamp(timestamp int64) time.Time {
entryTimeStampAsTime := time.UnixMilli(timestamp)
return entryTimeStampAsTime.Add(-1 * InternalBucketThreshold / 2).Round(InternalBucketThreshold)
}
func getFirstBucketTime(endTime time.Time, intervalSeconds int, numberOfBars int) time.Time {
lastBucketTime := endTime.Add(-1 * time.Second * time.Duration(intervalSeconds) / 2).Round(time.Second * time.Duration(intervalSeconds))
firstBucketTime := lastBucketTime.Add(-1 * time.Second * time.Duration(intervalSeconds*(numberOfBars-1)))
return firstBucketTime
}
func convertAccumulativeStatsTimelineDictToArray(methodsPerProtocolPerTimeAggregated map[time.Time]map[string]map[string]*AccumulativeStatsCounter, protocolToColor map[string]string) []*AccumulativeStatsProtocolTime {
finalResult := make([]*AccumulativeStatsProtocolTime, 0)
for timeKey, item := range methodsPerProtocolPerTimeAggregated {
protocolsData := make([]*AccumulativeStatsProtocol, 0)
for protocolName := range item {
entriesCount := 0
volumeSizeBytes := 0
methods := make([]*AccumulativeStatsCounter, 0)
for _, methodAccData := range methodsPerProtocolPerTimeAggregated[timeKey][protocolName] {
entriesCount += methodAccData.EntriesCount
volumeSizeBytes += methodAccData.VolumeSizeBytes
methods = append(methods, methodAccData)
}
protocolsData = append(protocolsData, &AccumulativeStatsProtocol{
AccumulativeStatsCounter: AccumulativeStatsCounter{
Name: protocolName,
EntriesCount: entriesCount,
VolumeSizeBytes: volumeSizeBytes,
},
Color: protocolToColor[protocolName],
Methods: methods,
})
}
finalResult = append(finalResult, &AccumulativeStatsProtocolTime{
Time: timeKey.UnixMilli(),
ProtocolsData: protocolsData,
})
}
return finalResult
}
func convertAccumulativeStatsDictToArray(methodsPerProtocolAggregated map[string]map[string]*AccumulativeStatsCounter, protocolToColor map[string]string) []*AccumulativeStatsProtocol {
protocolsData := make([]*AccumulativeStatsProtocol, 0)
for protocolName, value := range methodsPerProtocolAggregated {
entriesCount := 0
volumeSizeBytes := 0
methods := make([]*AccumulativeStatsCounter, 0)
for _, methodAccData := range value {
entriesCount += methodAccData.EntriesCount
volumeSizeBytes += methodAccData.VolumeSizeBytes
methods = append(methods, methodAccData)
}
protocolsData = append(protocolsData, &AccumulativeStatsProtocol{
AccumulativeStatsCounter: AccumulativeStatsCounter{
Name: protocolName,
EntriesCount: entriesCount,
VolumeSizeBytes: volumeSizeBytes,
},
Color: protocolToColor[protocolName],
Methods: methods,
})
}
return protocolsData
}
func getBucketStatsCopy() BucketStats {
bucketStatsCopy := BucketStats{}
bucketStatsLocker.Lock()
if err := copier.Copy(&bucketStatsCopy, bucketsStats); err != nil {
logger.Log.Errorf("Error while copying src stats into temporary copied object")
return nil
}
bucketStatsLocker.Unlock()
return bucketStatsCopy
}
func getAggregatedResultTimingFromSpecificTime(intervalSeconds int, bucketStats BucketStats, firstBucketTime time.Time) (map[time.Time]map[string]map[string]*AccumulativeStatsCounter, map[string]string) {
protocolToColor := map[string]string{}
methodsPerProtocolPerTimeAggregated := map[time.Time]map[string]map[string]*AccumulativeStatsCounter{}
bucketStatsIndex := len(bucketStats) - 1
for bucketStatsIndex >= 0 {
currentBucketTime := bucketStats[bucketStatsIndex].BucketTime
if currentBucketTime.After(firstBucketTime) || currentBucketTime.Equal(firstBucketTime) {
resultBucketRoundedKey := currentBucketTime.Add(-1 * time.Second * time.Duration(intervalSeconds) / 2).Round(time.Second * time.Duration(intervalSeconds))
for protocolName, data := range bucketStats[bucketStatsIndex].ProtocolStats {
if _, ok := protocolToColor[protocolName]; !ok {
protocolToColor[protocolName] = data.Color
}
for methodName, dataOfMethod := range data.MethodsStats {
if _, ok := methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey]; !ok {
methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey] = map[string]map[string]*AccumulativeStatsCounter{}
}
if _, ok := methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey][protocolName]; !ok {
methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey][protocolName] = map[string]*AccumulativeStatsCounter{}
}
if _, ok := methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey][protocolName][methodName]; !ok {
methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey][protocolName][methodName] = &AccumulativeStatsCounter{
Name: methodName,
EntriesCount: 0,
VolumeSizeBytes: 0,
}
}
methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey][protocolName][methodName].EntriesCount += dataOfMethod.EntriesCount
methodsPerProtocolPerTimeAggregated[resultBucketRoundedKey][protocolName][methodName].VolumeSizeBytes += dataOfMethod.VolumeInBytes
}
}
}
bucketStatsIndex--
}
return methodsPerProtocolPerTimeAggregated, protocolToColor
}
func getAggregatedStatsAllTime(bucketStatsCopy BucketStats) (map[string]map[string]*AccumulativeStatsCounter, map[string]string) {
protocolToColor := make(map[string]string, 0)
methodsPerProtocolAggregated := make(map[string]map[string]*AccumulativeStatsCounter, 0)
for _, countersOfTimeFrame := range bucketStatsCopy {
for protocolName, value := range countersOfTimeFrame.ProtocolStats {
if _, ok := protocolToColor[protocolName]; !ok {
protocolToColor[protocolName] = value.Color
}
for method, countersValue := range value.MethodsStats {
if _, found := methodsPerProtocolAggregated[protocolName]; !found {
methodsPerProtocolAggregated[protocolName] = map[string]*AccumulativeStatsCounter{}
}
if _, found := methodsPerProtocolAggregated[protocolName][method]; !found {
methodsPerProtocolAggregated[protocolName][method] = &AccumulativeStatsCounter{
Name: method,
EntriesCount: 0,
VolumeSizeBytes: 0,
}
}
methodsPerProtocolAggregated[protocolName][method].EntriesCount += countersValue.EntriesCount
methodsPerProtocolAggregated[protocolName][method].VolumeSizeBytes += countersValue.VolumeInBytes
}
}
}
return methodsPerProtocolAggregated, protocolToColor
}

View File

@@ -0,0 +1,331 @@
package providers
import (
"fmt"
"reflect"
"testing"
"time"
)
func TestGetBucketOfTimeStamp(t *testing.T) {
tests := map[int64]time.Time{
time.Date(2022, time.Month(1), 1, 10, 34, 45, 0, time.Local).UnixMilli(): time.Date(2022, time.Month(1), 1, 10, 34, 00, 0, time.Local),
time.Date(2022, time.Month(1), 1, 10, 34, 00, 0, time.Local).UnixMilli(): time.Date(2022, time.Month(1), 1, 10, 34, 00, 0, time.Local),
time.Date(2022, time.Month(1), 1, 10, 59, 01, 0, time.Local).UnixMilli(): time.Date(2022, time.Month(1), 1, 10, 59, 00, 0, time.Local),
}
for key, value := range tests {
t.Run(fmt.Sprintf("%v", key), func(t *testing.T) {
actual := getBucketFromTimeStamp(key)
if actual != value {
t.Errorf("unexpected result - expected: %v, actual: %v", value, actual)
}
})
}
}
type DataForBucketBorderFunction struct {
EndTime time.Time
IntervalInSeconds int
NumberOfBars int
}
func TestGetBucketBorders(t *testing.T) {
tests := map[DataForBucketBorderFunction]time.Time{
DataForBucketBorderFunction{
time.Date(2022, time.Month(1), 1, 10, 34, 45, 0, time.UTC),
300,
10,
}: time.Date(2022, time.Month(1), 1, 9, 45, 0, 0, time.UTC),
DataForBucketBorderFunction{
time.Date(2022, time.Month(1), 1, 10, 35, 45, 0, time.UTC),
60,
5,
}: time.Date(2022, time.Month(1), 1, 10, 31, 00, 0, time.UTC),
}
for key, value := range tests {
t.Run(fmt.Sprintf("%v", key), func(t *testing.T) {
actual := getFirstBucketTime(key.EndTime, key.IntervalInSeconds, key.NumberOfBars)
if actual != value {
t.Errorf("unexpected result - expected: %v, actual: %v", value, actual)
}
})
}
}
func TestGetAggregatedStatsAllTime(t *testing.T) {
bucketStatsForTest := BucketStats{
&TimeFrameStatsValue{
BucketTime: time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC),
ProtocolStats: map[string]ProtocolStats{
"http": {
MethodsStats: map[string]*SizeAndEntriesCount{
"get": {
EntriesCount: 1,
VolumeInBytes: 2,
},
"post": {
EntriesCount: 2,
VolumeInBytes: 3,
},
},
},
"kafka": {
MethodsStats: map[string]*SizeAndEntriesCount{
"listTopics": {
EntriesCount: 5,
VolumeInBytes: 6,
},
},
},
},
},
&TimeFrameStatsValue{
BucketTime: time.Date(2022, time.Month(1), 1, 10, 01, 00, 0, time.UTC),
ProtocolStats: map[string]ProtocolStats{
"http": {
MethodsStats: map[string]*SizeAndEntriesCount{
"get": {
EntriesCount: 1,
VolumeInBytes: 2,
},
"post": {
EntriesCount: 2,
VolumeInBytes: 3,
},
},
},
"redis": {
MethodsStats: map[string]*SizeAndEntriesCount{
"set": {
EntriesCount: 5,
VolumeInBytes: 6,
},
},
},
},
},
}
expected := map[string]map[string]*AccumulativeStatsCounter{
"http": {
"post": {
Name: "post",
EntriesCount: 4,
VolumeSizeBytes: 6,
},
"get": {
Name: "get",
EntriesCount: 2,
VolumeSizeBytes: 4,
},
},
"kafka": {
"listTopics": {
Name: "listTopics",
EntriesCount: 5,
VolumeSizeBytes: 6,
},
},
"redis": {
"set": {
Name: "set",
EntriesCount: 5,
VolumeSizeBytes: 6,
},
},
}
actual, _ := getAggregatedStatsAllTime(bucketStatsForTest)
if !reflect.DeepEqual(actual, expected) {
t.Errorf("unexpected result - expected: %v, actual: %v", 3, len(actual))
}
}
func TestGetAggregatedStatsFromSpecificTime(t *testing.T) {
bucketStatsForTest := BucketStats{
&TimeFrameStatsValue{
BucketTime: time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC),
ProtocolStats: map[string]ProtocolStats{
"http": {
MethodsStats: map[string]*SizeAndEntriesCount{
"get": {
EntriesCount: 1,
VolumeInBytes: 2,
},
},
},
"kafka": {
MethodsStats: map[string]*SizeAndEntriesCount{
"listTopics": {
EntriesCount: 5,
VolumeInBytes: 6,
},
},
},
},
},
&TimeFrameStatsValue{
BucketTime: time.Date(2022, time.Month(1), 1, 10, 01, 00, 0, time.UTC),
ProtocolStats: map[string]ProtocolStats{
"http": {
MethodsStats: map[string]*SizeAndEntriesCount{
"get": {
EntriesCount: 1,
VolumeInBytes: 2,
},
"post": {
EntriesCount: 2,
VolumeInBytes: 3,
},
},
},
"redis": {
MethodsStats: map[string]*SizeAndEntriesCount{
"set": {
EntriesCount: 5,
VolumeInBytes: 6,
},
},
},
},
},
}
expected := map[time.Time]map[string]map[string]*AccumulativeStatsCounter{
time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC): {
"http": {
"post": {
Name: "post",
EntriesCount: 2,
VolumeSizeBytes: 3,
},
"get": {
Name: "get",
EntriesCount: 2,
VolumeSizeBytes: 4,
},
},
"kafka": {
"listTopics": {
Name: "listTopics",
EntriesCount: 5,
VolumeSizeBytes: 6,
},
},
"redis": {
"set": {
Name: "set",
EntriesCount: 5,
VolumeSizeBytes: 6,
},
},
},
}
actual, _ := getAggregatedResultTimingFromSpecificTime(300, bucketStatsForTest, time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC))
if !reflect.DeepEqual(actual, expected) {
t.Errorf("unexpected result - expected: %v, actual: %v", 3, len(actual))
}
}
func TestGetAggregatedStatsFromSpecificTimeMultipleBuckets(t *testing.T) {
bucketStatsForTest := BucketStats{
&TimeFrameStatsValue{
BucketTime: time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC),
ProtocolStats: map[string]ProtocolStats{
"http": {
MethodsStats: map[string]*SizeAndEntriesCount{
"get": {
EntriesCount: 1,
VolumeInBytes: 2,
},
},
},
"kafka": {
MethodsStats: map[string]*SizeAndEntriesCount{
"listTopics": {
EntriesCount: 5,
VolumeInBytes: 6,
},
},
},
},
},
&TimeFrameStatsValue{
BucketTime: time.Date(2022, time.Month(1), 1, 10, 01, 00, 0, time.UTC),
ProtocolStats: map[string]ProtocolStats{
"http": {
MethodsStats: map[string]*SizeAndEntriesCount{
"get": {
EntriesCount: 1,
VolumeInBytes: 2,
},
"post": {
EntriesCount: 2,
VolumeInBytes: 3,
},
},
},
"redis": {
MethodsStats: map[string]*SizeAndEntriesCount{
"set": {
EntriesCount: 5,
VolumeInBytes: 6,
},
},
},
},
},
}
expected := map[time.Time]map[string]map[string]*AccumulativeStatsCounter{
time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC): {
"http": {
"get": {
Name: "get",
EntriesCount: 1,
VolumeSizeBytes: 2,
},
},
"kafka": {
"listTopics": {
Name: "listTopics",
EntriesCount: 5,
VolumeSizeBytes: 6,
},
},
},
time.Date(2022, time.Month(1), 1, 10, 01, 00, 0, time.UTC): {
"http": {
"post": {
Name: "post",
EntriesCount: 2,
VolumeSizeBytes: 3,
},
"get": {
Name: "get",
EntriesCount: 1,
VolumeSizeBytes: 2,
},
},
"redis": {
"set": {
Name: "set",
EntriesCount: 5,
VolumeSizeBytes: 6,
},
},
},
}
actual, _ := getAggregatedResultTimingFromSpecificTime(60, bucketStatsForTest, time.Date(2022, time.Month(1), 1, 10, 00, 00, 0, time.UTC))
if !reflect.DeepEqual(actual, expected) {
t.Errorf("unexpected result - expected: %v, actual: %v", 3, len(actual))
}
}

View File

@@ -3,8 +3,10 @@ package providers_test
import (
"fmt"
"testing"
"time"
"github.com/up9inc/mizu/agent/pkg/providers"
"github.com/up9inc/mizu/tap/api"
)
func TestNoEntryAddedCount(t *testing.T) {
@@ -22,10 +24,13 @@ func TestNoEntryAddedCount(t *testing.T) {
func TestEntryAddedCount(t *testing.T) {
tests := []int{1, 5, 10, 100, 500, 1000}
entryBucketKey := time.Date(2021, 1, 1, 10, 0, 0, 0, time.UTC)
valueLessThanBucketThreshold := time.Second * 130
mockSummery := &api.BaseEntry{Protocol: api.Protocol{Name: "mock"}, Method: "mock-method", Timestamp: entryBucketKey.Add(valueLessThanBucketThreshold).UnixNano()}
for _, entriesCount := range tests {
t.Run(fmt.Sprintf("%d", entriesCount), func(t *testing.T) {
for i := 0; i < entriesCount; i++ {
providers.EntryAdded(0)
providers.EntryAdded(0, mockSummery)
}
entriesStats := providers.GetGeneralStats()
@@ -38,7 +43,14 @@ func TestEntryAddedCount(t *testing.T) {
t.Errorf("unexpected result - expected: %v, actual: %v", 0, entriesStats.EntriesVolumeInGB)
}
t.Cleanup(providers.ResetGeneralStats)
t.Cleanup(func() {
providers.ResetGeneralStats()
generalStats := providers.GetGeneralStats()
if generalStats.EntriesCount != 0 {
t.Errorf("unexpected result - expected: %v, actual: %v", 0, generalStats.EntriesCount)
}
})
})
}
}
@@ -49,12 +61,14 @@ func TestEntryAddedVolume(t *testing.T) {
var expectedEntriesCount int
var expectedVolumeInGB float64
mockSummery := &api.BaseEntry{Protocol: api.Protocol{Name: "mock"}, Method: "mock-method", Timestamp: time.Date(2021, 1, 1, 10, 0, 0, 0, time.UTC).UnixNano()}
for _, data := range tests {
t.Run(fmt.Sprintf("%d", len(data)), func(t *testing.T) {
expectedEntriesCount++
expectedVolumeInGB += float64(len(data)) / (1 << 30)
providers.EntryAdded(len(data))
providers.EntryAdded(len(data), mockSummery)
entriesStats := providers.GetGeneralStats()
@@ -67,5 +81,4 @@ func TestEntryAddedVolume(t *testing.T) {
}
})
}
}

View File

@@ -44,9 +44,10 @@ func Set(tappedPodsToSet []*shared.PodInfo) {
func GetTappedPodsStatus() []shared.TappedPodStatus {
tappedPodsStatus := make([]shared.TappedPodStatus, 0)
tapperStatus := tappers.GetStatus()
for _, pod := range Get() {
var status string
if tapperStatus, ok := tappers.GetStatus()[pod.NodeName]; ok {
if tapperStatus, ok := tapperStatus[pod.NodeName]; ok {
status = strings.ToLower(tapperStatus.Status)
}

View File

@@ -1,522 +0,0 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0 h1:3ithwDMr7/3vpAMXiH+ZQnYbuIsh+OPhUPMFC9enmn0=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0 h1:xE3CPsOgttP4ACBePh79zTKALtXwn/Edhcr16R5hMWU=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0 h1:/May9ojXjRkPBNVrq+oWLqmWCkr4OU5uRY29bu0mRyQ=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0 h1:Lpy6hKgdcl7a3WGSfJIFmxmcdjSpP6OmBEfcOv1Y680=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0 h1:UDpwYIwla4jHGzZJaEJYx1tOejbgSoNqsAfHAUYe2r8=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9 h1:VpgP7xuJadIUuKccphEpTJnWhS2jkQyMt6Y7pJCD7fY=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs=
github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
github.com/Azure/go-autorest/autorest v0.11.12 h1:gI8ytXbxMfI+IVbI9mP2JGCTXIuhHLgRlvQ9X4PsnHE=
github.com/Azure/go-autorest/autorest v0.11.12/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw=
github.com/Azure/go-autorest/autorest/adal v0.9.5 h1:Y3bBUV4rTuxenJJs41HU3qmqsb+auo+a3Lz+PlJPpL0=
github.com/Azure/go-autorest/autorest/adal v0.9.5/go.mod h1:B7KF7jKIeC9Mct5spmyCB/A8CG/sEz1vwIRGv/bbw7A=
github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw=
github.com/Azure/go-autorest/autorest/date v0.3.0/go.mod h1:BI0uouVdmngYNUzGWeSYnokU+TrmwEsOqdt8Y6sso74=
github.com/Azure/go-autorest/autorest/mocks v0.4.1 h1:K0laFcLE6VLTOwNgSxaGbUcLPuGXlNkbVvq4cW4nIHk=
github.com/Azure/go-autorest/autorest/mocks v0.4.1/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k=
github.com/Azure/go-autorest/logger v0.2.0 h1:e4RVHVZKC5p6UANLJHkM4OfR1UKZPj8Wt8Pcx+3oqrE=
github.com/Azure/go-autorest/logger v0.2.0/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8=
github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo=
github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802 h1:1BDTz0u9nC3//pOCMdNH+CiXJVYJh5UQNCOBG7jbELc=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46 h1:lsxEuwrXEAokXB9qhlbKWPpo3KMLZQ5WB5WLQRW1uq0=
github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=
github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/census-instrumentation/opencensus-proto v0.2.1 h1:glEXhBS5PSLLv4IXzLA5yPRVX4bilULVyxxbrfOtDAk=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10 h1:Swpa1K6QvQznwJRcfTfQJmTE72DqScAa40E+fbHEXEE=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e h1:fY5BOSpyZCqRo5OhCuC+XN+r/bBCmeuuJtjz+bCNIf8=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1 h1:q763qf9huN11kDQavWsoZXJNW3xEE4JJyHa5Q25/sd8=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/creack/pty v1.1.9 h1:uDmaGzcdjhF4i/plgjmEsriH11Y0o7RKapEf/LDaM3w=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815 h1:bWDMxwH3px2JBh6AyO7hdCn/PkvCZXii8TGj7sbtEbQ=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 h1:yUdfgN0XgIJw7foRItutHYUIhlcKzcSf5vDpdhQAKTc=
github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633 h1:H2pdYOb3KQ1/YsqVWoWNLQO+fusocsw354rqGTZtAgw=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473 h1:4cmBvAEBNJaGARUEs3/suWRyfyBfhf7I60WBZq+bv2w=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/protoc-gen-validate v0.1.0 h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/evanphx/json-patch v4.9.0+incompatible h1:kLcOMZeuLAJvL2BPWLMIj5oaZQobrkAqrL+WFZwQses=
github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
github.com/form3tech-oss/jwt-go v3.2.2+incompatible h1:TcekIExNqud5crz4xD2pavyTgWiPvpYe4Xau31I0PRk=
github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1 h1:QbL/5oDUmRBzO9/Z7Seo6zf912W/a6Sr4Eu0G/3Jho0=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4 h1:WtGNWLvXpe6ZudgnXrq0barxBImvnnJoMEhXAzcbM0I=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
github.com/go-logr/logr v0.4.0 h1:K7/B1jt6fIBQVd4Owv2MqGQClcgf0R266+7C/QjRcLc=
github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
github.com/go-openapi/jsonpointer v0.19.3 h1:gihV7YNZK1iK6Tgwwsxo2rJbD1GTbdm72325Bq8FI3w=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
github.com/go-openapi/jsonreference v0.19.3 h1:5cxNfTy0UVC3X8JL5ymxzyoUZmo8iZb+jeTWn7tUa8o=
github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8=
github.com/go-openapi/spec v0.19.3 h1:0XRyw8kguri6Yw4SxhsQA/atC88yqrk0+G4YhI2wabc=
github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo=
github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/swag v0.19.5 h1:lTz6Ys4CmqqCQmZPBlbQENR1/GucA2bzYTE12Pw4tFY=
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1 h1:ocYkMQY5RrXTYgXl7ICpV0IXwlEQGwKIsery4gyXa1U=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=
github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlSUeNy6/ycRUF2yBAKdAQswoHUk=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I=
github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6 h1:UDMh68UUwekSh5iP2OMhRRZJiiBccgV7axzUG8vi56c=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/imdario/mergo v0.3.5 h1:JboBksRwiiAJWvIYJVo46AfV+IAIKZpfrSzVKj42R4Q=
github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/kisielk/errcheck v1.5.0 h1:e8esj/e4R+SAOwFwN+n3zr0nYeCyeweozKfO23MvHzY=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs=
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.5 h1:hyz3dwM5QLc1Rfoz4FuWJQG5BN7tc6K1MndAUnGpQr4=
github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e h1:hB2xlXdHp/pmPZq0y3QnmWAArdw9PqbmotexnWx/FU8=
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8=
github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d h1:7PxY7LVfSZm7PEeBTyK1rj1gABdCO2mbri6GKO1cMDs=
github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.11.0 h1:JAKSXpt1YjtLA7YpPiqO9ss6sNXEsPfSGdwN0UHqzrw=
github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
github.com/onsi/gomega v1.7.0 h1:XPnZz8VVBHjVsy1vzJmRwIcSwiUO+JFfrv/xGiigmME=
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 h1:gQz4mCbXsO+nc9n1hCxHcGA3Zx3Eo+UHZoInFGUIXNM=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/go-internal v1.3.0 h1:RR9dF3JtopPvtkroDZuVD7qquD0bnHlKSqaQhgwt8yk=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/spf13/afero v1.2.2 h1:5jhuqJyZCZf2JRofRvN/nIFgIWNzPa3/Vz8mYylgbWc=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.2.0 h1:Hbg2NidpLE8veEBkEZTL3CvlkUIVzuU9jDplZO54c48=
github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1 h1:ruQGxdhGHe7FWOJPT0mKs5+pD2Xs1Bm/kdGlHO04FmM=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83 h1:/ZScEX8SfEmUGRHs0gxpqteO5nfNW6axyZbBdw9A12g=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6 h1:QE6XYQK6naiK1EPAe1g/ILLxN5RBoH5xkJk3CqlMI/Y=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b h1:+qEpEAPhDZ1o0x3tHzZTQDArnOixOzGD9HUJfcg0mb4=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b h1:Wh+f8QHJXR411sJR8/vRBTZ7YapZaRvUcLFFJhusH0k=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028 h1:4+4C/Iv2U4fMZBiMCc98MG1In4gJY5YRhtpDNeDeHWs=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7 h1:OgUuv8lsRpBibGNbSizVwKWlysjaNzmC9gYMhPVfqFM=
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9 h1:SQFwaSi55rU7vdNs9Yr0Z324VNlrF+0wMqRXT4St8ck=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073 h1:8qxJSnu+7dRq6upnbntrmriWByIakBuct5OM/MdQC1M=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d h1:SZxvLBoTP5yHO3Frd4z4vrF+DBX9vMVanchswa69toE=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4 h1:0YWbFKbhXG/wIiuHDSKpS0Iy7FSA+u45VtBMfQcFTTc=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba h1:O8mE0/t419eoIwhTFpKVkHiTs/Igowgfkj25AcZrtiE=
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a h1:CB3a9Nez8M13wwlr/E2YtwoU+qYHKfC+JrDa45RXXoQ=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.20.0 h1:jz2KixHX7EcCPiQrySzPdnYT7DbINAypCqKZ1Z7GM40=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA=
google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0 h1:0vLT13EuvQ0hNvakwLuFZ/jYrLp5F3kcWHXdRggjCE8=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
honnef.co/go/tools v0.0.1-2020.1.3 h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U=
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
k8s.io/api v0.21.0 h1:gu5iGF4V6tfVCQ/R+8Hc0h7H1JuEhzyEi9S4R5LM8+Y=
k8s.io/api v0.21.0/go.mod h1:+YbrhBBGgsxbF6o6Kj4KJPJnBmAKuXDeS3E18bgHNVU=
k8s.io/apimachinery v0.21.0 h1:3Fx+41if+IRavNcKOz09FwEXDBG6ORh6iMsTSelhkMA=
k8s.io/apimachinery v0.21.0/go.mod h1:jbreFvJo3ov9rj7eWT7+sYiRx+qZuCYXwWT1bcDswPY=
k8s.io/client-go v0.21.0 h1:n0zzzJsAQmJngpC0IhgFcApZyoGXPrDIAD601HD09ag=
k8s.io/client-go v0.21.0/go.mod h1:nNBytTF9qPFDEhoqgEPaarobC8QPae13bElIVHzIglA=
k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac h1:sAvhNk5RRuc6FNYGqe7Ygz3PSo/2wGWbulskmzRX8Vs=
k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0=
k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE=
k8s.io/klog/v2 v2.8.0 h1:Q3gmuM9hKEjefWFFYF0Mat+YyFJvsUyYuwyNNJ5C9Ts=
k8s.io/klog/v2 v2.8.0/go.mod h1:hy9LJ/NvuK+iVyP4Ehqva4HxZG/oXyIS3n3Jmire4Ec=
k8s.io/kube-openapi v0.0.0-20210305001622-591a79e4bda7 h1:vEx13qjvaZ4yfObSSXW7BrMc/KQBBT/Jyee8XtLf4x0=
k8s.io/kube-openapi v0.0.0-20210305001622-591a79e4bda7/go.mod h1:wXW5VT87nVfh/iLV8FpR2uDvrFyomxbtb1KivDbvPTE=
k8s.io/utils v0.0.0-20201110183641-67b214c5f920 h1:CbnUZsM497iRC5QMVkHwyl8s2tB3g7yaSHkYPkpgelw=
k8s.io/utils v0.0.0-20201110183641-67b214c5f920/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA=
rsc.io/binaryregexp v0.2.0 h1:HfqmD5MEmC0zvwBuF187nq9mdnXjXsSivRiXN7SmRkE=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0 h1:9JKUTTIUgS6kzR9mK1YuGKv6Nl+DijDNIc0ghT58FaY=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/QiW4=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sigs.k8s.io/structured-merge-diff/v4 v4.0.2/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw=
sigs.k8s.io/structured-merge-diff/v4 v4.1.0 h1:C4r9BgJ98vrKnnVCjwCSXcWjWe0NKcUQkmzDXZXGwH8=
sigs.k8s.io/structured-merge-diff/v4 v4.1.0/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw=
sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q=
sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc=

View File

@@ -15,9 +15,9 @@ func NewFromInCluster(errOut chan error, namespace string) (*Resolver, error) {
if err != nil {
return nil, err
}
clientset, err := kubernetes.NewForConfig(config)
clientSet, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, err
}
return &Resolver{clientConfig: config, clientSet: clientset, nameMap: cmap.New(), serviceMap: cmap.New(), errOut: errOut, namespace: namespace}, nil
return &Resolver{clientConfig: config, clientSet: clientSet, nameMap: cmap.New(), serviceMap: cmap.New(), errOut: errOut, namespace: namespace}, nil
}

View File

@@ -16,6 +16,8 @@ func StatusRoutes(ginApp *gin.Engine) {
routeGroup.GET("/tap", controllers.GetTappingStatus)
routeGroup.GET("/general", controllers.GetGeneralStats) // get general stats about entries in DB
routeGroup.GET("/accumulative", controllers.GetAccumulativeStats)
routeGroup.GET("/accumulativeTiming", controllers.GetAccumulativeStatsTiming)
routeGroup.GET("/resolving", controllers.GetCurrentResolvingInformation)
}

View File

@@ -56,7 +56,7 @@ func MatchRequestPolicy(harEntry har.Entry, service string) (resultPolicyToSend
}
if rule.Type == "json" {
var bodyJsonMap interface{}
contentTextDecoded, _ := base64.StdEncoding.DecodeString(string(harEntry.Response.Content.Text))
contentTextDecoded, _ := base64.StdEncoding.DecodeString(harEntry.Response.Content.Text)
if err := json.Unmarshal(contentTextDecoded, &bodyJsonMap); err != nil {
continue
}

View File

@@ -1,8 +1,9 @@
package utils
import (
"github.com/djherbis/atime"
"os"
"github.com/djherbis/atime"
)
type ByModTime []os.FileInfo
@@ -19,7 +20,6 @@ func (fis ByModTime) Less(i, j int) bool {
return fis[i].ModTime().Before(fis[j].ModTime())
}
type ByName []os.FileInfo
func (fis ByName) Len() int {
@@ -34,7 +34,6 @@ func (fis ByName) Less(i, j int) bool {
return fis[i].Name() < fis[j].Name()
}
type ByCreationTime []os.FileInfo
func (fis ByCreationTime) Len() int {
@@ -47,4 +46,4 @@ func (fis ByCreationTime) Swap(i, j int) {
func (fis ByCreationTime) Less(i, j int) bool {
return atime.Get(fis[i]).Unix() < atime.Get(fis[j]).Unix()
}
}

View File

@@ -6,7 +6,6 @@ import (
"fmt"
"io/ioutil"
"net/http"
"net/url"
"os"
"os/signal"
"syscall"
@@ -59,17 +58,6 @@ func CheckErr(e error) {
}
}
func SetHostname(address, newHostname string) string {
replacedUrl, err := url.Parse(address)
if err != nil {
logger.Log.Errorf("error replacing hostname to %s in address %s, returning original %v", newHostname, address, err)
return address
}
replacedUrl.Host = newHostname
return replacedUrl.String()
}
func ReadJsonFile(filePath string, value interface{}) error {
if content, err := ioutil.ReadFile(filePath); err != nil {
return err

View File

@@ -25,7 +25,7 @@ func runMizuInstall() {
var sb strings.Builder
sb.WriteString("Hello! This command can be used to install Mizu Pro edition on your Kubernetes cluster.")
sb.WriteString("\nPlease run:")
sb.WriteString("\n\tmizu install -o | kubectl apply -f -")
sb.WriteString("\n\tmizu install -o | kubectl apply -n mizu -f -")
sb.WriteString("\n\nor use helm chart as described in https://getmizu.io/docs/installing-mizu/centralized-installation\n")
fmt.Print(sb.String())

View File

@@ -54,7 +54,6 @@ func init() {
tapCmd.Flags().String(configStructs.InsertionFilterName, defaultTapConfig.InsertionFilter, "Set the insertion filter. Accepts string or a file path.")
tapCmd.Flags().Bool(configStructs.DryRunTapName, defaultTapConfig.DryRun, "Preview of all pods matching the regex, without tapping them")
tapCmd.Flags().String(configStructs.EnforcePolicyFile, defaultTapConfig.EnforcePolicyFile, "Yaml file path with policy rules")
tapCmd.Flags().String(configStructs.ContractFile, defaultTapConfig.ContractFile, "OAS/Swagger file to validate to monitor the contracts")
tapCmd.Flags().Bool(configStructs.ServiceMeshName, defaultTapConfig.ServiceMesh, "Record decrypted traffic if the cluster is configured with a service mesh and with mtls")
tapCmd.Flags().Bool(configStructs.TlsName, defaultTapConfig.Tls, "Record tls traffic")
tapCmd.Flags().Bool(configStructs.ProfilerName, defaultTapConfig.Profiler, "Run pprof server")

View File

@@ -4,7 +4,6 @@ import (
"context"
"errors"
"fmt"
"io/ioutil"
"regexp"
"strings"
"time"
@@ -13,7 +12,6 @@ import (
"github.com/up9inc/mizu/cli/telemetry"
"github.com/up9inc/mizu/cli/utils"
"github.com/getkin/kin-openapi/openapi3"
"gopkg.in/yaml.v3"
core "k8s.io/api/core/v1"
k8serrors "k8s.io/apimachinery/pkg/api/errors"
@@ -57,30 +55,6 @@ func RunMizuTap() {
}
}
// Read and validate the OAS file
var serializedContract string
if config.Config.Tap.ContractFile != "" {
bytes, err := ioutil.ReadFile(config.Config.Tap.ContractFile)
if err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error reading contract file: %v", errormessage.FormatError(err)))
return
}
serializedContract = string(bytes)
ctx := context.Background()
loader := &openapi3.Loader{Context: ctx}
doc, err := loader.LoadFromData(bytes)
if err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error loading contract file: %v", errormessage.FormatError(err)))
return
}
err = doc.Validate(ctx)
if err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error validating contract file: %v", errormessage.FormatError(err)))
return
}
}
kubernetesProvider, err := getKubernetesProviderForCli()
if err != nil {
return
@@ -124,7 +98,7 @@ func RunMizuTap() {
}
logger.Log.Infof("Waiting for Mizu Agent to start...")
if state.mizuServiceAccountExists, err = resources.CreateTapMizuResources(ctx, kubernetesProvider, serializedValidationRules, serializedContract, serializedMizuConfig, config.Config.IsNsRestrictedMode(), config.Config.MizuResourcesNamespace, config.Config.AgentImage, config.Config.Tap.MaxEntriesDBSizeBytes(), config.Config.Tap.ApiServerResources, config.Config.ImagePullPolicy(), config.Config.LogLevel(), config.Config.Tap.Profiler); err != nil {
if state.mizuServiceAccountExists, err = resources.CreateTapMizuResources(ctx, kubernetesProvider, serializedValidationRules, serializedMizuConfig, config.Config.IsNsRestrictedMode(), config.Config.MizuResourcesNamespace, config.Config.AgentImage, config.Config.Tap.MaxEntriesDBSizeBytes(), config.Config.Tap.ApiServerResources, config.Config.ImagePullPolicy(), config.Config.LogLevel(), config.Config.Tap.Profiler); err != nil {
var statusError *k8serrors.StatusError
if errors.As(err, &statusError) && (statusError.ErrStatus.Reason == metav1.StatusReasonAlreadyExists) {
logger.Log.Info("Mizu is already running in this namespace, change the `mizu-resources-namespace` configuration or run `mizu clean` to remove the currently running Mizu instance")

View File

@@ -39,7 +39,7 @@ type ConfigStruct struct {
HeadlessMode bool `yaml:"headless" default:"false"`
LogLevelStr string `yaml:"log-level,omitempty" default:"INFO" readonly:""`
ServiceMap bool `yaml:"service-map" default:"true"`
OAS bool `yaml:"oas" default:"true"`
OAS shared.OASConfig `yaml:"oas"`
}
func (config *ConfigStruct) validate() error {

View File

@@ -24,7 +24,6 @@ const (
InsertionFilterName = "insertion-filter"
DryRunTapName = "dry-run"
EnforcePolicyFile = "traffic-validation-file"
ContractFile = "contract"
ServiceMeshName = "service-mesh"
TlsName = "tls"
ProfilerName = "profiler"
@@ -43,7 +42,6 @@ type TapConfig struct {
InsertionFilter string `yaml:"insertion-filter" default:""`
DryRun bool `yaml:"dry-run" default:"false"`
EnforcePolicyFile string `yaml:"traffic-validation-file"`
ContractFile string `yaml:"contract"`
ApiServerResources shared.Resources `yaml:"api-server-resources"`
TapperResources shared.Resources `yaml:"tapper-resources"`
ServiceMesh bool `yaml:"service-mesh" default:"false"`

View File

@@ -5,16 +5,13 @@ go 1.17
require (
github.com/creasty/defaults v1.5.2
github.com/denisbrodbeck/machineid v1.0.1
github.com/getkin/kin-openapi v0.89.0
github.com/google/go-github/v37 v37.0.0
github.com/google/uuid v1.3.0
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/spf13/cobra v1.3.0
github.com/spf13/pflag v1.0.5
github.com/up9inc/mizu/logger v0.0.0
github.com/up9inc/mizu/shared v0.0.0
github.com/up9inc/mizu/tap/api v0.0.0
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
k8s.io/api v0.23.3
k8s.io/apimachinery v0.23.3
@@ -39,7 +36,6 @@ require (
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
github.com/fvbommel/sortorder v1.0.2 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-errors/errors v1.4.2 // indirect
github.com/go-logr/logr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
@@ -52,8 +48,8 @@ require (
github.com/google/go-cmp v0.5.7 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/googleapis/gnostic v0.5.5 // indirect
github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
github.com/imdario/mergo v0.3.12 // indirect
@@ -79,6 +75,7 @@ require (
go.starlark.net v0.0.0-20220203230714-bb14e151c28f // indirect
golang.org/x/crypto v0.0.0-20220208050332-20e1d8d225ab // indirect
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
golang.org/x/sys v0.0.0-20220207234003-57398862261d // indirect
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
golang.org/x/text v0.3.7 // indirect

View File

@@ -190,9 +190,6 @@ github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui72
github.com/fvbommel/sortorder v1.0.2 h1:mV4o8B2hKboCdkJm+a7uX/SIpZob4JzUpc5GGnM45eo=
github.com/fvbommel/sortorder v1.0.2/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
github.com/getkin/kin-openapi v0.89.0 h1:p4nagHchUKGn85z/f+pse4aSh50nIBOYjOhMIku2hiA=
github.com/getkin/kin-openapi v0.89.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
@@ -294,7 +291,6 @@ github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/
github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=

View File

@@ -10,32 +10,13 @@ import (
"strings"
"time"
"github.com/up9inc/mizu/cli/apiserver"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/cli/pkg/version"
"github.com/up9inc/mizu/logger"
"github.com/google/go-github/v37/github"
"github.com/up9inc/mizu/cli/uiUtils"
)
func CheckVersionCompatibility(apiServerProvider *apiserver.Provider) (bool, error) {
apiVer, err := apiServerProvider.GetVersion()
if err != nil {
return false, err
}
if equals, err := version.AreEquals(apiVer, mizu.Ver); err != nil {
return false, fmt.Errorf("Failed to check version equality between mizuVer: %s and apiVer: %s, error: %w", mizu.Ver, apiVer, err)
} else if !equals {
logger.Log.Errorf(uiUtils.Red, fmt.Sprintf("cli version (%s) is not compatible with api version (%s)", mizu.Ver, apiVer))
return false, nil
}
logger.Log.Debug("cli version %s is compatible with api version %s", mizu.Ver, apiVer)
return true, nil
}
func CheckNewerVersion(versionChan chan string) {
if _, present := os.LookupEnv(mizu.DEVENVVAR); present {
versionChan <- ""

View File

@@ -14,14 +14,14 @@ import (
core "k8s.io/api/core/v1"
)
func CreateTapMizuResources(ctx context.Context, kubernetesProvider *kubernetes.Provider, serializedValidationRules string, serializedContract string, serializedMizuConfig string, isNsRestrictedMode bool, mizuResourcesNamespace string, agentImage string, maxEntriesDBSizeBytes int64, apiServerResources shared.Resources, imagePullPolicy core.PullPolicy, logLevel logging.Level, profiler bool) (bool, error) {
func CreateTapMizuResources(ctx context.Context, kubernetesProvider *kubernetes.Provider, serializedValidationRules string, serializedMizuConfig string, isNsRestrictedMode bool, mizuResourcesNamespace string, agentImage string, maxEntriesDBSizeBytes int64, apiServerResources shared.Resources, imagePullPolicy core.PullPolicy, logLevel logging.Level, profiler bool) (bool, error) {
if !isNsRestrictedMode {
if err := createMizuNamespace(ctx, kubernetesProvider, mizuResourcesNamespace); err != nil {
return false, err
}
}
if err := createMizuConfigmap(ctx, kubernetesProvider, serializedValidationRules, serializedContract, serializedMizuConfig, mizuResourcesNamespace); err != nil {
if err := createMizuConfigmap(ctx, kubernetesProvider, serializedValidationRules, serializedMizuConfig, mizuResourcesNamespace); err != nil {
return false, err
}
@@ -71,8 +71,8 @@ func createMizuNamespace(ctx context.Context, kubernetesProvider *kubernetes.Pro
return err
}
func createMizuConfigmap(ctx context.Context, kubernetesProvider *kubernetes.Provider, serializedValidationRules string, serializedContract string, serializedMizuConfig string, mizuResourcesNamespace string) error {
err := kubernetesProvider.CreateConfigMap(ctx, mizuResourcesNamespace, kubernetes.ConfigMapName, serializedValidationRules, serializedContract, serializedMizuConfig)
func createMizuConfigmap(ctx context.Context, kubernetesProvider *kubernetes.Provider, serializedValidationRules string, serializedMizuConfig string, mizuResourcesNamespace string) error {
err := kubernetesProvider.CreateConfigMap(ctx, mizuResourcesNamespace, kubernetes.ConfigMapName, serializedValidationRules, serializedMizuConfig)
return err
}

View File

@@ -73,11 +73,7 @@ func shouldRunTelemetry() bool {
return false
}
if mizu.Branch != "main" && mizu.Branch != "develop" {
return false
}
return true
return mizu.Branch == "main" || mizu.Branch == "develop"
}
func sendTelemetry(argsMap map[string]interface{}) error {

12
devops/install-capstone.sh Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/bash
SUDO=''
if (( $EUID != 0 )); then
SUDO='sudo'
fi
curl https://github.com/capstone-engine/capstone/releases/download/5.0-rc2/capstone-5.0-rc2.tar.xz -Lo ./capstone.tar.xz \
&& tar -xf capstone.tar.xz && mv ./capstone-* ./capstone \
&& cd capstone \
&& CAPSTONE_ARCHS="aarch64 x86" ./make.sh \
&& $SUDO ./make.sh install

View File

@@ -17,4 +17,14 @@ RUN curl https://www.tcpdump.org/release/libpcap-1.10.1.tar.gz -Lo ./libpcap.tar
WORKDIR /work/libpcap
RUN ./configure --host=arm && make \
&& cp /work/libpcap/libpcap.a /usr/xcc/aarch64-linux-musl-cross/lib/gcc/aarch64-linux-musl/*/
WORKDIR /work
# Build and install Capstone from source
RUN curl https://github.com/capstone-engine/capstone/releases/download/5.0-rc2/capstone-5.0-rc2.tar.xz -Lo ./capstone.tar.xz \
&& tar -xf capstone.tar.xz && mv ./capstone-* ./capstone
WORKDIR /work/capstone
RUN CAPSTONE_ARCHS="aarch64" CAPSTONE_STATIC=yes ./make.sh \
&& cp /work/capstone/libcapstone.a /usr/xcc/aarch64-linux-musl-cross/lib/gcc/aarch64-linux-musl/*/
# Install eBPF related dependencies
RUN apt-get -y install clang llvm libbpf-dev

View File

@@ -0,0 +1,5 @@
#!/bin/bash
set -e
# Build it on x86_64
docker build . -t up9inc/linux-arm64-musl-go-libpcap-capstone-bpf:capstone-5.0-rc2 && docker push up9inc/linux-arm64-musl-go-libpcap-capstone-bpf:capstone-5.0-rc2

View File

@@ -1,4 +0,0 @@
#!/bin/bash
set -e
docker build . -t up9inc/linux-arm64-musl-go-libpcap && docker push up9inc/linux-arm64-musl-go-libpcap

View File

@@ -1,5 +1,18 @@
FROM messense/rust-musl-cross:x86_64-musl AS builder-from-arm64v8-to-amd64
WORKDIR /
# Install eBPF related dependencies
RUN apt-get update
RUN apt-get -y install clang llvm libelf-dev pkg-config
# Build and install libbpf from source
RUN curl https://github.com/libbpf/libbpf/archive/refs/tags/v0.8.0.tar.gz -Lo ./libbpf.tar.gz \
&& tar -xzf libbpf.tar.gz && mv ./libbpf-* ./libbpf
WORKDIR /libbpf/src
RUN make && make install
WORKDIR /
ENV CROSS_TRIPLE x86_64-unknown-linux-musl
ENV CROSS_ROOT /usr/local/musl
@@ -12,7 +25,6 @@ ENV AS=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-as \
FC=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-gfortran
# Install Go
WORKDIR /
RUN curl https://go.dev/dl/go1.17.6.linux-arm64.tar.gz -Lo ./go.linux-arm64.tar.gz \
&& curl https://go.dev/dl/go1.17.6.linux-arm64.tar.gz.asc -Lo ./go.linux-arm64.tar.gz.asc \
&& curl https://dl.google.com/dl/linux/linux_signing_key.pub -Lo linux_signing_key.pub \
@@ -29,3 +41,11 @@ RUN curl https://www.tcpdump.org/release/libpcap-1.10.1.tar.gz -Lo ./libpcap.tar
WORKDIR /libpcap
RUN ./configure --host=x86_64 && make \
&& cp /libpcap/libpcap.a /usr/local/musl/lib/gcc/x86_64-unknown-linux-musl/*/
WORKDIR /
# Build and install Capstone from source
RUN curl https://github.com/capstone-engine/capstone/releases/download/5.0-rc2/capstone-5.0-rc2.tar.xz -Lo ./capstone.tar.xz \
&& tar -xf capstone.tar.xz && mv ./capstone-* ./capstone
WORKDIR /capstone
RUN CAPSTONE_ARCHS="x86" CAPSTONE_STATIC=yes ./make.sh \
&& cp /capstone/libcapstone.a /usr/local/musl/lib/gcc/x86_64-unknown-linux-musl/*/

View File

@@ -0,0 +1,5 @@
#!/bin/bash
set -e
# Build it on arm64
docker build . -t up9inc/linux-x86_64-musl-go-libpcap-capstone-bpf:capstone-5.0-rc2 && docker push up9inc/linux-x86_64-musl-go-libpcap-capstone-bpf:capstone-5.0-rc2

View File

@@ -1,4 +0,0 @@
#!/bin/bash
set -e
docker build . -t up9inc/linux-x86_64-musl-go-libpcap && docker push up9inc/linux-x86_64-musl-go-libpcap

View File

@@ -57,7 +57,7 @@ def extract_samples(f: typing.IO) -> typing.Tuple[pd.Series, pd.Series, pd.Serie
append_sample('"matchedPairs"', line, matched_samples)
append_sample('"liveTcpStreams"', line, live_samples)
append_sample('"processedBytes"', line, processed_samples)
append_sample('mem', line, heap_samples)
append_sample('heap-alloc', line, heap_samples)
append_sample('goroutines', line, goroutines_samples)
cpu_samples = pd.Series(cpu_samples)

View File

@@ -7,7 +7,6 @@ const (
ConfigDirPath = "/app/config/"
DataDirPath = "/app/data/"
ValidationRulesFileName = "validation-rules.yaml"
ContractFileName = "contract-oas.yaml"
ConfigFileName = "mizu-config.json"
DefaultApiServerPort = 8899
LogLevelEnvVar = "LOG_LEVEL"

View File

@@ -684,14 +684,11 @@ func (provider *Provider) handleRemovalError(err error) error {
return err
}
func (provider *Provider) CreateConfigMap(ctx context.Context, namespace string, configMapName string, serializedValidationRules string, serializedContract string, serializedMizuConfig string) error {
func (provider *Provider) CreateConfigMap(ctx context.Context, namespace string, configMapName string, serializedValidationRules string, serializedMizuConfig string) error {
configMapData := make(map[string]string)
if serializedValidationRules != "" {
configMapData[shared.ValidationRulesFileName] = serializedValidationRules
}
if serializedContract != "" {
configMapData[shared.ContractFileName] = serializedContract
}
configMapData[shared.ConfigFileName] = serializedMizuConfig
configMap := &core.ConfigMap{

View File

@@ -11,11 +11,11 @@ import (
)
const (
EventAdded watch.EventType = watch.Added
EventModified watch.EventType = watch.Modified
EventDeleted watch.EventType = watch.Deleted
EventBookmark watch.EventType = watch.Bookmark
EventError watch.EventType = watch.Error
EventAdded = watch.Added
EventModified = watch.Modified
EventDeleted = watch.Deleted
EventBookmark = watch.Bookmark
EventError = watch.Error
)
type InvalidObjectType struct {

View File

@@ -32,6 +32,11 @@ type Resources struct {
MemoryRequests string `yaml:"memory-requests" default:"50Mi"`
}
type OASConfig struct {
Enable bool `yaml:"enabled" default:"true"`
MaxExampleLen int `yaml:"max-example-len" default:"10240"`
}
type MizuAgentConfig struct {
MaxDBSizeBytes int64 `json:"maxDBSizeBytes"`
InsertionFilter string `json:"insertionFilter"`
@@ -42,7 +47,7 @@ type MizuAgentConfig struct {
MizuResourcesNamespace string `json:"mizuResourceNamespace"`
AgentDatabasePath string `json:"agentDatabasePath"`
ServiceMap bool `json:"serviceMap"`
OAS bool `json:"oas"`
OAS OASConfig `json:"oas"`
Telemetry bool `json:"telemetry"`
}
@@ -194,7 +199,7 @@ func DecodeEnforcePolicy(path string) (RulesPolicy, error) {
if err != nil {
return enforcePolicy, err
}
err = yaml.Unmarshal([]byte(content), &enforcePolicy)
err = yaml.Unmarshal(content, &enforcePolicy)
if err != nil {
return enforcePolicy, err
}

View File

@@ -2,27 +2,16 @@ package api
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net"
"net/http"
"os"
"sort"
"sync"
"time"
"github.com/google/martian/har"
"github.com/up9inc/mizu/tap/dbgctl"
)
const mizuTestEnvVar = "MIZU_TEST"
const UNKNOWN_NAMESPACE = ""
const UnknownNamespace = ""
var UnknownIp net.IP = net.IP{0, 0, 0, 0}
var UnknownIp = net.IP{0, 0, 0, 0}
var UnknownPort uint16 = 0
type Protocol struct {
@@ -161,26 +150,21 @@ func (e *Emitting) Emit(item *OutputChannelItem) {
}
type Entry struct {
Id string `json:"id"`
Protocol Protocol `json:"proto"`
Capture Capture `json:"capture"`
Source *TCP `json:"src"`
Destination *TCP `json:"dst"`
Namespace string `json:"namespace"`
Outgoing bool `json:"outgoing"`
Timestamp int64 `json:"timestamp"`
StartTime time.Time `json:"startTime"`
Request map[string]interface{} `json:"request"`
Response map[string]interface{} `json:"response"`
RequestSize int `json:"requestSize"`
ResponseSize int `json:"responseSize"`
ElapsedTime int64 `json:"elapsedTime"`
Rules ApplicableRules `json:"rules,omitempty"`
ContractStatus ContractStatus `json:"contractStatus,omitempty"`
ContractRequestReason string `json:"contractRequestReason,omitempty"`
ContractResponseReason string `json:"contractResponseReason,omitempty"`
ContractContent string `json:"contractContent,omitempty"`
HTTPPair string `json:"httpPair,omitempty"`
Id string `json:"id"`
Protocol Protocol `json:"proto"`
Capture Capture `json:"capture"`
Source *TCP `json:"src"`
Destination *TCP `json:"dst"`
Namespace string `json:"namespace"`
Outgoing bool `json:"outgoing"`
Timestamp int64 `json:"timestamp"`
StartTime time.Time `json:"startTime"`
Request map[string]interface{} `json:"request"`
Response map[string]interface{} `json:"response"`
RequestSize int `json:"requestSize"`
ResponseSize int `json:"responseSize"`
ElapsedTime int64 `json:"elapsedTime"`
Rules ApplicableRules `json:"rules,omitempty"`
}
type EntryWrapper struct {
@@ -193,22 +177,21 @@ type EntryWrapper struct {
}
type BaseEntry struct {
Id string `json:"id"`
Protocol Protocol `json:"proto,omitempty"`
Capture Capture `json:"capture"`
Summary string `json:"summary,omitempty"`
SummaryQuery string `json:"summaryQuery,omitempty"`
Status int `json:"status"`
StatusQuery string `json:"statusQuery"`
Method string `json:"method,omitempty"`
MethodQuery string `json:"methodQuery,omitempty"`
Timestamp int64 `json:"timestamp,omitempty"`
Source *TCP `json:"src"`
Destination *TCP `json:"dst"`
IsOutgoing bool `json:"isOutgoing,omitempty"`
Latency int64 `json:"latency"`
Rules ApplicableRules `json:"rules,omitempty"`
ContractStatus ContractStatus `json:"contractStatus"`
Id string `json:"id"`
Protocol Protocol `json:"proto,omitempty"`
Capture Capture `json:"capture"`
Summary string `json:"summary,omitempty"`
SummaryQuery string `json:"summaryQuery,omitempty"`
Status int `json:"status"`
StatusQuery string `json:"statusQuery"`
Method string `json:"method,omitempty"`
MethodQuery string `json:"methodQuery,omitempty"`
Timestamp int64 `json:"timestamp,omitempty"`
Source *TCP `json:"src"`
Destination *TCP `json:"dst"`
IsOutgoing bool `json:"isOutgoing,omitempty"`
Latency int64 `json:"latency"`
Rules ApplicableRules `json:"rules,omitempty"`
}
type ApplicableRules struct {
@@ -217,15 +200,6 @@ type ApplicableRules struct {
NumberOfRules int `json:"numberOfRules,omitempty"`
}
type ContractStatus int
type Contract struct {
Status ContractStatus `json:"status"`
RequestReason string `json:"requestReason"`
ResponseReason string `json:"responseReason"`
Content string `json:"content"`
}
const (
TABLE string = "table"
BODY string = "body"
@@ -246,170 +220,6 @@ type TableData struct {
Selector string `json:"selector"`
}
const (
TypeHttpRequest = iota
TypeHttpResponse
)
type HTTPPayload struct {
Type uint8
Data interface{}
}
type HTTPPayloader interface {
MarshalJSON() ([]byte, error)
}
type HTTPWrapper struct {
Method string `json:"method"`
Url string `json:"url"`
Details interface{} `json:"details"`
RawRequest *HTTPRequestWrapper `json:"rawRequest"`
RawResponse *HTTPResponseWrapper `json:"rawResponse"`
}
func (h HTTPPayload) MarshalJSON() ([]byte, error) {
_, testEnvEnabled := os.LookupEnv(mizuTestEnvVar)
switch h.Type {
case TypeHttpRequest:
harRequest, err := har.NewRequest(h.Data.(*http.Request), true)
if err != nil {
return nil, errors.New("Failed converting request to HAR")
}
sort.Slice(harRequest.Headers, func(i, j int) bool {
if harRequest.Headers[i].Name < harRequest.Headers[j].Name {
return true
}
if harRequest.Headers[i].Name > harRequest.Headers[j].Name {
return false
}
return harRequest.Headers[i].Value < harRequest.Headers[j].Value
})
sort.Slice(harRequest.QueryString, func(i, j int) bool {
if harRequest.QueryString[i].Name < harRequest.QueryString[j].Name {
return true
}
if harRequest.QueryString[i].Name > harRequest.QueryString[j].Name {
return false
}
return harRequest.QueryString[i].Value < harRequest.QueryString[j].Value
})
if harRequest.PostData != nil {
sort.Slice(harRequest.PostData.Params, func(i, j int) bool {
if harRequest.PostData.Params[i].Name < harRequest.PostData.Params[j].Name {
return true
}
if harRequest.PostData.Params[i].Name > harRequest.PostData.Params[j].Name {
return false
}
return harRequest.PostData.Params[i].Value < harRequest.PostData.Params[j].Value
})
}
if testEnvEnabled {
harRequest.URL = ""
}
var reqWrapper *HTTPRequestWrapper
if !testEnvEnabled {
reqWrapper = &HTTPRequestWrapper{Request: h.Data.(*http.Request)}
}
return json.Marshal(&HTTPWrapper{
Method: harRequest.Method,
Details: harRequest,
RawRequest: reqWrapper,
})
case TypeHttpResponse:
harResponse, err := har.NewResponse(h.Data.(*http.Response), true)
if err != nil {
return nil, errors.New("Failed converting response to HAR")
}
sort.Slice(harResponse.Headers, func(i, j int) bool {
if harResponse.Headers[i].Name < harResponse.Headers[j].Name {
return true
}
if harResponse.Headers[i].Name > harResponse.Headers[j].Name {
return false
}
return harResponse.Headers[i].Value < harResponse.Headers[j].Value
})
sort.Slice(harResponse.Cookies, func(i, j int) bool {
if harResponse.Cookies[i].Name < harResponse.Cookies[j].Name {
return true
}
if harResponse.Cookies[i].Name > harResponse.Cookies[j].Name {
return false
}
return harResponse.Cookies[i].Value < harResponse.Cookies[j].Value
})
var resWrapper *HTTPResponseWrapper
if !testEnvEnabled {
resWrapper = &HTTPResponseWrapper{Response: h.Data.(*http.Response)}
}
return json.Marshal(&HTTPWrapper{
Method: "",
Url: "",
Details: harResponse,
RawResponse: resWrapper,
})
default:
panic(fmt.Sprintf("HTTP payload cannot be marshaled: %v", h.Type))
}
}
type HTTPWrapperTricky struct {
Method string `json:"method"`
Url string `json:"url"`
Details interface{} `json:"details"`
RawRequest *http.Request `json:"rawRequest"`
RawResponse *http.Response `json:"rawResponse"`
}
type HTTPMessage struct {
IsRequest bool `json:"isRequest"`
CaptureTime time.Time `json:"captureTime"`
Payload HTTPWrapperTricky `json:"payload"`
}
type HTTPRequestResponsePair struct {
Request HTTPMessage `json:"request"`
Response HTTPMessage `json:"response"`
}
type HTTPRequestWrapper struct {
*http.Request
}
func (r *HTTPRequestWrapper) MarshalJSON() ([]byte, error) {
body, _ := ioutil.ReadAll(r.Request.Body)
r.Request.Body = ioutil.NopCloser(bytes.NewBuffer(body))
return json.Marshal(&struct { //nolint
Body string `json:"Body,omitempty"`
GetBody string `json:"GetBody,omitempty"`
Cancel string `json:"Cancel,omitempty"`
*http.Request
}{
Body: string(body),
Request: r.Request,
})
}
type HTTPResponseWrapper struct {
*http.Response
}
func (r *HTTPResponseWrapper) MarshalJSON() ([]byte, error) {
body, _ := ioutil.ReadAll(r.Response.Body)
r.Response.Body = ioutil.NopCloser(bytes.NewBuffer(body))
return json.Marshal(&struct { //nolint
Body string `json:"Body,omitempty"`
GetBody string `json:"GetBody,omitempty"`
Cancel string `json:"Cancel,omitempty"`
*http.Response
}{
Body: string(body),
Response: r.Response,
})
}
type TcpReaderDataMsg interface {
GetBytes() []byte
GetTimestamp() time.Time

View File

@@ -2,9 +2,6 @@ module github.com/up9inc/mizu/tap/api
go 1.17
require (
github.com/google/martian v2.1.0+incompatible
github.com/up9inc/mizu/tap/dbgctl v0.0.0
)
require github.com/up9inc/mizu/tap/dbgctl v0.0.0
replace github.com/up9inc/mizu/tap/dbgctl v0.0.0 => ../dbgctl

View File

@@ -1,2 +0,0 @@
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=

View File

@@ -16,6 +16,8 @@ type AppStats struct {
MatchedPairs uint64 `json:"matchedPairs"`
DroppedTcpStreams uint64 `json:"droppedTcpStreams"`
LiveTcpStreams uint64 `json:"liveTcpStreams"`
IgnoredLastAckCount uint64 `json:"ignoredLastAckCount"`
ThrottledPackets uint64 `json:"throttledPackets"`
}
func (as *AppStats) IncMatchedPairs() {
@@ -39,6 +41,14 @@ func (as *AppStats) IncIgnoredPacketsCount() {
atomic.AddUint64(&as.IgnoredPacketsCount, 1)
}
func (as *AppStats) IncIgnoredLastAckCount() {
atomic.AddUint64(&as.IgnoredLastAckCount, 1)
}
func (as *AppStats) IncThrottledPackets() {
atomic.AddUint64(&as.ThrottledPackets, 1)
}
func (as *AppStats) IncReassembledTcpPayloadsCount() {
atomic.AddUint64(&as.ReassembledTcpPayloadsCount, 1)
}
@@ -74,6 +84,8 @@ func (as *AppStats) DumpStats() *AppStats {
currentAppStats.TlsConnectionsCount = resetUint64(&as.TlsConnectionsCount)
currentAppStats.MatchedPairs = resetUint64(&as.MatchedPairs)
currentAppStats.DroppedTcpStreams = resetUint64(&as.DroppedTcpStreams)
currentAppStats.IgnoredLastAckCount = resetUint64(&as.IgnoredLastAckCount)
currentAppStats.ThrottledPackets = resetUint64(&as.ThrottledPackets)
currentAppStats.LiveTcpStreams = as.LiveTcpStreams
return currentAppStats

View File

@@ -10,14 +10,11 @@ import (
)
type CleanerStats struct {
flushed int
closed int
deleted int
}
type Cleaner struct {
assembler *reassembly.Assembler
assemblerMutex *sync.Mutex
cleanPeriod time.Duration
connectionTimeout time.Duration
stats CleanerStats
@@ -28,11 +25,6 @@ type Cleaner struct {
func (cl *Cleaner) clean() {
startCleanTime := time.Now()
cl.assemblerMutex.Lock()
logger.Log.Debugf("Assembler Stats before cleaning %s", cl.assembler.Dump())
flushed, closed := cl.assembler.FlushCloseOlderThan(startCleanTime.Add(-cl.connectionTimeout))
cl.assemblerMutex.Unlock()
cl.streamsMap.Range(func(k, v interface{}) bool {
reqResMatchers := v.(api.TcpStream).GetReqResMatchers()
for _, reqResMatcher := range reqResMatchers {
@@ -47,8 +39,6 @@ func (cl *Cleaner) clean() {
cl.statsMutex.Lock()
logger.Log.Debugf("Assembler Stats after cleaning %s", cl.assembler.Dump())
cl.stats.flushed += flushed
cl.stats.closed += closed
cl.statsMutex.Unlock()
}
@@ -67,17 +57,12 @@ func (cl *Cleaner) dumpStats() CleanerStats {
cl.statsMutex.Lock()
stats := CleanerStats{
flushed: cl.stats.flushed,
closed: cl.stats.closed,
deleted: cl.stats.deleted,
}
cl.stats.flushed = 0
cl.stats.closed = 0
cl.stats.deleted = 0
cl.statsMutex.Unlock()
return stats
}

View File

@@ -5,11 +5,11 @@ import (
)
var (
MizuTapperDisablePcap bool = os.Getenv("MIZU_DEBUG_DISABLE_PCAP") == "true"
MizuTapperDisableTcpReassembly bool = os.Getenv("MIZU_DEBUG_DISABLE_TCP_REASSEMBLY") == "true"
MizuTapperDisableTcpStream bool = os.Getenv("MIZU_DEBUG_DISABLE_TCP_STREAM") == "true"
MizuTapperDisableDissectors bool = os.Getenv("MIZU_DEBUG_DISABLE_DISSECTORS") == "true"
MizuTapperDisableEmitting bool = os.Getenv("MIZU_DEBUG_DISABLE_EMITTING") == "true"
MizuTapperDisableSending bool = os.Getenv("MIZU_DEBUG_DISABLE_SENDING") == "true"
MizuTapperDisableNonHttpExtensions bool = os.Getenv("MIZU_DEBUG_DISABLE_NON_HTTP_EXTENSSION") == "true"
MizuTapperDisablePcap = os.Getenv("MIZU_DEBUG_DISABLE_PCAP") == "true"
MizuTapperDisableTcpReassembly = os.Getenv("MIZU_DEBUG_DISABLE_TCP_REASSEMBLY") == "true"
MizuTapperDisableTcpStream = os.Getenv("MIZU_DEBUG_DISABLE_TCP_STREAM") == "true"
MizuTapperDisableDissectors = os.Getenv("MIZU_DEBUG_DISABLE_DISSECTORS") == "true"
MizuTapperDisableEmitting = os.Getenv("MIZU_DEBUG_DISABLE_EMITTING") == "true"
MizuTapperDisableSending = os.Getenv("MIZU_DEBUG_DISABLE_SENDING") == "true"
MizuTapperDisableNonHttpExtensions = os.Getenv("MIZU_DEBUG_DISABLE_NON_HTTP_EXTENSSION") == "true"
)

View File

@@ -13,4 +13,4 @@ test-pull-bin:
test-pull-expect:
@mkdir -p expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect8/amqp/\* expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect11/amqp/\* expect

View File

@@ -9,7 +9,6 @@ require (
require (
github.com/davecgh/go-spew v1.1.0 // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/up9inc/mizu/tap/dbgctl v0.0.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c // indirect

View File

@@ -1,7 +1,5 @@
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=

View File

@@ -12,7 +12,7 @@ import (
"github.com/up9inc/mizu/tap/api"
)
var protocol api.Protocol = api.Protocol{
var protocol = api.Protocol{
Name: "amqp",
LongName: "Advanced Message Queuing Protocol 0-9-1",
Abbreviation: "AMQP",
@@ -74,13 +74,13 @@ func (d dissecting) Dissect(b *bufio.Reader, reader api.TcpReader, options *api.
var lastMethodFrameMessage Message
for {
frame, err := r.ReadFrame()
frameVal, err := r.readFrame()
if err == io.EOF {
// We must read until we see an EOF... very important!
return err
}
switch f := frame.(type) {
switch f := frameVal.(type) {
case *HeartbeatFrame:
// drop
@@ -203,7 +203,7 @@ func (d dissecting) Dissect(b *bufio.Reader, reader api.TcpReader, options *api.
}
default:
// log.Printf("unexpected frame: %+v", f)
// log.Printf("unexpected frameVal: %+v", f)
}
}
}
@@ -276,22 +276,21 @@ func (d dissecting) Summarize(entry *api.Entry) *api.BaseEntry {
}
return &api.BaseEntry{
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: 0,
StatusQuery: "",
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
ContractStatus: entry.ContractStatus,
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: 0,
StatusQuery: "",
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
}
}

View File

@@ -43,14 +43,14 @@ In realistic implementations where performance is a concern, we would use
“gathering reads” to avoid doing three separate system calls to read a frame.
*/
func (r *AmqpReader) ReadFrame() (frame frame, err error) {
func (r *AmqpReader) readFrame() (frame frame, err error) {
var scratch [7]byte
if _, err = io.ReadFull(r.R, scratch[:7]); err != nil {
return
}
typ := uint8(scratch[0])
typ := scratch[0]
channel := binary.BigEndian.Uint16(scratch[1:3])
size := binary.BigEndian.Uint32(scratch[3:7])
@@ -94,20 +94,20 @@ func (r *AmqpReader) ReadFrame() (frame frame, err error) {
return
}
func readShortstr(r io.Reader) (v string, err error) {
func readShortStr(r io.Reader) (v string, err error) {
var length uint8
if err = binary.Read(r, binary.BigEndian, &length); err != nil {
return
}
bytes := make([]byte, length)
if _, err = io.ReadFull(r, bytes); err != nil {
bytesValue := make([]byte, length)
if _, err = io.ReadFull(r, bytesValue); err != nil {
return
}
return string(bytes), nil
return string(bytesValue), nil
}
func readLongstr(r io.Reader) (v string, err error) {
func readLongStr(r io.Reader) (v string, err error) {
var length uint32
if err = binary.Read(r, binary.BigEndian, &length); err != nil {
return
@@ -118,11 +118,11 @@ func readLongstr(r io.Reader) (v string, err error) {
return
}
bytes := make([]byte, length)
if _, err = io.ReadFull(r, bytes); err != nil {
bytesValue := make([]byte, length)
if _, err = io.ReadFull(r, bytesValue); err != nil {
return
}
return string(bytes), nil
return string(bytesValue), nil
}
func readDecimal(r io.Reader) (v Decimal, err error) {
@@ -171,7 +171,7 @@ func readField(r io.Reader) (v interface{}, err error) {
if err = binary.Read(r, binary.BigEndian, &value); err != nil {
return
}
return (value != 0), nil
return value != 0, nil
case 'b':
var value [1]byte
@@ -219,7 +219,7 @@ func readField(r io.Reader) (v interface{}, err error) {
return readDecimal(r)
case 'S':
return readLongstr(r)
return readLongStr(r)
case 'A':
return readArray(r)
@@ -231,12 +231,12 @@ func readField(r io.Reader) (v interface{}, err error) {
return readTable(r)
case 'x':
var len int32
if err = binary.Read(r, binary.BigEndian, &len); err != nil {
var lenVal int32
if err = binary.Read(r, binary.BigEndian, &lenVal); err != nil {
return nil, err
}
value := make([]byte, len)
value := make([]byte, lenVal)
if _, err = io.ReadFull(r, value); err != nil {
return nil, err
}
@@ -261,7 +261,7 @@ func readTable(r io.Reader) (table Table, err error) {
var nested bytes.Buffer
var str string
if str, err = readLongstr(r); err != nil {
if str, err = readLongStr(r); err != nil {
return
}
@@ -273,7 +273,7 @@ func readTable(r io.Reader) (table Table, err error) {
var key string
var value interface{}
if key, err = readShortstr(&nested); err != nil {
if key, err = readShortStr(&nested); err != nil {
return
}
@@ -349,12 +349,12 @@ func (r *AmqpReader) parseHeaderFrame(channel uint16, size uint32) (frame frame,
}
if hasProperty(flags, flagContentType) {
if hf.Properties.ContentType, err = readShortstr(r.R); err != nil {
if hf.Properties.ContentType, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagContentEncoding) {
if hf.Properties.ContentEncoding, err = readShortstr(r.R); err != nil {
if hf.Properties.ContentEncoding, err = readShortStr(r.R); err != nil {
return
}
}
@@ -374,22 +374,22 @@ func (r *AmqpReader) parseHeaderFrame(channel uint16, size uint32) (frame frame,
}
}
if hasProperty(flags, flagCorrelationId) {
if hf.Properties.CorrelationId, err = readShortstr(r.R); err != nil {
if hf.Properties.CorrelationId, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagReplyTo) {
if hf.Properties.ReplyTo, err = readShortstr(r.R); err != nil {
if hf.Properties.ReplyTo, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagExpiration) {
if hf.Properties.Expiration, err = readShortstr(r.R); err != nil {
if hf.Properties.Expiration, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagMessageId) {
if hf.Properties.MessageId, err = readShortstr(r.R); err != nil {
if hf.Properties.MessageId, err = readShortStr(r.R); err != nil {
return
}
}
@@ -399,22 +399,22 @@ func (r *AmqpReader) parseHeaderFrame(channel uint16, size uint32) (frame frame,
}
}
if hasProperty(flags, flagType) {
if hf.Properties.Type, err = readShortstr(r.R); err != nil {
if hf.Properties.Type, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagUserId) {
if hf.Properties.UserId, err = readShortstr(r.R); err != nil {
if hf.Properties.UserId, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagAppId) {
if hf.Properties.AppId, err = readShortstr(r.R); err != nil {
if hf.Properties.AppId, err = readShortStr(r.R); err != nil {
return
}
}
if hasProperty(flags, flagReserved1) {
if hf.Properties.reserved1, err = readShortstr(r.R); err != nil {
if hf.Properties.reserved1, err = readShortStr(r.R); err != nil {
return
}
}
@@ -436,7 +436,7 @@ func (r *AmqpReader) parseBodyFrame(channel uint16, size uint32) (frame frame, e
return bf, nil
}
var errHeartbeatPayload = errors.New("Heartbeats should not have a payload")
var errHeartbeatPayload = errors.New("heartbeats should not have a payload")
func (r *AmqpReader) parseHeartbeatFrame(channel uint16, size uint32) (frame frame, err error) {
hf := &HeartbeatFrame{

View File

@@ -70,10 +70,10 @@ func (msg *ConnectionStart) read(r io.Reader) (err error) {
return
}
if msg.Mechanisms, err = readLongstr(r); err != nil {
if msg.Mechanisms, err = readLongStr(r); err != nil {
return
}
if msg.Locales, err = readLongstr(r); err != nil {
if msg.Locales, err = readLongStr(r); err != nil {
return
}
@@ -93,15 +93,15 @@ func (msg *ConnectionStartOk) read(r io.Reader) (err error) {
return
}
if msg.Mechanism, err = readShortstr(r); err != nil {
if msg.Mechanism, err = readShortStr(r); err != nil {
return
}
if msg.Response, err = readLongstr(r); err != nil {
if msg.Response, err = readLongStr(r); err != nil {
return
}
if msg.Locale, err = readShortstr(r); err != nil {
if msg.Locale, err = readShortStr(r); err != nil {
return
}
@@ -114,7 +114,7 @@ type connectionSecure struct {
func (msg *connectionSecure) read(r io.Reader) (err error) {
if msg.Challenge, err = readLongstr(r); err != nil {
if msg.Challenge, err = readLongStr(r); err != nil {
return
}
@@ -127,7 +127,7 @@ type connectionSecureOk struct {
func (msg *connectionSecureOk) read(r io.Reader) (err error) {
if msg.Response, err = readLongstr(r); err != nil {
if msg.Response, err = readLongStr(r); err != nil {
return
}
@@ -189,17 +189,17 @@ type connectionOpen struct {
func (msg *connectionOpen) read(r io.Reader) (err error) {
var bits byte
if msg.VirtualHost, err = readShortstr(r); err != nil {
if msg.VirtualHost, err = readShortStr(r); err != nil {
return
}
if msg.reserved1, err = readShortstr(r); err != nil {
if msg.reserved1, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.reserved2 = (bits&(1<<0) > 0)
msg.reserved2 = bits&(1<<0) > 0
return
}
@@ -210,7 +210,7 @@ type connectionOpenOk struct {
func (msg *connectionOpenOk) read(r io.Reader) (err error) {
if msg.reserved1, err = readShortstr(r); err != nil {
if msg.reserved1, err = readShortStr(r); err != nil {
return
}
@@ -230,7 +230,7 @@ func (msg *ConnectionClose) read(r io.Reader) (err error) {
return
}
if msg.ReplyText, err = readShortstr(r); err != nil {
if msg.ReplyText, err = readShortStr(r); err != nil {
return
}
@@ -258,7 +258,7 @@ type connectionBlocked struct {
func (msg *connectionBlocked) read(r io.Reader) (err error) {
if msg.Reason, err = readShortstr(r); err != nil {
if msg.Reason, err = readShortStr(r); err != nil {
return
}
@@ -279,7 +279,7 @@ type channelOpen struct {
func (msg *channelOpen) read(r io.Reader) (err error) {
if msg.reserved1, err = readShortstr(r); err != nil {
if msg.reserved1, err = readShortStr(r); err != nil {
return
}
@@ -292,7 +292,7 @@ type channelOpenOk struct {
func (msg *channelOpenOk) read(r io.Reader) (err error) {
if msg.reserved1, err = readLongstr(r); err != nil {
if msg.reserved1, err = readLongStr(r); err != nil {
return
}
@@ -309,7 +309,7 @@ func (msg *channelFlow) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Active = (bits&(1<<0) > 0)
msg.Active = bits&(1<<0) > 0
return
}
@@ -324,7 +324,7 @@ func (msg *channelFlowOk) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Active = (bits&(1<<0) > 0)
msg.Active = bits&(1<<0) > 0
return
}
@@ -342,7 +342,7 @@ func (msg *channelClose) read(r io.Reader) (err error) {
return
}
if msg.ReplyText, err = readShortstr(r); err != nil {
if msg.ReplyText, err = readShortStr(r); err != nil {
return
}
@@ -383,21 +383,21 @@ func (msg *ExchangeDeclare) read(r io.Reader) (err error) {
return
}
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.Type, err = readShortstr(r); err != nil {
if msg.Type, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Passive = (bits&(1<<0) > 0)
msg.Durable = (bits&(1<<1) > 0)
msg.AutoDelete = (bits&(1<<2) > 0)
msg.Internal = (bits&(1<<3) > 0)
msg.NoWait = (bits&(1<<4) > 0)
msg.Passive = bits&(1<<0) > 0
msg.Durable = bits&(1<<1) > 0
msg.AutoDelete = bits&(1<<2) > 0
msg.Internal = bits&(1<<3) > 0
msg.NoWait = bits&(1<<4) > 0
if msg.Arguments, err = readTable(r); err != nil {
return
@@ -428,15 +428,15 @@ func (msg *exchangeDelete) read(r io.Reader) (err error) {
return
}
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.IfUnused = (bits&(1<<0) > 0)
msg.NoWait = (bits&(1<<1) > 0)
msg.IfUnused = bits&(1<<0) > 0
msg.NoWait = bits&(1<<1) > 0
return
}
@@ -465,20 +465,20 @@ func (msg *exchangeBind) read(r io.Reader) (err error) {
return
}
if msg.Destination, err = readShortstr(r); err != nil {
if msg.Destination, err = readShortStr(r); err != nil {
return
}
if msg.Source, err = readShortstr(r); err != nil {
if msg.Source, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoWait = (bits&(1<<0) > 0)
msg.NoWait = bits&(1<<0) > 0
if msg.Arguments, err = readTable(r); err != nil {
return
@@ -511,20 +511,20 @@ func (msg *exchangeUnbind) read(r io.Reader) (err error) {
return
}
if msg.Destination, err = readShortstr(r); err != nil {
if msg.Destination, err = readShortStr(r); err != nil {
return
}
if msg.Source, err = readShortstr(r); err != nil {
if msg.Source, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoWait = (bits&(1<<0) > 0)
msg.NoWait = bits&(1<<0) > 0
if msg.Arguments, err = readTable(r); err != nil {
return
@@ -559,18 +559,18 @@ func (msg *QueueDeclare) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Passive = (bits&(1<<0) > 0)
msg.Durable = (bits&(1<<1) > 0)
msg.Exclusive = (bits&(1<<2) > 0)
msg.AutoDelete = (bits&(1<<3) > 0)
msg.NoWait = (bits&(1<<4) > 0)
msg.Passive = bits&(1<<0) > 0
msg.Durable = bits&(1<<1) > 0
msg.Exclusive = bits&(1<<2) > 0
msg.AutoDelete = bits&(1<<3) > 0
msg.NoWait = bits&(1<<4) > 0
if msg.Arguments, err = readTable(r); err != nil {
return
@@ -587,7 +587,7 @@ type QueueDeclareOk struct {
func (msg *QueueDeclareOk) read(r io.Reader) (err error) {
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
@@ -617,20 +617,20 @@ func (msg *QueueBind) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoWait = (bits&(1<<0) > 0)
msg.NoWait = bits&(1<<0) > 0
if msg.Arguments, err = readTable(r); err != nil {
return
@@ -661,13 +661,13 @@ func (msg *queueUnbind) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
@@ -699,14 +699,14 @@ func (msg *queuePurge) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoWait = (bits&(1<<0) > 0)
msg.NoWait = bits&(1<<0) > 0
return
}
@@ -739,16 +739,16 @@ func (msg *queueDelete) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.IfUnused = (bits&(1<<0) > 0)
msg.IfEmpty = (bits&(1<<1) > 0)
msg.NoWait = (bits&(1<<2) > 0)
msg.IfUnused = bits&(1<<0) > 0
msg.IfEmpty = bits&(1<<1) > 0
msg.NoWait = bits&(1<<2) > 0
return
}
@@ -786,7 +786,7 @@ func (msg *basicQos) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Global = (bits&(1<<0) > 0)
msg.Global = bits&(1<<0) > 0
return
}
@@ -817,20 +817,20 @@ func (msg *BasicConsume) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if msg.ConsumerTag, err = readShortstr(r); err != nil {
if msg.ConsumerTag, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoLocal = (bits&(1<<0) > 0)
msg.NoAck = (bits&(1<<1) > 0)
msg.Exclusive = (bits&(1<<2) > 0)
msg.NoWait = (bits&(1<<3) > 0)
msg.NoLocal = bits&(1<<0) > 0
msg.NoAck = bits&(1<<1) > 0
msg.Exclusive = bits&(1<<2) > 0
msg.NoWait = bits&(1<<3) > 0
if msg.Arguments, err = readTable(r); err != nil {
return
@@ -845,7 +845,7 @@ type BasicConsumeOk struct {
func (msg *BasicConsumeOk) read(r io.Reader) (err error) {
if msg.ConsumerTag, err = readShortstr(r); err != nil {
if msg.ConsumerTag, err = readShortStr(r); err != nil {
return
}
@@ -860,14 +860,14 @@ type basicCancel struct {
func (msg *basicCancel) read(r io.Reader) (err error) {
var bits byte
if msg.ConsumerTag, err = readShortstr(r); err != nil {
if msg.ConsumerTag, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoWait = (bits&(1<<0) > 0)
msg.NoWait = bits&(1<<0) > 0
return
}
@@ -878,7 +878,7 @@ type basicCancelOk struct {
func (msg *basicCancelOk) read(r io.Reader) (err error) {
if msg.ConsumerTag, err = readShortstr(r); err != nil {
if msg.ConsumerTag, err = readShortStr(r); err != nil {
return
}
@@ -910,18 +910,18 @@ func (msg *BasicPublish) read(r io.Reader) (err error) {
return
}
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Mandatory = (bits&(1<<0) > 0)
msg.Immediate = (bits&(1<<1) > 0)
msg.Mandatory = bits&(1<<0) > 0
msg.Immediate = bits&(1<<1) > 0
return
}
@@ -949,13 +949,13 @@ func (msg *basicReturn) read(r io.Reader) (err error) {
return
}
if msg.ReplyText, err = readShortstr(r); err != nil {
if msg.ReplyText, err = readShortStr(r); err != nil {
return
}
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
@@ -983,7 +983,7 @@ func (msg *BasicDeliver) setContent(props Properties, body []byte) {
func (msg *BasicDeliver) read(r io.Reader) (err error) {
var bits byte
if msg.ConsumerTag, err = readShortstr(r); err != nil {
if msg.ConsumerTag, err = readShortStr(r); err != nil {
return
}
@@ -994,12 +994,12 @@ func (msg *BasicDeliver) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Redelivered = (bits&(1<<0) > 0)
msg.Redelivered = bits&(1<<0) > 0
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
@@ -1019,14 +1019,14 @@ func (msg *basicGet) read(r io.Reader) (err error) {
return
}
if msg.Queue, err = readShortstr(r); err != nil {
if msg.Queue, err = readShortStr(r); err != nil {
return
}
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.NoAck = (bits&(1<<0) > 0)
msg.NoAck = bits&(1<<0) > 0
return
}
@@ -1059,12 +1059,12 @@ func (msg *basicGetOk) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Redelivered = (bits&(1<<0) > 0)
msg.Redelivered = bits&(1<<0) > 0
if msg.Exchange, err = readShortstr(r); err != nil {
if msg.Exchange, err = readShortStr(r); err != nil {
return
}
if msg.RoutingKey, err = readShortstr(r); err != nil {
if msg.RoutingKey, err = readShortStr(r); err != nil {
return
}
@@ -1081,7 +1081,7 @@ type basicGetEmpty struct {
func (msg *basicGetEmpty) read(r io.Reader) (err error) {
if msg.reserved1, err = readShortstr(r); err != nil {
if msg.reserved1, err = readShortStr(r); err != nil {
return
}
@@ -1103,7 +1103,7 @@ func (msg *basicAck) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Multiple = (bits&(1<<0) > 0)
msg.Multiple = bits&(1<<0) > 0
return
}
@@ -1123,7 +1123,7 @@ func (msg *basicReject) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Requeue = (bits&(1<<0) > 0)
msg.Requeue = bits&(1<<0) > 0
return
}
@@ -1138,7 +1138,7 @@ func (msg *basicRecoverAsync) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Requeue = (bits&(1<<0) > 0)
msg.Requeue = bits&(1<<0) > 0
return
}
@@ -1153,7 +1153,7 @@ func (msg *basicRecover) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Requeue = (bits&(1<<0) > 0)
msg.Requeue = bits&(1<<0) > 0
return
}
@@ -1182,8 +1182,8 @@ func (msg *basicNack) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Multiple = (bits&(1<<0) > 0)
msg.Requeue = (bits&(1<<1) > 0)
msg.Multiple = bits&(1<<0) > 0
msg.Requeue = bits&(1<<1) > 0
return
}
@@ -1246,7 +1246,7 @@ func (msg *confirmSelect) read(r io.Reader) (err error) {
if err = binary.Read(r, binary.BigEndian, &bits); err != nil {
return
}
msg.Nowait = (bits&(1<<0) > 0)
msg.Nowait = bits&(1<<0) > 0
return
}

View File

@@ -13,4 +13,4 @@ test-pull-bin:
test-pull-expect:
@mkdir -p expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect10/http/\* expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect12/http/\* expect

View File

@@ -4,6 +4,7 @@ go 1.17
require (
github.com/beevik/etree v1.1.0
github.com/google/martian v2.1.0+incompatible
github.com/mertyildiran/gqlparser/v2 v2.4.6
github.com/stretchr/testify v1.7.0
github.com/up9inc/mizu/tap/api v0.0.0
@@ -12,7 +13,6 @@ require (
require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/up9inc/mizu/tap/dbgctl v0.0.0 // indirect
golang.org/x/text v0.3.7 // indirect

View File

@@ -32,7 +32,7 @@ func replaceForwardedFor(item *api.OutputChannelItem) {
return
}
request := item.Pair.Request.Payload.(api.HTTPPayload).Data.(*http.Request)
request := item.Pair.Request.Payload.(HTTPPayload).Data.(*http.Request)
forwardedFor := request.Header.Get("X-Forwarded-For")
if forwardedFor == "" {

View File

@@ -91,7 +91,7 @@ func (ga *Http2Assembler) readMessage() (streamID uint32, messageHTTP1 interface
// Exactly one Framer is used for each half connection.
// (Instead of creating a new Framer for each ReadFrame operation)
// This is needed in order to decompress the headers,
// because the compression context is updated with each requests/response.
// because the compression context is updated with each request/response.
frame, err := ga.framer.ReadFrame()
if err != nil {
return

View File

@@ -14,7 +14,7 @@ import (
"github.com/up9inc/mizu/tap/api"
)
var http10protocol api.Protocol = api.Protocol{
var http10protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol -- HTTP/1.0",
Abbreviation: "HTTP",
@@ -28,7 +28,7 @@ var http10protocol api.Protocol = api.Protocol{
Priority: 0,
}
var http11protocol api.Protocol = api.Protocol{
var http11protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol -- HTTP/1.1",
Abbreviation: "HTTP",
@@ -42,7 +42,7 @@ var http11protocol api.Protocol = api.Protocol{
Priority: 0,
}
var http2Protocol api.Protocol = api.Protocol{
var http2Protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol Version 2 (HTTP/2)",
Abbreviation: "HTTP/2",
@@ -56,7 +56,7 @@ var http2Protocol api.Protocol = api.Protocol{
Priority: 0,
}
var grpcProtocol api.Protocol = api.Protocol{
var grpcProtocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol Version 2 (HTTP/2) [ gRPC over HTTP/2 ]",
Abbreviation: "gRPC",
@@ -70,7 +70,7 @@ var grpcProtocol api.Protocol = api.Protocol{
Priority: 0,
}
var graphQL1Protocol api.Protocol = api.Protocol{
var graphQL1Protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol -- HTTP/1.1 [ GraphQL over HTTP/1.1 ]",
Abbreviation: "GQL",
@@ -84,7 +84,7 @@ var graphQL1Protocol api.Protocol = api.Protocol{
Priority: 0,
}
var graphQL2Protocol api.Protocol = api.Protocol{
var graphQL2Protocol = api.Protocol{
Name: "http",
LongName: "Hypertext Transfer Protocol Version 2 (HTTP/2) [ GraphQL over HTTP/2 ]",
Abbreviation: "GQL",
@@ -279,7 +279,7 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string,
if elapsedTime < 0 {
elapsedTime = 0
}
httpPair, _ := json.Marshal(item.Pair)
return &api.Entry{
Protocol: item.Protocol,
Capture: item.Capture,
@@ -302,7 +302,6 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string,
Timestamp: item.Timestamp,
StartTime: item.Pair.Request.CaptureTime,
ElapsedTime: elapsedTime,
HTTPPair: string(httpPair),
}
}
@@ -315,22 +314,21 @@ func (d dissecting) Summarize(entry *api.Entry) *api.BaseEntry {
statusQuery := fmt.Sprintf(`response.status == %d`, status)
return &api.BaseEntry{
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: status,
StatusQuery: statusQuery,
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
ContractStatus: entry.ContractStatus,
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: status,
StatusQuery: statusQuery,
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
}
}

View File

@@ -29,7 +29,7 @@ func (matcher *requestResponseMatcher) registerRequest(ident string, request *ht
IsRequest: true,
CaptureTime: captureTime,
CaptureSize: captureSize,
Payload: api.HTTPPayload{
Payload: HTTPPayload{
Type: TypeHttpRequest,
Data: request,
},
@@ -53,7 +53,7 @@ func (matcher *requestResponseMatcher) registerResponse(ident string, response *
IsRequest: false,
CaptureTime: captureTime,
CaptureSize: captureSize,
Payload: api.HTTPPayload{
Payload: HTTPPayload{
Type: TypeHttpResponse,
Data: response,
},

View File

@@ -30,7 +30,7 @@ func IsIgnoredUserAgent(item *api.OutputChannelItem, options *api.TrafficFilteri
return false
}
request := item.Pair.Request.Payload.(api.HTTPPayload).Data.(*http.Request)
request := item.Pair.Request.Payload.(HTTPPayload).Data.(*http.Request)
for headerKey, headerValues := range request.Header {
if strings.ToLower(headerKey) == userAgent {
@@ -50,8 +50,8 @@ func IsIgnoredUserAgent(item *api.OutputChannelItem, options *api.TrafficFilteri
}
func FilterSensitiveData(item *api.OutputChannelItem, options *api.TrafficFilteringOptions) {
request := item.Pair.Request.Payload.(api.HTTPPayload).Data.(*http.Request)
response := item.Pair.Response.Payload.(api.HTTPPayload).Data.(*http.Response)
request := item.Pair.Request.Payload.(HTTPPayload).Data.(*http.Request)
response := item.Pair.Response.Payload.(HTTPPayload).Data.(*http.Response)
filterHeaders(&request.Header)
filterHeaders(&response.Header)
@@ -66,7 +66,7 @@ func filterRequestBody(request *http.Request, options *api.TrafficFilteringOptio
if err != nil {
return
}
filteredBody, err := filterHttpBody([]byte(body), contenType, options)
filteredBody, err := filterHttpBody(body, contenType, options)
if err == nil {
request.Body = ioutil.NopCloser(bytes.NewBuffer(filteredBody))
} else {
@@ -80,7 +80,7 @@ func filterResponseBody(response *http.Response, options *api.TrafficFilteringOp
if err != nil {
return
}
filteredBody, err := filterHttpBody([]byte(body), contentType, options)
filteredBody, err := filterHttpBody(body, contentType, options)
if err == nil {
response.Body = ioutil.NopCloser(bytes.NewBuffer(filteredBody))
} else {

View File

@@ -0,0 +1,100 @@
package http
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"sort"
"github.com/google/martian/har"
)
type HTTPPayload struct {
Type uint8
Data interface{}
}
type HTTPPayloader interface {
MarshalJSON() ([]byte, error)
}
type HTTPWrapper struct {
Method string `json:"method"`
Url string `json:"url"`
Details interface{} `json:"details"`
}
func (h HTTPPayload) MarshalJSON() ([]byte, error) {
switch h.Type {
case TypeHttpRequest:
harRequest, err := har.NewRequest(h.Data.(*http.Request), true)
if err != nil {
return nil, errors.New("Failed converting request to HAR")
}
sort.Slice(harRequest.Headers, func(i, j int) bool {
if harRequest.Headers[i].Name < harRequest.Headers[j].Name {
return true
}
if harRequest.Headers[i].Name > harRequest.Headers[j].Name {
return false
}
return harRequest.Headers[i].Value < harRequest.Headers[j].Value
})
sort.Slice(harRequest.QueryString, func(i, j int) bool {
if harRequest.QueryString[i].Name < harRequest.QueryString[j].Name {
return true
}
if harRequest.QueryString[i].Name > harRequest.QueryString[j].Name {
return false
}
return harRequest.QueryString[i].Value < harRequest.QueryString[j].Value
})
if harRequest.PostData != nil {
sort.Slice(harRequest.PostData.Params, func(i, j int) bool {
if harRequest.PostData.Params[i].Name < harRequest.PostData.Params[j].Name {
return true
}
if harRequest.PostData.Params[i].Name > harRequest.PostData.Params[j].Name {
return false
}
return harRequest.PostData.Params[i].Value < harRequest.PostData.Params[j].Value
})
}
return json.Marshal(&HTTPWrapper{
Method: harRequest.Method,
Url: "",
Details: harRequest,
})
case TypeHttpResponse:
harResponse, err := har.NewResponse(h.Data.(*http.Response), true)
if err != nil {
return nil, errors.New("Failed converting response to HAR")
}
sort.Slice(harResponse.Headers, func(i, j int) bool {
if harResponse.Headers[i].Name < harResponse.Headers[j].Name {
return true
}
if harResponse.Headers[i].Name > harResponse.Headers[j].Name {
return false
}
return harResponse.Headers[i].Value < harResponse.Headers[j].Value
})
sort.Slice(harResponse.Cookies, func(i, j int) bool {
if harResponse.Cookies[i].Name < harResponse.Cookies[j].Name {
return true
}
if harResponse.Cookies[i].Name > harResponse.Cookies[j].Name {
return false
}
return harResponse.Cookies[i].Value < harResponse.Cookies[j].Value
})
return json.Marshal(&HTTPWrapper{
Method: "",
Url: "",
Details: harResponse,
})
default:
panic(fmt.Sprintf("HTTP payload cannot be marshaled: %v", h.Type))
}
}

View File

@@ -13,4 +13,4 @@ test-pull-bin:
test-pull-expect:
@mkdir -p expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect9/kafka/\* expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect11/kafka/\* expect

View File

@@ -14,7 +14,6 @@ require (
require (
github.com/davecgh/go-spew v1.1.0 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/klauspost/compress v1.14.2 // indirect
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect

View File

@@ -11,8 +11,6 @@ github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/klauspost/compress v1.9.8/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.14.2 h1:S0OHlFk/Gbon/yauFJ4FfJJF5V0fc5HbBTJazi28pRw=
github.com/klauspost/compress v1.14.2/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=

View File

@@ -817,12 +817,12 @@ func representDeleteTopicsRequest(data map[string]interface{}) []interface{} {
repPayload, _ := json.Marshal([]api.TableData{
{
Name: "TopicNames",
Value: string(topicNames),
Value: topicNames,
Selector: `request.payload.topicNames`,
},
{
Name: "Topics",
Value: string(topics),
Value: topics,
Selector: `request.payload.topics`,
},
{

View File

@@ -10,7 +10,7 @@ import (
"github.com/up9inc/mizu/tap/api"
)
var _protocol api.Protocol = api.Protocol{
var _protocol = api.Protocol{
Name: "kafka",
LongName: "Apache Kafka Protocol",
Abbreviation: "KAFKA",
@@ -186,22 +186,21 @@ func (d dissecting) Summarize(entry *api.Entry) *api.BaseEntry {
}
return &api.BaseEntry{
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: status,
StatusQuery: statusQuery,
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
ContractStatus: entry.ContractStatus,
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: status,
StatusQuery: statusQuery,
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
}
}

View File

@@ -56,6 +56,6 @@ func makeArray(t reflect.Type, n int) array {
func (a array) index(i int) value { return value{val: a.val.Index(i)} }
func indexOf(s reflect.StructField) index { return index(s.Index) }
func indexOf(s reflect.StructField) index { return s.Index }
func bytesToString(b []byte) string { return string(b) }

View File

@@ -13,4 +13,4 @@ test-pull-bin:
test-pull-expect:
@mkdir -p expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect8/redis/\* expect
@[ "${skipexpect}" ] && echo "Skipping downloading expected JSONs" || gsutil -o 'GSUtil:parallel_process_count=5' -o 'GSUtil:parallel_thread_count=5' -m cp -r gs://static.up9.io/mizu/test-pcap/expect11/redis/\* expect

View File

@@ -9,7 +9,6 @@ require (
require (
github.com/davecgh/go-spew v1.1.0 // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/up9inc/mizu/tap/dbgctl v0.0.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c // indirect

View File

@@ -1,7 +1,5 @@
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=

View File

@@ -10,7 +10,7 @@ import (
"github.com/up9inc/mizu/tap/api"
)
var protocol api.Protocol = api.Protocol{
var protocol = api.Protocol{
Name: "redis",
LongName: "Redis Serialization Protocol",
Abbreviation: "REDIS",
@@ -114,22 +114,21 @@ func (d dissecting) Summarize(entry *api.Entry) *api.BaseEntry {
}
return &api.BaseEntry{
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: status,
StatusQuery: statusQuery,
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
ContractStatus: entry.ContractStatus,
Id: entry.Id,
Protocol: entry.Protocol,
Capture: entry.Capture,
Summary: summary,
SummaryQuery: summaryQuery,
Status: status,
StatusQuery: statusQuery,
Method: method,
MethodQuery: methodQuery,
Timestamp: entry.Timestamp,
Source: entry.Source,
Destination: entry.Destination,
IsOutgoing: entry.Outgoing,
Latency: entry.ElapsedTime,
Rules: entry.Rules,
}
}

View File

@@ -4,7 +4,7 @@ type RedisType string
type RedisCommand string
type RedisKeyword string
var types map[rune]RedisType = map[rune]RedisType{
var types = map[rune]RedisType{
plusByte: "Simple String",
dollarByte: "Bulk String",
asteriskByte: "Array",
@@ -13,7 +13,7 @@ var types map[rune]RedisType = map[rune]RedisType{
notApplicableByte: "N/A",
}
var commands []RedisCommand = []RedisCommand{
var commands = []RedisCommand{
"PING",
"SET",
"GET",
@@ -200,7 +200,7 @@ var commands []RedisCommand = []RedisCommand{
"XCLAIM",
}
var keywords []RedisKeyword = []RedisKeyword{
var keywords = []RedisKeyword{
"AGGREGATE",
"ALPHA",
"ASC",

View File

@@ -3,10 +3,12 @@ module github.com/up9inc/mizu/tap
go 1.17
require (
github.com/cilium/ebpf v0.8.0
github.com/Masterminds/semver v1.5.0
github.com/cilium/ebpf v0.8.1
github.com/go-errors/errors v1.4.2
github.com/google/gopacket v1.1.19
github.com/hashicorp/golang-lru v0.5.4
github.com/knightsc/gapstone v0.0.0-20191231144527-6fa5afaf11a9
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/struCoder/pidusage v0.2.1
github.com/up9inc/mizu/logger v0.0.0
@@ -22,7 +24,6 @@ require (
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/go-cmp v0.5.7 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/martian v2.1.0+incompatible // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect

View File

@@ -1,12 +1,14 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
github.com/Masterminds/semver v1.5.0/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cilium/ebpf v0.8.0 h1:2V6KSg3FRADVU2BMIRemZ0hV+9OM+aAHhZDjQyjJTAs=
github.com/cilium/ebpf v0.8.0/go.mod h1:f5zLIM0FSNuAkSyLAN7X+Hy6yznlF1mNiWUMfxMtrgk=
github.com/cilium/ebpf v0.8.1 h1:bLSSEbBLqGPXxls55pGr5qWZaTqcmfDJHhou7t254ao=
github.com/cilium/ebpf v0.8.1/go.mod h1:f5zLIM0FSNuAkSyLAN7X+Hy6yznlF1mNiWUMfxMtrgk=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -66,8 +68,6 @@ github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=
github.com/google/gopacket v1.1.19/go.mod h1:iJ8V8n6KS+z2U1A8pUwu8bW5SyEMkXJB8Yo/Vo+TKTo=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2cUuW7uA/OeU=
github.com/googleapis/gnostic v0.5.5/go.mod h1:7+EbHbldMins07ALC74bsA81Ovc97DwqyJO1AENw9kA=
@@ -81,6 +81,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/knightsc/gapstone v0.0.0-20191231144527-6fa5afaf11a9 h1:1KszOoXSFt0aRQ6wxxcKm7QKgfLPI0TWO47UcY/f+vA=
github.com/knightsc/gapstone v0.0.0-20191231144527-6fa5afaf11a9/go.mod h1:1K5hEzsMBLTPdRJKEHqBFJ8Zt2VRqDhomcQ11KH0WW4=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
@@ -290,5 +292,6 @@ sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2/go.mod h1:B+TnT182UBxE84DiCz
sigs.k8s.io/structured-merge-diff/v4 v4.0.2/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw=
sigs.k8s.io/structured-merge-diff/v4 v4.2.1 h1:bKCqE9GvQ5tiVHn5rfn1r+yao3aLQEaLzkkmAkf+A6Y=
sigs.k8s.io/structured-merge-diff/v4 v4.2.1/go.mod h1:j/nl6xW8vLS49O8YvXW1ocPhZawJtm+Yrr7PPRQ0Vg4=
sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q=
sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc=
sigs.k8s.io/yaml v1.3.0 h1:a2VclLzOGrwOHDiV8EfBGhvjHvP46CtW5j6POvhYGGo=
sigs.k8s.io/yaml v1.3.0/go.mod h1:GeOyir5tyXNByN85N/dRIT9es5UQNerPYEKK56eTBm8=

View File

@@ -14,9 +14,9 @@ import (
"fmt"
"os"
"runtime"
"strconv"
"strings"
"time"
"strconv"
"github.com/shirou/gopsutil/cpu"
"github.com/struCoder/pidusage"
@@ -45,6 +45,7 @@ var quiet = flag.Bool("quiet", false, "Be quiet regarding errors")
var hexdumppkt = flag.Bool("dumppkt", false, "Dump packet as hex")
var procfs = flag.String("procfs", "/proc", "The procfs directory, used when mapping host volumes into a container")
var ignoredPorts = flag.String("ignore-ports", "", "A comma separated list of ports to ignore")
var maxLiveStreams = flag.Int("max-live-streams", 500, "Maximum live streams to handle concurrently")
// capture
var iface = flag.String("i", "en0", "Interface to read packets from")
@@ -59,8 +60,10 @@ var tls = flag.Bool("tls", false, "Enable TLS tapper")
var memprofile = flag.String("memprofile", "", "Write memory profile")
type TapOpts struct {
HostMode bool
IgnoredPorts []uint16
HostMode bool
IgnoredPorts []uint16
maxLiveStreams int
staleConnectionTimeout time.Duration
}
var extensions []*api.Extension // global
@@ -89,7 +92,13 @@ func StartPassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem,
diagnose.StartMemoryProfiler(os.Getenv(MemoryProfilingDumpPath), os.Getenv(MemoryProfilingTimeIntervalSeconds))
}
assembler := initializePassiveTapper(opts, outputItems, streamsMap)
assembler, err := initializePassiveTapper(opts, outputItems, streamsMap)
if err != nil {
logger.Log.Errorf("Error initializing tapper %w", err)
return
}
go startPassiveTapper(streamsMap, assembler)
}
@@ -100,7 +109,7 @@ func UpdateTapTargets(newTapTargets []v1.Pod) {
packetSourceManager.UpdatePods(tapTargets, !*nodefrag, mainPacketInputChan)
if tlsTapperInstance != nil {
if tlsTapperInstance != nil && os.Getenv("MIZU_GLOBAL_GOLANG_PID") == "" {
if err := tlstapper.UpdateTapTargets(tlsTapperInstance, &tapTargets, *procfs); err != nil {
tlstapper.LogError(err)
success = false
@@ -124,7 +133,7 @@ func printNewTapTargets(success bool) {
}
}
func printPeriodicStats(cleaner *Cleaner) {
func printPeriodicStats(cleaner *Cleaner, assembler *tcpAssembler) {
statsPeriod := time.Second * time.Duration(*statsevery)
ticker := time.NewTicker(statsPeriod)
@@ -162,8 +171,10 @@ func printPeriodicStats(cleaner *Cleaner) {
}
}
logger.Log.Infof(
"mem: %d, goroutines: %d, cpu: %f, cores: %d/%d, rss: %f",
"heap-alloc: %d, heap-idle: %d, heap-objects: %d, goroutines: %d, cpu: %f, cores: %d/%d, rss: %f",
memStats.HeapAlloc,
memStats.HeapIdle,
memStats.HeapObjects,
runtime.NumGoroutine(),
sysInfo.CPU,
logicalCoreCount,
@@ -172,15 +183,19 @@ func printPeriodicStats(cleaner *Cleaner) {
// Since the last print
cleanStats := cleaner.dumpStats()
assemblerStats := assembler.DumpStats()
logger.Log.Infof(
"cleaner - flushed connections: %d, closed connections: %d, deleted messages: %d",
cleanStats.flushed,
cleanStats.closed,
assemblerStats.flushedConnections,
assemblerStats.closedConnections,
cleanStats.deleted,
)
currentAppStats := diagnose.AppStats.DumpStats()
appStatsJSON, _ := json.Marshal(currentAppStats)
logger.Log.Infof("app stats - %v", string(appStatsJSON))
// At the moment
logger.Log.Infof("assembler-stats: %s, packet-source-stats: %s", assembler.Dump(), packetSourceManager.Stats())
}
}
@@ -208,7 +223,7 @@ func initializePacketSources() error {
return err
}
func initializePassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem, streamsMap api.TcpStreamMap) *tcpAssembler {
func initializePassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem, streamsMap api.TcpStreamMap) (*tcpAssembler, error) {
diagnose.InitializeErrorsMap(*debug, *verbose, *quiet)
diagnose.InitializeTapperInternalStats()
@@ -219,10 +234,10 @@ func initializePassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelI
}
opts.IgnoredPorts = append(opts.IgnoredPorts, buildIgnoredPortsList(*ignoredPorts)...)
opts.maxLiveStreams = *maxLiveStreams
opts.staleConnectionTimeout = time.Duration(*staleTimeoutSeconds) * time.Second
assembler := NewTcpAssembler(outputItems, streamsMap, opts)
return assembler
return NewTcpAssembler(outputItems, streamsMap, opts)
}
func startPassiveTapper(streamsMap api.TcpStreamMap, assembler *tcpAssembler) {
@@ -233,14 +248,13 @@ func startPassiveTapper(streamsMap api.TcpStreamMap, assembler *tcpAssembler) {
staleConnectionTimeout := time.Second * time.Duration(*staleTimeoutSeconds)
cleaner := Cleaner{
assembler: assembler.Assembler,
assemblerMutex: &assembler.assemblerMutex,
cleanPeriod: cleanPeriod,
connectionTimeout: staleConnectionTimeout,
streamsMap: streamsMap,
}
cleaner.start()
go printPeriodicStats(&cleaner)
go printPeriodicStats(&cleaner, assembler)
assembler.processPackets(*hexdumppkt, mainPacketInputChan)
@@ -278,7 +292,16 @@ func startTlsTapper(extension *api.Extension, outputItems chan *api.OutputChanne
// A quick way to instrument libssl.so without PID filtering - used for debuging and troubleshooting
//
if os.Getenv("MIZU_GLOBAL_SSL_LIBRARY") != "" {
if err := tls.GlobalTap(os.Getenv("MIZU_GLOBAL_SSL_LIBRARY")); err != nil {
if err := tls.GlobalSSLLibTap(os.Getenv("MIZU_GLOBAL_SSL_LIBRARY")); err != nil {
tlstapper.LogError(err)
return nil
}
}
// A quick way to instrument Go `crypto/tls` without PID filtering - used for debuging and troubleshooting
//
if os.Getenv("MIZU_GLOBAL_GOLANG_PID") != "" {
if err := tls.GlobalGoTap(*procfs, os.Getenv("MIZU_GLOBAL_GOLANG_PID")); err != nil {
tlstapper.LogError(err)
return nil
}

View File

@@ -160,3 +160,19 @@ func (m *PacketSourceManager) Close() {
src.close()
}
}
func (m *PacketSourceManager) Stats() string {
result := ""
for _, source := range m.sources {
stats, err := source.Stats()
if err != nil {
result = result + fmt.Sprintf("[%s: err:%s]", source.String(), err)
} else {
result = result + fmt.Sprintf("[%s: rec: %d dropped: %d]", source.String(), stats.PacketsReceived, stats.PacketsDropped)
}
}
return result
}

View File

@@ -116,6 +116,10 @@ func (source *tcpPacketSource) close() {
}
}
func (source *tcpPacketSource) Stats() (stat *pcap.Stats, err error) {
return source.handle.Stats()
}
func (source *tcpPacketSource) readPackets(ipdefrag bool, packets chan<- TcpPacketInfo) {
if dbgctl.MizuTapperDisablePcap {
return

View File

@@ -2,14 +2,15 @@ package tap
import (
"encoding/hex"
"fmt"
"os"
"os/signal"
"sync"
"time"
"github.com/google/gopacket"
"github.com/google/gopacket/layers"
"github.com/google/gopacket/reassembly"
"github.com/hashicorp/golang-lru/simplelru"
"github.com/up9inc/mizu/logger"
"github.com/up9inc/mizu/tap/api"
"github.com/up9inc/mizu/tap/dbgctl"
@@ -17,14 +18,33 @@ import (
"github.com/up9inc/mizu/tap/source"
)
const PACKETS_SEEN_LOG_THRESHOLD = 1000
const (
lastClosedConnectionsMaxItems = 1000
packetsSeenLogThreshold = 1000
lastAckThreshold = time.Duration(3) * time.Second
)
type connectionId string
func NewConnectionId(c string) connectionId {
return connectionId(c)
}
type AssemblerStats struct {
flushedConnections int
closedConnections int
}
type tcpAssembler struct {
*reassembly.Assembler
streamPool *reassembly.StreamPool
streamFactory *tcpStreamFactory
assemblerMutex sync.Mutex
ignoredPorts []uint16
streamPool *reassembly.StreamPool
streamFactory *tcpStreamFactory
ignoredPorts []uint16
lastClosedConnections *simplelru.LRU // Actual type is map[string]int64 which is "connId -> lastSeen"
liveConnections map[connectionId]bool
maxLiveStreams int
staleConnectionTimeout time.Duration
stats AssemblerStats
}
// Context
@@ -38,108 +58,166 @@ func (c *context) GetCaptureInfo() gopacket.CaptureInfo {
return c.CaptureInfo
}
func NewTcpAssembler(outputItems chan *api.OutputChannelItem, streamsMap api.TcpStreamMap, opts *TapOpts) *tcpAssembler {
func NewTcpAssembler(outputItems chan *api.OutputChannelItem, streamsMap api.TcpStreamMap, opts *TapOpts) (*tcpAssembler, error) {
var emitter api.Emitter = &api.Emitting{
AppStats: &diagnose.AppStats,
OutputChannel: outputItems,
}
streamFactory := NewTcpStreamFactory(emitter, streamsMap, opts)
streamPool := reassembly.NewStreamPool(streamFactory)
assembler := reassembly.NewAssembler(streamPool)
lastClosedConnections, err := simplelru.NewLRU(lastClosedConnectionsMaxItems, func(key interface{}, value interface{}) {})
if err != nil {
return nil, err
}
a := &tcpAssembler{
ignoredPorts: opts.IgnoredPorts,
lastClosedConnections: lastClosedConnections,
liveConnections: make(map[connectionId]bool),
maxLiveStreams: opts.maxLiveStreams,
staleConnectionTimeout: opts.staleConnectionTimeout,
stats: AssemblerStats{},
}
a.streamFactory = NewTcpStreamFactory(emitter, streamsMap, opts, a)
a.streamPool = reassembly.NewStreamPool(a.streamFactory)
a.Assembler = reassembly.NewAssembler(a.streamPool)
maxBufferedPagesTotal := GetMaxBufferedPagesPerConnection()
maxBufferedPagesPerConnection := GetMaxBufferedPagesTotal()
logger.Log.Infof("Assembler options: maxBufferedPagesTotal=%d, maxBufferedPagesPerConnection=%d, opts=%v",
maxBufferedPagesTotal, maxBufferedPagesPerConnection, opts)
assembler.AssemblerOptions.MaxBufferedPagesTotal = maxBufferedPagesTotal
assembler.AssemblerOptions.MaxBufferedPagesPerConnection = maxBufferedPagesPerConnection
a.Assembler.AssemblerOptions.MaxBufferedPagesTotal = maxBufferedPagesTotal
a.Assembler.AssemblerOptions.MaxBufferedPagesPerConnection = maxBufferedPagesPerConnection
return &tcpAssembler{
Assembler: assembler,
streamPool: streamPool,
streamFactory: streamFactory,
ignoredPorts: opts.IgnoredPorts,
}
return a, nil
}
func (a *tcpAssembler) processPackets(dumpPacket bool, packets <-chan source.TcpPacketInfo) {
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, os.Interrupt)
ticker := time.NewTicker(a.staleConnectionTimeout)
for packetInfo := range packets {
packetsCount := diagnose.AppStats.IncPacketsCount()
if packetsCount%PACKETS_SEEN_LOG_THRESHOLD == 0 {
logger.Log.Debugf("Packets seen: #%d", packetsCount)
}
packet := packetInfo.Packet
data := packet.Data()
diagnose.AppStats.UpdateProcessedBytes(uint64(len(data)))
if dumpPacket {
logger.Log.Debugf("Packet content (%d/0x%x) - %s", len(data), len(data), hex.Dump(data))
}
tcp := packet.Layer(layers.LayerTypeTCP)
if tcp != nil {
diagnose.AppStats.IncTcpPacketsCount()
tcp := tcp.(*layers.TCP)
if a.shouldIgnorePort(uint16(tcp.DstPort)) {
diagnose.AppStats.IncIgnoredPacketsCount()
} else {
c := context{
CaptureInfo: packet.Metadata().CaptureInfo,
Origin: packetInfo.Source.Origin,
}
diagnose.InternalStats.Totalsz += len(tcp.Payload)
if !dbgctl.MizuTapperDisableTcpReassembly {
a.assemblerMutex.Lock()
a.AssembleWithContext(packet.NetworkLayer().NetworkFlow(), tcp, &c)
a.assemblerMutex.Unlock()
}
}
}
done := *maxcount > 0 && int64(diagnose.AppStats.PacketsCount) >= *maxcount
if done {
errorMapLen, _ := diagnose.TapErrors.GetErrorsSummary()
logger.Log.Infof("Processed %v packets (%v bytes) in %v (errors: %v, errTypes:%v)",
diagnose.AppStats.PacketsCount,
diagnose.AppStats.ProcessedBytes,
time.Since(diagnose.AppStats.StartTime),
diagnose.TapErrors.ErrorsCount,
errorMapLen)
}
out:
for {
select {
case packetInfo, ok := <-packets:
if !ok {
break out
}
if a.processPacket(packetInfo, dumpPacket) {
break out
}
case <-signalChan:
logger.Log.Infof("Caught SIGINT: aborting")
done = true
default:
// NOP: continue
}
if done {
break
break out
case <-ticker.C:
a.periodicClean()
}
}
a.assemblerMutex.Lock()
closed := a.FlushAll()
a.assemblerMutex.Unlock()
logger.Log.Debugf("Final flush: %d closed", closed)
}
func (a *tcpAssembler) processPacket(packetInfo source.TcpPacketInfo, dumpPacket bool) bool {
packetsCount := diagnose.AppStats.IncPacketsCount()
if packetsCount%packetsSeenLogThreshold == 0 {
logger.Log.Debugf("Packets seen: #%d", packetsCount)
}
packet := packetInfo.Packet
data := packet.Data()
diagnose.AppStats.UpdateProcessedBytes(uint64(len(data)))
if dumpPacket {
logger.Log.Debugf("Packet content (%d/0x%x) - %s", len(data), len(data), hex.Dump(data))
}
tcp := packet.Layer(layers.LayerTypeTCP)
if tcp != nil {
a.processTcpPacket(packetInfo.Source.Origin, packet, tcp.(*layers.TCP))
}
done := *maxcount > 0 && int64(diagnose.AppStats.PacketsCount) >= *maxcount
if done {
errorMapLen, _ := diagnose.TapErrors.GetErrorsSummary()
logger.Log.Infof("Processed %v packets (%v bytes) in %v (errors: %v, errTypes:%v)",
diagnose.AppStats.PacketsCount,
diagnose.AppStats.ProcessedBytes,
time.Since(diagnose.AppStats.StartTime),
diagnose.TapErrors.ErrorsCount,
errorMapLen)
}
return done
}
func (a *tcpAssembler) processTcpPacket(origin api.Capture, packet gopacket.Packet, tcp *layers.TCP) {
diagnose.AppStats.IncTcpPacketsCount()
if a.shouldIgnorePort(uint16(tcp.DstPort)) || a.shouldIgnorePort(uint16(tcp.SrcPort)) {
diagnose.AppStats.IncIgnoredPacketsCount()
return
}
id := getConnectionId(packet.NetworkLayer().NetworkFlow().Src().String(),
packet.TransportLayer().TransportFlow().Src().String(),
packet.NetworkLayer().NetworkFlow().Dst().String(),
packet.TransportLayer().TransportFlow().Dst().String())
if a.isRecentlyClosed(id) {
diagnose.AppStats.IncIgnoredLastAckCount()
return
}
if a.shouldThrottle(id) {
diagnose.AppStats.IncThrottledPackets()
return
}
c := context{
CaptureInfo: packet.Metadata().CaptureInfo,
Origin: origin,
}
diagnose.InternalStats.Totalsz += len(tcp.Payload)
if !dbgctl.MizuTapperDisableTcpReassembly {
a.AssembleWithContext(packet.NetworkLayer().NetworkFlow(), tcp, &c)
}
}
func (a *tcpAssembler) tcpStreamCreated(stream *tcpStream) {
a.liveConnections[stream.connectionId] = true
}
func (a *tcpAssembler) tcpStreamClosed(stream *tcpStream) {
a.lastClosedConnections.Add(stream.connectionId, time.Now().UnixMilli())
delete(a.liveConnections, stream.connectionId)
}
func (a *tcpAssembler) isRecentlyClosed(c connectionId) bool {
if closedTimeMillis, ok := a.lastClosedConnections.Get(c); ok {
timeSinceClosed := time.Since(time.UnixMilli(closedTimeMillis.(int64)))
if timeSinceClosed < lastAckThreshold {
return true
}
}
return false
}
func (a *tcpAssembler) shouldThrottle(c connectionId) bool {
if _, ok := a.liveConnections[c]; ok {
return false
}
return len(a.liveConnections) > a.maxLiveStreams
}
func (a *tcpAssembler) dumpStreamPool() {
a.streamPool.Dump()
}
func (a *tcpAssembler) waitAndDump() {
a.streamFactory.WaitGoRoutines()
a.assemblerMutex.Lock()
logger.Log.Debugf("%s", a.Dump())
a.assemblerMutex.Unlock()
}
func (a *tcpAssembler) shouldIgnorePort(port uint16) bool {
@@ -151,3 +229,26 @@ func (a *tcpAssembler) shouldIgnorePort(port uint16) bool {
return false
}
func (a *tcpAssembler) periodicClean() {
flushed, closed := a.FlushCloseOlderThan(time.Now().Add(-a.staleConnectionTimeout))
stats := a.stats
stats.closedConnections += closed
stats.flushedConnections += flushed
}
func (a *tcpAssembler) DumpStats() AssemblerStats {
result := a.stats
a.stats = AssemblerStats{}
return result
}
func getConnectionId(saddr string, sport string, daddr string, dport string) connectionId {
s := fmt.Sprintf("%s:%s", saddr, sport)
d := fmt.Sprintf("%s:%s", daddr, dport)
if s > d {
return NewConnectionId(fmt.Sprintf("%s#%s", s, d))
} else {
return NewConnectionId(fmt.Sprintf("%s#%s", d, s))
}
}

View File

@@ -151,6 +151,6 @@ func (t *tcpReassemblyStream) ReassemblyComplete(ac reassembly.AssemblerContext)
if t.tcpStream.GetIsTapTarget() && !t.tcpStream.GetIsClosed() {
t.tcpStream.close()
}
// do not remove the connection to allow last ACK
return false
return true
}

View File

@@ -8,6 +8,11 @@ import (
"github.com/up9inc/mizu/tap/dbgctl"
)
type tcpStreamCallbacks interface {
tcpStreamCreated(stream *tcpStream)
tcpStreamClosed(stream *tcpStream)
}
/* It's a connection (bidirectional)
* Implements gopacket.reassembly.Stream interface (Accept, ReassembledSG, ReassemblyComplete)
* ReassembledSG gets called when new reassembled data is ready (i.e. bytes in order, no duplicates, complete)
@@ -25,16 +30,25 @@ type tcpStream struct {
reqResMatchers []api.RequestResponseMatcher
createdAt time.Time
streamsMap api.TcpStreamMap
connectionId connectionId
callbacks tcpStreamCallbacks
sync.Mutex
}
func NewTcpStream(isTapTarget bool, streamsMap api.TcpStreamMap, capture api.Capture) *tcpStream {
return &tcpStream{
isTapTarget: isTapTarget,
streamsMap: streamsMap,
origin: capture,
createdAt: time.Now(),
func NewTcpStream(isTapTarget bool, streamsMap api.TcpStreamMap, capture api.Capture,
connectionId connectionId, callbacks tcpStreamCallbacks) *tcpStream {
t := &tcpStream{
isTapTarget: isTapTarget,
streamsMap: streamsMap,
origin: capture,
createdAt: time.Now(),
connectionId: connectionId,
callbacks: callbacks,
}
t.callbacks.tcpStreamCreated(t)
return t
}
func (t *tcpStream) getId() int64 {
@@ -56,9 +70,9 @@ func (t *tcpStream) close() {
t.isClosed = true
t.streamsMap.Delete(t.id)
t.client.close()
t.server.close()
t.callbacks.tcpStreamClosed(t)
}
func (t *tcpStream) addCounterPair(counterPair *api.CounterPair) {

View File

@@ -19,14 +19,15 @@ import (
* Generates a new tcp stream for each new tcp connection. Closes the stream when the connection closes.
*/
type tcpStreamFactory struct {
wg sync.WaitGroup
emitter api.Emitter
streamsMap api.TcpStreamMap
ownIps []string
opts *TapOpts
wg sync.WaitGroup
emitter api.Emitter
streamsMap api.TcpStreamMap
ownIps []string
opts *TapOpts
streamsCallbacks tcpStreamCallbacks
}
func NewTcpStreamFactory(emitter api.Emitter, streamsMap api.TcpStreamMap, opts *TapOpts) *tcpStreamFactory {
func NewTcpStreamFactory(emitter api.Emitter, streamsMap api.TcpStreamMap, opts *TapOpts, streamsCallbacks tcpStreamCallbacks) *tcpStreamFactory {
var ownIps []string
if localhostIPs, err := getLocalhostIPs(); err != nil {
@@ -39,10 +40,11 @@ func NewTcpStreamFactory(emitter api.Emitter, streamsMap api.TcpStreamMap, opts
}
return &tcpStreamFactory{
emitter: emitter,
streamsMap: streamsMap,
ownIps: ownIps,
opts: opts,
emitter: emitter,
streamsMap: streamsMap,
ownIps: ownIps,
opts: opts,
streamsCallbacks: streamsCallbacks,
}
}
@@ -57,7 +59,8 @@ func (factory *tcpStreamFactory) New(net, transport gopacket.Flow, tcpLayer *lay
props := factory.getStreamProps(srcIp, srcPort, dstIp, dstPort)
isTapTarget := props.isTapTarget
stream := NewTcpStream(isTapTarget, factory.streamsMap, getPacketOrigin(ac))
connectionId := getConnectionId(srcIp, srcPort, dstIp, dstPort)
stream := NewTcpStream(isTapTarget, factory.streamsMap, getPacketOrigin(ac), connectionId, factory.streamsCallbacks)
reassemblyStream := NewTcpReassemblyStream(fmt.Sprintf("%s:%s", net, transport), tcpLayer, fsmOptions, stream)
if stream.GetIsTapTarget() {
stream.setId(factory.streamsMap.NextId())

View File

@@ -1,5 +1,5 @@
FROM alpine:3.14
FROM golang:1.17-alpine
RUN apk --no-cache update && apk --no-cache add clang llvm libbpf-dev go linux-headers
RUN apk --no-cache update && apk --no-cache add clang llvm libbpf-dev linux-headers
WORKDIR /mizu

View File

@@ -6,17 +6,22 @@ MIZU_HOME=$(realpath ../../../)
docker build -t mizu-ebpf-builder . || exit 1
BPF_TARGET=amd64
BPF_CFLAGS="-O2 -g -D__TARGET_ARCH_x86"
ARCH=$(uname -m)
if [[ $ARCH == "aarch64" ]]; then
BPF_TARGET=arm64
BPF_CFLAGS="-O2 -g -D__TARGET_ARCH_arm64"
fi
docker run --rm \
--name mizu-ebpf-builder \
-v $MIZU_HOME:/mizu \
-v $(go env GOPATH):/root/go \
-it mizu-ebpf-builder \
sh -c "
go generate tap/tlstapper/tls_tapper.go
chown $(id -u):$(id -g) tap/tlstapper/tlstapper_bpfeb.go
chown $(id -u):$(id -g) tap/tlstapper/tlstapper_bpfeb.o
chown $(id -u):$(id -g) tap/tlstapper/tlstapper_bpfel.go
chown $(id -u):$(id -g) tap/tlstapper/tlstapper_bpfel.o
BPF_TARGET=\"$BPF_TARGET\" BPF_CFLAGS=\"$BPF_CFLAGS\" go generate tap/tlstapper/tls_tapper.go
chown $(id -u):$(id -g) tap/tlstapper/tlstapper_bpf*
" || exit 1
popd

141
tap/tlstapper/bpf/common.c Normal file
View File

@@ -0,0 +1,141 @@
/*
Note: This file is licenced differently from the rest of the project
SPDX-License-Identifier: GPL-2.0
Copyright (C) UP9 Inc.
*/
#include "include/headers.h"
#include "include/util.h"
#include "include/maps.h"
#include "include/log.h"
#include "include/logger_messages.h"
#include "include/common.h"
static __always_inline int add_address_to_chunk(struct pt_regs *ctx, struct tls_chunk* chunk, __u64 id, __u32 fd) {
__u32 pid = id >> 32;
__u64 key = (__u64) pid << 32 | fd;
struct fd_info *fdinfo = bpf_map_lookup_elem(&file_descriptor_to_ipv4, &key);
if (fdinfo == NULL) {
return 0;
}
int err = bpf_probe_read(chunk->address, sizeof(chunk->address), fdinfo->ipv4_addr);
chunk->flags |= (fdinfo->flags & FLAGS_IS_CLIENT_BIT);
if (err != 0) {
log_error(ctx, LOG_ERROR_READING_FD_ADDRESS, id, err, 0l);
return 0;
}
return 1;
}
static __always_inline void send_chunk_part(struct pt_regs *ctx, __u8* buffer, __u64 id,
struct tls_chunk* chunk, int start, int end) {
size_t recorded = MIN(end - start, sizeof(chunk->data));
if (recorded <= 0) {
return;
}
chunk->recorded = recorded;
chunk->start = start;
// This ugly trick is for the ebpf verifier happiness
//
long err = 0;
if (chunk->recorded == sizeof(chunk->data)) {
err = bpf_probe_read(chunk->data, sizeof(chunk->data), buffer + start);
} else {
recorded &= (sizeof(chunk->data) - 1); // Buffer must be N^2
err = bpf_probe_read(chunk->data, recorded, buffer + start);
}
if (err != 0) {
log_error(ctx, LOG_ERROR_READING_FROM_SSL_BUFFER, id, err, 0l);
return;
}
bpf_perf_event_output(ctx, &chunks_buffer, BPF_F_CURRENT_CPU, chunk, sizeof(struct tls_chunk));
}
static __always_inline void send_chunk(struct pt_regs *ctx, __u8* buffer, __u64 id, struct tls_chunk* chunk) {
// ebpf loops must be bounded at compile time, we can't use (i < chunk->len / CHUNK_SIZE)
//
// https://lwn.net/Articles/794934/
//
// However we want to run in kernel older than 5.3, hence we use "#pragma unroll" anyway
//
#pragma unroll
for (int i = 0; i < MAX_CHUNKS_PER_OPERATION; i++) {
if (chunk->len <= (CHUNK_SIZE * i)) {
break;
}
send_chunk_part(ctx, buffer, id, chunk, CHUNK_SIZE * i, chunk->len);
}
}
static __always_inline void output_ssl_chunk(struct pt_regs *ctx, struct ssl_info* info, int count_bytes, __u64 id, __u32 flags) {
if (count_bytes > (CHUNK_SIZE * MAX_CHUNKS_PER_OPERATION)) {
log_error(ctx, LOG_ERROR_BUFFER_TOO_BIG, id, count_bytes, 0l);
return;
}
struct tls_chunk* chunk;
int zero = 0;
// If other thread, running on the same CPU get to this point at the same time like us (context switch)
// the data will be corrupted - protection may be added in the future
//
chunk = bpf_map_lookup_elem(&heap, &zero);
if (!chunk) {
log_error(ctx, LOG_ERROR_ALLOCATING_CHUNK, id, 0l, 0l);
return;
}
chunk->flags = flags;
chunk->pid = id >> 32;
chunk->tgid = id;
chunk->len = count_bytes;
chunk->fd = info->fd;
if (!add_address_to_chunk(ctx, chunk, id, chunk->fd)) {
// Without an address, we drop the chunk because there is not much to do with it in Go
//
return;
}
send_chunk(ctx, info->buffer, id, chunk);
}
static __always_inline struct ssl_info new_ssl_info() {
struct ssl_info info = { .fd = invalid_fd, .created_at_nano = bpf_ktime_get_ns() };
return info;
}
static __always_inline struct ssl_info lookup_ssl_info(struct pt_regs *ctx, struct bpf_map_def* map_fd, __u64 pid_tgid) {
struct ssl_info *infoPtr = bpf_map_lookup_elem(map_fd, &pid_tgid);
struct ssl_info info = new_ssl_info();
if (infoPtr != NULL) {
long err = bpf_probe_read(&info, sizeof(struct ssl_info), infoPtr);
if (err != 0) {
log_error(ctx, LOG_ERROR_READING_SSL_CONTEXT, pid_tgid, err, ORIGIN_SSL_UPROBE_CODE);
}
if ((bpf_ktime_get_ns() - info.created_at_nano) > SSL_INFO_MAX_TTL_NANO) {
// If the ssl info is too old, we don't want to use its info because it may be incorrect.
//
info.fd = invalid_fd;
info.created_at_nano = bpf_ktime_get_ns();
}
}
return info;
}

View File

@@ -28,7 +28,7 @@ void sys_enter_read(struct sys_enter_read_ctx *ctx) {
return;
}
struct ssl_info *infoPtr = bpf_map_lookup_elem(&ssl_read_context, &id);
struct ssl_info *infoPtr = bpf_map_lookup_elem(&openssl_read_context, &id);
if (infoPtr == NULL) {
return;
@@ -44,7 +44,7 @@ void sys_enter_read(struct sys_enter_read_ctx *ctx) {
info.fd = ctx->fd;
err = bpf_map_update_elem(&ssl_read_context, &id, &info, BPF_ANY);
err = bpf_map_update_elem(&openssl_read_context, &id, &info, BPF_ANY);
if (err != 0) {
log_error(ctx, LOG_ERROR_PUTTING_FILE_DESCRIPTOR, id, err, ORIGIN_SYS_ENTER_READ_CODE);
@@ -68,7 +68,7 @@ void sys_enter_write(struct sys_enter_write_ctx *ctx) {
return;
}
struct ssl_info *infoPtr = bpf_map_lookup_elem(&ssl_write_context, &id);
struct ssl_info *infoPtr = bpf_map_lookup_elem(&openssl_write_context, &id);
if (infoPtr == NULL) {
return;
@@ -84,7 +84,7 @@ void sys_enter_write(struct sys_enter_write_ctx *ctx) {
info.fd = ctx->fd;
err = bpf_map_update_elem(&ssl_write_context, &id, &info, BPF_ANY);
err = bpf_map_update_elem(&openssl_write_context, &id, &info, BPF_ANY);
if (err != 0) {
log_error(ctx, LOG_ERROR_PUTTING_FILE_DESCRIPTOR, id, err, ORIGIN_SYS_ENTER_WRITE_CODE);

View File

@@ -0,0 +1,191 @@
/*
Note: This file is licenced differently from the rest of the project
SPDX-License-Identifier: GPL-2.0
Copyright (C) UP9 Inc.
---
README
Go does not follow any platform ABI like x86-64 System V ABI.
Before 1.17, Go followed stack-based Plan9 (Bell Labs) calling convention. (ABI0)
After 1.17, Go switched to an internal register-based calling convention. (ABIInternal)
For now, the probes in this file supports only ABIInternal (Go 1.17+)
`uretprobe` in Linux kernel uses trampoline pattern to jump to original return
address of the probed function. A Goroutine's stack size is 2Kb while a C thread is 2MB on Linux.
If stack size exceeds 2Kb, Go runtime relocates the stack. That causes the
return address to become incorrect in case of `uretprobe` and probed Go program crashes.
Therefore `uretprobe` CAN'T BE USED for a Go program.
`_ex_uprobe` suffixed probes suppose to be `uretprobe`(s) are actually `uprobe`(s)
because of the non-standard ABI of Go. Therefore we probe all `ret` mnemonics under the symbol
by automatically finding them through reading the ELF binary and disassembling the symbols.
Disassembly related code located in `go_offsets.go` file and it uses Capstone Engine.
Solution based on: https://github.com/iovisor/bcc/issues/1320#issuecomment-407927542
*Example* We probe an arbitrary point in a function body (offset +559):
https://github.com/golang/go/blob/go1.17.6/src/crypto/tls/conn.go#L1299
We get the file descriptor using the common $rax register that holds the address
of `go.itab.*net.TCPConn,net.Conn` and through a series of dereferencing
using `bpf_probe_read` calls in `go_crypto_tls_get_fd_from_tcp_conn` function.
---
SOURCES:
Tracing Go Functions with eBPF (before 1.17): https://www.grant.pizza/blog/tracing-go-functions-with-ebpf-part-2/
Challenges of BPF Tracing Go: https://blog.0x74696d.com/posts/challenges-of-bpf-tracing-go/
x86 calling conventions: https://en.wikipedia.org/wiki/X86_calling_conventions
Plan 9 from Bell Labs: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
The issue for calling convention change in Go: https://github.com/golang/go/issues/40724
Proposal of Register-based Go calling convention: https://go.googlesource.com/proposal/+/master/design/40724-register-calling.md
Go internal ABI (1.17) specification: https://go.googlesource.com/go/+/refs/heads/dev.regabi/src/cmd/compile/internal-abi.md
Go internal ABI (current) specification: https://go.googlesource.com/go/+/refs/heads/master/src/cmd/compile/abi-internal.md
A Quick Guide to Go's Assembler: https://go.googlesource.com/go/+/refs/heads/dev.regabi/doc/asm.html
Dissecting Go Binaries: https://www.grant.pizza/blog/dissecting-go-binaries/
Capstone Engine: https://www.capstone-engine.org/
*/
#include "include/headers.h"
#include "include/util.h"
#include "include/maps.h"
#include "include/log.h"
#include "include/logger_messages.h"
#include "include/pids.h"
#include "include/common.h"
#include "include/go_abi_internal.h"
#include "include/go_types.h"
static __always_inline __u32 go_crypto_tls_get_fd_from_tcp_conn(struct pt_regs *ctx) {
struct go_interface conn;
long err;
__u64 addr;
#if defined(bpf_target_arm64)
err = bpf_probe_read(&addr, sizeof(addr), (void*)GO_ABI_INTERNAL_PT_REGS_SP(ctx)+0x8);
if (err != 0) {
return invalid_fd;
}
#else
addr = GO_ABI_INTERNAL_PT_REGS_R1(ctx);
#endif
err = bpf_probe_read(&conn, sizeof(conn), (void*)addr);
if (err != 0) {
return invalid_fd;
}
void* net_fd_ptr;
err = bpf_probe_read(&net_fd_ptr, sizeof(net_fd_ptr), conn.ptr);
if (err != 0) {
return invalid_fd;
}
__u32 fd;
err = bpf_probe_read(&fd, sizeof(fd), net_fd_ptr + 0x10);
if (err != 0) {
return invalid_fd;
}
return fd;
}
static __always_inline void go_crypto_tls_uprobe(struct pt_regs *ctx, struct bpf_map_def* go_context) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u64 pid = pid_tgid >> 32;
if (!should_tap(pid)) {
return;
}
struct ssl_info info = new_ssl_info();
long err;
#if defined(bpf_target_arm64)
err = bpf_probe_read(&info.buffer_len, sizeof(__u32), (void*)GO_ABI_INTERNAL_PT_REGS_SP(ctx)+0x18);
if (err != 0) {
log_error(ctx, LOG_ERROR_READING_BYTES_COUNT, pid_tgid, err, ORIGIN_SSL_UPROBE_CODE);
return;
}
#else
info.buffer_len = GO_ABI_INTERNAL_PT_REGS_R2(ctx);
#endif
info.buffer = (void*)GO_ABI_INTERNAL_PT_REGS_R4(ctx);
info.fd = go_crypto_tls_get_fd_from_tcp_conn(ctx);
// GO_ABI_INTERNAL_PT_REGS_GP is Goroutine address
__u64 pid_fp = pid << 32 | GO_ABI_INTERNAL_PT_REGS_GP(ctx);
err = bpf_map_update_elem(go_context, &pid_fp, &info, BPF_ANY);
if (err != 0) {
log_error(ctx, LOG_ERROR_PUTTING_SSL_CONTEXT, pid_tgid, err, 0l);
}
return;
}
static __always_inline void go_crypto_tls_ex_uprobe(struct pt_regs *ctx, struct bpf_map_def* go_context, __u32 flags) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u64 pid = pid_tgid >> 32;
if (!should_tap(pid)) {
return;
}
// GO_ABI_INTERNAL_PT_REGS_GP is Goroutine address
__u64 pid_fp = pid << 32 | GO_ABI_INTERNAL_PT_REGS_GP(ctx);
struct ssl_info *info_ptr = bpf_map_lookup_elem(go_context, &pid_fp);
if (info_ptr == NULL) {
return;
}
bpf_map_delete_elem(go_context, &pid_fp);
struct ssl_info info;
long err = bpf_probe_read(&info, sizeof(struct ssl_info), info_ptr);
if (err != 0) {
log_error(ctx, LOG_ERROR_READING_SSL_CONTEXT, pid_tgid, err, ORIGIN_SSL_URETPROBE_CODE);
return;
}
// In case of read, the length is determined on return
if (flags == FLAGS_IS_READ_BIT) {
#if defined(bpf_target_arm64)
// On ARM64 we look at a general-purpose register as an indicator of error return
if (GO_ABI_INTERNAL_PT_REGS_R6(ctx) == 0x10) {
return;
}
info.buffer_len = GO_ABI_INTERNAL_PT_REGS_R7(ctx); // n in return n, nil
#else
info.buffer_len = GO_ABI_INTERNAL_PT_REGS_R1(ctx); // n in return n, nil
#endif
// This check achieves ignoring 0 length reads (the reads result with an error)
if (info.buffer_len <= 0) {
return;
}
}
output_ssl_chunk(ctx, &info, info.buffer_len, pid_tgid, flags);
return;
}
SEC("uprobe/go_crypto_tls_write")
void BPF_KPROBE(go_crypto_tls_write) {
go_crypto_tls_uprobe(ctx, &go_write_context);
}
SEC("uprobe/go_crypto_tls_write_ex")
void BPF_KPROBE(go_crypto_tls_write_ex) {
go_crypto_tls_ex_uprobe(ctx, &go_write_context, 0);
}
SEC("uprobe/go_crypto_tls_read")
void BPF_KPROBE(go_crypto_tls_read) {
go_crypto_tls_uprobe(ctx, &go_read_context);
}
SEC("uprobe/go_crypto_tls_read_ex")
void BPF_KPROBE(go_crypto_tls_read_ex) {
go_crypto_tls_ex_uprobe(ctx, &go_read_context, FLAGS_IS_READ_BIT);
}

Some files were not shown because too many files have changed in this diff Show More