Compare commits

...

18 Commits

Author SHA1 Message Date
RoyUP9
47f2e69b7e fixed entries port when fetching data from db (#376) 2021-10-19 17:37:57 +03:00
M. Mert Yıldıran
6240d85377 Bring back the lines that didn't meant to be removed (#375) 2021-10-19 16:22:47 +03:00
M. Mert Yıldıran
29ba963c48 Remove github.com/romana/rlog dependency completely (#374)
* Remove `github.com/romana/rlog` dependency completely

* Comment out all the unnecessary logging in the protocol extensions

* Remove commented out all the unnecessary logging lines

* Remove two more lines related to logging
2021-10-19 16:13:03 +03:00
RamiBerm
0473181f0a TRA-3803 handle k8s watch timeouts (#372)
* Update watch.go and debounce.go

* Update debounce.go

* Update watch.go

* Update watch.go

* Update watch.go

* Update watch.go

* Update watch.go

Co-authored-by: Rami <rami@rami-work>
2021-10-19 14:41:37 +03:00
M. Mert Yıldıran
145e7cda01 Add OAS contract monitoring support (#325)
* Add OAS contract monitoring support

* Pass the contract failure reason to UI

* Fix the issues related to contract validation

* Fix rest of the issues in the UI

* Add documentation related to contract monitoring feature

* Fix a typo in the docs

* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled

* Fix an issue caused by the merge commit

* Slightly change the logic in the `validateOAS` method

Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.

* Retrieve and display the failure reason for both request and response

Also display the content of the contract/OAS file in the UI.

* Display the OAS under `CONTRACT` tab with syntax highlighting

Also fix the styling in the entry feed.

* Remove `EnforcePolicyFileDeprecated` constant

* Log the other errors as well

* Get context from caller instead

* Define a type for the contract status and make its values enum-like

* Remove an unnecessary `if` statement

* Validate OAS in the CLI before passing it to Agent

* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`

* Fix an artifact from the merge conflict
2021-10-19 14:24:22 +03:00
M. Mert Yıldıran
b7ff076571 Set the default log level for Agent to INFO and raise it to DEBUG if dump-logs=true is provided (#373)
* Set the default log level for Agent to `INFO` and change it to `DEBUG` if `dump-logs=true` is provided

* Remove `Trace` method and replace its calls with `Debug`

* Export logging levels from `logger` by defining functions

* Revert "Export logging levels from `logger` by defining functions"

This reverts commit e554e40f4a.

* Run `go mod tidy` on agent

* Define a method named `determineLogLevel`
2021-10-19 14:22:20 +03:00
RoyUP9
3aafbd7e1c added upsert workspace before dumping traffic (#368) 2021-10-19 11:06:51 +03:00
M. Mert Yıldıran
58e9363fda Replace all rlog occurrences with the shared logger in tap (#369) 2021-10-18 16:35:42 +03:00
M. Mert Yıldıran
6a85ab53eb Fix the go.mod of acceptanceTests (#371)
* Fix the `go.mod` of `acceptanceTests`

* Enable acceptance tests

* Revert "Enable acceptance tests"

This reverts commit e21c527e69.
2021-10-18 16:35:10 +03:00
M. Mert Yıldıran
212e4687d8 Fix the dependency error in acceptance tests (#370) 2021-10-17 18:36:30 +03:00
M. Mert Yıldıran
167b17dfd2 Move the 8899 integer and string literals into a const named DefaultApiServerPort in shared (#367) 2021-10-17 15:28:33 +03:00
M. Mert Yıldıran
9d179c7227 Ignore an eslint error (#351)
* Ignore an eslint error

* Change the fix
2021-10-17 15:28:03 +03:00
M. Mert Yıldıran
147e812edb Replace all rlog occurrences with the shared logger (#350)
* Replace all `rlog` occurrences with the shared logger

* Use the same log format in `InitLoggerStderrOnly` as well

* Convert one more `log.Fatal` to `logger.Log.Errorf` as well in the `cli`

* Replace `log.` occurrences with `logger.Log.` in `agent`

* Fix `cannot use err (type error)`

* Change the logging level to `DEBUG`

* Replace an `Errorf` with `Fatal`

* Add informative message
2021-10-17 12:15:30 +03:00
M. Mert Yıldıran
91196bb306 Add readiness and liveness probes to API server (#365)
* Add readiness and liveness probes to API server

* Use `intstr.FromInt(8899)` instead
2021-10-17 11:40:18 +03:00
Igor Gov
26834a6e04 Fix documentation from "mizu-image" to "agent-image" (#363) 2021-10-15 14:28:00 +03:00
M. Mert Yıldıran
754f385865 Improve formatting in bug_report.md issue template (#352) 2021-10-15 14:14:51 +03:00
M. Mert Yıldıran
b30b62ef77 Move cli/logger to shared, and refactor all its usages in cli (#349)
* Move `cli/logger` to `shared`, and refactor all its usages in `cli`

* Remove indirect for `op/go-logging` in `shared`
2021-10-14 10:18:01 +03:00
RoyUP9
26788bb3a6 organize routes (#348) 2021-10-13 17:31:15 +03:00
89 changed files with 1202 additions and 658 deletions

View File

@@ -12,9 +12,9 @@ A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Run mizu <command> '...'
2. Click on '....'
3. Scroll down to '....'
1. Run `mizu <command> ...`
2. Click on '...'
3. Scroll down to '...'
4. See error
**Expected behavior**
@@ -22,17 +22,17 @@ A clear and concise description of what you expected to happen.
**Logs**
Upload logs:
1. Run the mizu command with `--set dump-logs=true` (e.g `mizu tap --set dump-logs=true`)
1. Run the mizu command with `--set dump-logs=true` (e.g `mizu tap --set dump-logs=true`)
2. Try to reproduce the issue
3. CNTRL+C on terminal tab which runs mizu
4. Upload the logs zip file from ~/.mizu/mizu_logs_**.zip
3. <kbd>CTRL</kbd>+<kbd>C</kbd> on terminal tab which runs `mizu`
4. Upload the logs zip file from `~/.mizu/mizu_logs_**.zip`
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome]
- OS: [e.g. macOS]
- Web Browser: [e.g. Google Chrome]
**Additional context**
Add any other context about the problem here.

View File

@@ -2,8 +2,10 @@ FROM node:14-slim AS site-build
WORKDIR /app/ui-build
COPY ui .
COPY ui/package.json .
COPY ui/package-lock.json .
RUN npm i
COPY ui .
RUN npm run build

View File

@@ -159,6 +159,13 @@ Such validation may test response for specific JSON fields, headers, etc.
Please see [TRAFFIC RULES](docs/POLICY_RULES.md) page for more details and syntax.
### OpenAPI Specification (OAS) Contract Monitoring
An OAS/Swagger file can contain schemas under `parameters` and `responses` fields. With `--contract catalogue.yaml`
CLI option, you can pass your API description to Mizu and the traffic will automatically be validated
against the contracts.
Please see [CONTRACT MONITORING](docs/CONTRACT_MONITORING.md) page for more details and syntax.
## How to Run local UI

View File

@@ -2,4 +2,9 @@ module github.com/up9inc/mizu/tests
go 1.16
require gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
require (
github.com/up9inc/mizu/shared v0.0.0
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
)
replace github.com/up9inc/mizu/shared v0.0.0 => ../shared

View File

@@ -1,3 +1,7 @@
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/golang-jwt/jwt/v4 v4.1.0 h1:XUgk2Ex5veyVFVeLm0xhusUTQybEbexJXrvPNOKkSY0=
github.com/golang-jwt/jwt/v4 v4.1.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=

View File

@@ -66,7 +66,7 @@ func TestTap(t *testing.T) {
entriesCheckFunc := func() error {
timestamp := time.Now().UnixNano() / int64(time.Millisecond)
entriesUrl := fmt.Sprintf("%v/api/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, entriesCount, timestamp)
entriesUrl := fmt.Sprintf("%v/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, entriesCount, timestamp)
requestResult, requestErr := executeHttpGetRequest(entriesUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entries, err: %v", requestErr)
@@ -79,7 +79,7 @@ func TestTap(t *testing.T) {
entry := entries[0].(map[string]interface{})
entryUrl := fmt.Sprintf("%v/api/entries/%v", apiServerUrl, entry["id"])
entryUrl := fmt.Sprintf("%v/entries/%v", apiServerUrl, entry["id"])
requestResult, requestErr = executeHttpGetRequest(entryUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entry, err: %v", requestErr)
@@ -188,7 +188,7 @@ func TestTapAllNamespaces(t *testing.T) {
return
}
podsUrl := fmt.Sprintf("%v/api/tapStatus", apiServerUrl)
podsUrl := fmt.Sprintf("%v/status/tap", apiServerUrl)
requestResult, requestErr := executeHttpGetRequest(podsUrl)
if requestErr != nil {
t.Errorf("failed to get tap status, err: %v", requestErr)
@@ -269,7 +269,7 @@ func TestTapMultipleNamespaces(t *testing.T) {
return
}
podsUrl := fmt.Sprintf("%v/api/tapStatus", apiServerUrl)
podsUrl := fmt.Sprintf("%v/status/tap", apiServerUrl)
requestResult, requestErr := executeHttpGetRequest(podsUrl)
if requestErr != nil {
t.Errorf("failed to get tap status, err: %v", requestErr)
@@ -352,7 +352,7 @@ func TestTapRegex(t *testing.T) {
return
}
podsUrl := fmt.Sprintf("%v/api/tapStatus", apiServerUrl)
podsUrl := fmt.Sprintf("%v/status/tap", apiServerUrl)
requestResult, requestErr := executeHttpGetRequest(podsUrl)
if requestErr != nil {
t.Errorf("failed to get tap status, err: %v", requestErr)
@@ -486,7 +486,7 @@ func TestTapRedact(t *testing.T) {
redactCheckFunc := func() error {
timestamp := time.Now().UnixNano() / int64(time.Millisecond)
entriesUrl := fmt.Sprintf("%v/api/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount, timestamp)
entriesUrl := fmt.Sprintf("%v/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount, timestamp)
requestResult, requestErr := executeHttpGetRequest(entriesUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entries, err: %v", requestErr)
@@ -499,7 +499,7 @@ func TestTapRedact(t *testing.T) {
firstEntry := entries[0].(map[string]interface{})
entryUrl := fmt.Sprintf("%v/api/entries/%v", apiServerUrl, firstEntry["id"])
entryUrl := fmt.Sprintf("%v/entries/%v", apiServerUrl, firstEntry["id"])
requestResult, requestErr = executeHttpGetRequest(entryUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entry, err: %v", requestErr)
@@ -601,7 +601,7 @@ func TestTapNoRedact(t *testing.T) {
redactCheckFunc := func() error {
timestamp := time.Now().UnixNano() / int64(time.Millisecond)
entriesUrl := fmt.Sprintf("%v/api/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount, timestamp)
entriesUrl := fmt.Sprintf("%v/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount, timestamp)
requestResult, requestErr := executeHttpGetRequest(entriesUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entries, err: %v", requestErr)
@@ -614,7 +614,7 @@ func TestTapNoRedact(t *testing.T) {
firstEntry := entries[0].(map[string]interface{})
entryUrl := fmt.Sprintf("%v/api/entries/%v", apiServerUrl, firstEntry["id"])
entryUrl := fmt.Sprintf("%v/entries/%v", apiServerUrl, firstEntry["id"])
requestResult, requestErr = executeHttpGetRequest(entryUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entry, err: %v", requestErr)
@@ -716,7 +716,7 @@ func TestTapRegexMasking(t *testing.T) {
redactCheckFunc := func() error {
timestamp := time.Now().UnixNano() / int64(time.Millisecond)
entriesUrl := fmt.Sprintf("%v/api/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount, timestamp)
entriesUrl := fmt.Sprintf("%v/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount, timestamp)
requestResult, requestErr := executeHttpGetRequest(entriesUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entries, err: %v", requestErr)
@@ -729,7 +729,7 @@ func TestTapRegexMasking(t *testing.T) {
firstEntry := entries[0].(map[string]interface{})
entryUrl := fmt.Sprintf("%v/api/entries/%v", apiServerUrl, firstEntry["id"])
entryUrl := fmt.Sprintf("%v/entries/%v", apiServerUrl, firstEntry["id"])
requestResult, requestErr = executeHttpGetRequest(entryUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entry, err: %v", requestErr)
@@ -823,7 +823,7 @@ func TestTapIgnoredUserAgents(t *testing.T) {
ignoredUserAgentsCheckFunc := func() error {
timestamp := time.Now().UnixNano() / int64(time.Millisecond)
entriesUrl := fmt.Sprintf("%v/api/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount * 2, timestamp)
entriesUrl := fmt.Sprintf("%v/entries?limit=%v&operator=lt&timestamp=%v", apiServerUrl, defaultEntriesCount * 2, timestamp)
requestResult, requestErr := executeHttpGetRequest(entriesUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entries, err: %v", requestErr)
@@ -835,7 +835,7 @@ func TestTapIgnoredUserAgents(t *testing.T) {
}
for _, entryInterface := range entries {
entryUrl := fmt.Sprintf("%v/api/entries/%v", apiServerUrl, entryInterface.(map[string]interface{})["id"])
entryUrl := fmt.Sprintf("%v/entries/%v", apiServerUrl, entryInterface.(map[string]interface{})["id"])
requestResult, requestErr = executeHttpGetRequest(entryUrl)
if requestErr != nil {
return fmt.Errorf("failed to get entry, err: %v", requestErr)

View File

@@ -12,12 +12,14 @@ import (
"strings"
"syscall"
"time"
"github.com/up9inc/mizu/shared"
)
const (
longRetriesCount = 100
shortRetriesCount = 10
defaultApiServerPort = 8899
defaultApiServerPort = shared.DefaultApiServerPort
defaultNamespaceName = "mizu-tests"
defaultServiceName = "httpbin"
defaultEntriesCount = 50

View File

@@ -12,7 +12,8 @@ Basic APIs:
`docker build . -t gcr.io/up9-docker-hub/mizu/debug:latest -f debug.Dockerfile && docker push gcr.io/up9-docker-hub/mizu/debug:latest`
### Connecting
1. Start mizu using the cli with the debug image `mizu tap --mizu-image gcr.io/up9-docker-hub/mizu/debug:latest {tapped_pod_name}`
1. Start mizu using the cli with the debug
image `mizu tap --set agent-image=gcr.io/up9-docker-hub/mizu/debug:latest {tapped_pod_name}`
2. Forward the debug port using `kubectl port-forward -n default mizu-api-server 2345:2345`
3. Run the run/debug configuration you've created earlier in Intellij.

View File

@@ -5,6 +5,7 @@ go 1.16
require (
github.com/djherbis/atime v1.0.0
github.com/fsnotify/fsnotify v1.4.9
github.com/getkin/kin-openapi v0.76.0
github.com/gin-contrib/static v0.0.1
github.com/gin-gonic/gin v1.7.2
github.com/go-playground/locales v0.13.0
@@ -12,9 +13,9 @@ require (
github.com/go-playground/validator/v10 v10.5.0
github.com/google/martian v2.1.0+incompatible
github.com/gorilla/websocket v1.4.2
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/orcaman/concurrent-map v0.0.0-20210106121528-16402b402231
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7
github.com/up9inc/mizu/shared v0.0.0
github.com/up9inc/mizu/tap v0.0.0
github.com/up9inc/mizu/tap/api v0.0.0

View File

@@ -68,6 +68,10 @@ github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoD
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/getkin/kin-openapi v0.76.0 h1:j77zg3Ec+k+r+GA3d8hBoXpAc6KX9TbBPrwQGBIy2sY=
github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-contrib/static v0.0.1 h1:JVxuvHPuUfkoul12N7dtQw7KRn/pSMq7Ue1Va9Swm1U=
@@ -83,10 +87,13 @@ github.com/go-logr/logr v0.4.0 h1:K7/B1jt6fIBQVd4Owv2MqGQClcgf0R266+7C/QjRcLc=
github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY=
github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8=
github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo=
github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/swag v0.19.5 h1:lTz6Ys4CmqqCQmZPBlbQENR1/GucA2bzYTE12Pw4tFY=
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-playground/assert/v2 v2.0.1 h1:MsBgLAaY856+nPRTKrp3/OZK38U/wa0CcBYNjji3q3A=
github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
@@ -178,6 +185,8 @@ github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I=
github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
@@ -216,6 +225,7 @@ github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e h1:hB2xlXdHp/pmPZq0y3QnmWAArdw9PqbmotexnWx/FU8=
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
@@ -241,6 +251,8 @@ github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 h1:lDH9UUVJtmYCjyT0CI4q8xvlXPxeZ0gYCVvWbmPlp88=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
github.com/orcaman/concurrent-map v0.0.0-20210106121528-16402b402231 h1:fa50YL1pzKW+1SsBnJDOHppJN9stOEwS+CRWyUtyYGU=
github.com/orcaman/concurrent-map v0.0.0-20210106121528-16402b402231/go.mod h1:Lu3tH6HLW3feq74c2GC+jIMS/K2CFcDWnWD9XkenwhI=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
@@ -256,8 +268,6 @@ github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:
github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7 h1:jkvpcEatpwuMF5O5LVxTnehj6YZ/aEZN4NWD/Xml4pI=
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7/go.mod h1:KTrHyWpO1sevuXPZwyeZc72ddWRFqNSKDFl7uVWKpg0=
github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
@@ -273,6 +283,7 @@ github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoH
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
@@ -531,6 +542,7 @@ gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWD
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -4,15 +4,7 @@ import (
"encoding/json"
"flag"
"fmt"
"github.com/gin-contrib/static"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/tap"
tapApi "github.com/up9inc/mizu/tap/api"
"io/ioutil"
"log"
"mizuserver/pkg/api"
"mizuserver/pkg/controllers"
"mizuserver/pkg/models"
@@ -26,6 +18,15 @@ import (
"path/filepath"
"plugin"
"sort"
"github.com/gin-contrib/static"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/op/go-logging"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap"
tapApi "github.com/up9inc/mizu/tap/api"
)
var tapperMode = flag.Bool("tap", false, "Run in tapper mode without API")
@@ -40,6 +41,8 @@ var extensions []*tapApi.Extension // global
var extensionsMap map[string]*tapApi.Extension // global
func main() {
logLevel := determineLogLevel()
logger.InitLoggerStderrOnly(logLevel)
flag.Parse()
loadExtensions()
@@ -63,7 +66,7 @@ func main() {
hostApi(nil)
} else if *tapperMode {
rlog.Infof("Starting tapper, websocket address: %s", *apiServerAddress)
logger.Log.Infof("Starting tapper, websocket address: %s", *apiServerAddress)
if *apiServerAddress == "" {
panic("API server address must be provided with --api-server-address when using --tap")
}
@@ -71,7 +74,7 @@ func main() {
tapTargets := getTapTargets()
if tapTargets != nil {
tap.SetFilterAuthorities(tapTargets)
rlog.Infof("Filtering for the following authorities: %v", tap.GetFilterIPs())
logger.Log.Infof("Filtering for the following authorities: %v", tap.GetFilterIPs())
}
filteredOutputItemsChannel := make(chan *tapApi.OutputChannelItem)
@@ -84,7 +87,7 @@ func main() {
if err != nil {
panic(fmt.Sprintf("Error connecting to socket server at %s %v", *apiServerAddress, err))
}
rlog.Infof("Connected successfully to websocket %s", *apiServerAddress)
logger.Log.Infof("Connected successfully to websocket %s", *apiServerAddress)
go pipeTapChannelToSocket(socketConnection, filteredOutputItemsChannel)
} else if *apiServerMode {
@@ -117,7 +120,7 @@ func main() {
signal.Notify(signalChan, os.Interrupt)
<-signalChan
rlog.Info("Exiting")
logger.Log.Info("Exiting")
}
func loadExtensions() {
@@ -126,13 +129,13 @@ func loadExtensions() {
files, err := ioutil.ReadDir(extensionsDir)
if err != nil {
log.Fatal(err)
logger.Log.Fatal(err)
}
extensions = make([]*tapApi.Extension, len(files))
extensionsMap = make(map[string]*tapApi.Extension)
for i, file := range files {
filename := file.Name()
rlog.Infof("Loading extension: %s\n", filename)
logger.Log.Infof("Loading extension: %s\n", filename)
extension := &tapApi.Extension{
Path: path.Join(extensionsDir, filename),
}
@@ -157,7 +160,7 @@ func loadExtensions() {
})
for _, extension := range extensions {
log.Printf("Extension Properties: %+v\n", extension)
logger.Log.Infof("Extension Properties: %+v\n", extension)
}
controllers.InitExtensionsMap(extensionsMap)
@@ -274,7 +277,7 @@ func pipeTapChannelToSocket(connection *websocket.Conn, messageDataChannel <-cha
for messageData := range messageDataChannel {
marshaledData, err := models.CreateWebsocketTappedEntryMessage(messageData)
if err != nil {
rlog.Errorf("error converting message to json %v, err: %s, (%v,%+v)", messageData, err, err, err)
logger.Log.Errorf("error converting message to json %v, err: %s, (%v,%+v)", messageData, err, err, err)
continue
}
@@ -282,7 +285,7 @@ func pipeTapChannelToSocket(connection *websocket.Conn, messageDataChannel <-cha
// and goes into the intermediate WebSocket.
err = connection.WriteMessage(websocket.TextMessage, marshaledData)
if err != nil {
rlog.Errorf("error sending message through socket server %v, err: %s, (%v,%+v)", messageData, err, err, err)
logger.Log.Errorf("error sending message through socket server %v, err: %s, (%v,%+v)", messageData, err, err, err)
continue
}
}
@@ -302,3 +305,11 @@ func getSyncEntriesConfig() *shared.SyncEntriesConfig {
return syncEntriesConfig
}
func determineLogLevel() (logLevel logging.Level) {
logLevel = logging.INFO
if os.Getenv(shared.DebugModeEnvVar) == "1" {
logLevel = logging.DEBUG
}
return
}

View File

@@ -0,0 +1,110 @@
package api
import (
"bytes"
"context"
"fmt"
"io/ioutil"
"net/http"
"github.com/getkin/kin-openapi/openapi3"
"github.com/getkin/kin-openapi/openapi3filter"
"github.com/getkin/kin-openapi/routers"
legacyrouter "github.com/getkin/kin-openapi/routers/legacy"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
const (
ContractNotApplicable api.ContractStatus = 0
ContractPassed api.ContractStatus = 1
ContractFailed api.ContractStatus = 2
)
func loadOAS(ctx context.Context) (doc *openapi3.T, contractContent string, router routers.Router, err error) {
path := fmt.Sprintf("%s/%s", shared.RulePolicyPath, shared.ContractFileName)
bytes, err := ioutil.ReadFile(path)
if err != nil {
logger.Log.Error(err.Error())
return
}
contractContent = string(bytes)
loader := &openapi3.Loader{Context: ctx}
doc, _ = loader.LoadFromData(bytes)
err = doc.Validate(ctx)
if err != nil {
logger.Log.Error(err.Error())
return
}
router, _ = legacyrouter.NewRouter(doc)
return
}
func validateOAS(ctx context.Context, doc *openapi3.T, router routers.Router, req *http.Request, res *http.Response) (isValid bool, reqErr error, resErr error) {
isValid = true
reqErr = nil
resErr = nil
// Find route
route, pathParams, err := router.FindRoute(req)
if err != nil {
return
}
// Validate request
requestValidationInput := &openapi3filter.RequestValidationInput{
Request: req,
PathParams: pathParams,
Route: route,
}
if reqErr = openapi3filter.ValidateRequest(ctx, requestValidationInput); reqErr != nil {
isValid = false
}
responseValidationInput := &openapi3filter.ResponseValidationInput{
RequestValidationInput: requestValidationInput,
Status: res.StatusCode,
Header: res.Header,
}
if res.Body != nil {
body, _ := ioutil.ReadAll(res.Body)
res.Body = ioutil.NopCloser(bytes.NewBuffer(body))
responseValidationInput.SetBodyBytes(body)
}
// Validate response.
if resErr = openapi3filter.ValidateResponse(ctx, responseValidationInput); resErr != nil {
isValid = false
}
return
}
func handleOAS(ctx context.Context, doc *openapi3.T, router routers.Router, req *http.Request, res *http.Response, contractContent string) (contract api.Contract) {
contract = api.Contract{
Content: contractContent,
Status: ContractNotApplicable,
}
isValid, reqErr, resErr := validateOAS(ctx, doc, router, req, res)
if isValid {
contract.Status = ContractPassed
} else {
contract.Status = ContractFailed
if reqErr != nil {
contract.RequestReason = reqErr.Error()
} else {
contract.RequestReason = ""
}
if resErr != nil {
contract.ResponseReason = resErr.Error()
} else {
contract.ResponseReason = ""
}
}
return
}

View File

@@ -17,7 +17,7 @@ import (
"go.mongodb.org/mongo-driver/bson/primitive"
"github.com/google/martian/har"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/logger"
tapApi "github.com/up9inc/mizu/tap/api"
"mizuserver/pkg/models"
@@ -31,7 +31,7 @@ func StartResolving(namespace string) {
errOut := make(chan error, 100)
res, err := resolver.NewFromInCluster(errOut, namespace)
if err != nil {
rlog.Infof("error creating k8s resolver %s", err)
logger.Log.Infof("error creating k8s resolver %s", err)
return
}
ctx := context.Background()
@@ -40,7 +40,7 @@ func StartResolving(namespace string) {
for {
select {
case err := <-errOut:
rlog.Infof("name resolving error %s", err)
logger.Log.Infof("name resolving error %s", err)
}
}
}()
@@ -59,7 +59,7 @@ func StartReadingEntries(harChannel <-chan *tapApi.OutputChannelItem, workingDir
func startReadingFiles(workingDir string) {
if err := os.MkdirAll(workingDir, os.ModePerm); err != nil {
rlog.Errorf("Failed to make dir: %s, err: %v", workingDir, err)
logger.Log.Errorf("Failed to make dir: %s, err: %v", workingDir, err)
return
}
@@ -76,7 +76,7 @@ func startReadingFiles(workingDir string) {
sort.Sort(utils.ByModTime(harFiles))
if len(harFiles) == 0 {
rlog.Infof("Waiting for new files\n")
logger.Log.Infof("Waiting for new files\n")
time.Sleep(3 * time.Second)
continue
}
@@ -99,6 +99,14 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
panic("Channel of captured messages is nil")
}
disableOASValidation := false
ctx := context.Background()
doc, contractContent, router, err := loadOAS(ctx)
if err != nil {
logger.Log.Infof("Disabled OAS validation: %s\n", err.Error())
disableOASValidation = true
}
for item := range outputItems {
providers.EntryAdded()
@@ -107,8 +115,19 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
mizuEntry := extension.Dissector.Analyze(item, primitive.NewObjectID().Hex(), resolvedSource, resolvedDestionation)
baseEntry := extension.Dissector.Summarize(mizuEntry)
mizuEntry.EstimatedSizeBytes = getEstimatedEntrySizeBytes(mizuEntry)
database.CreateEntry(mizuEntry)
if extension.Protocol.Name == "http" {
if !disableOASValidation {
var httpPair tapApi.HTTPRequestResponsePair
json.Unmarshal([]byte(mizuEntry.Entry), &httpPair)
contract := handleOAS(ctx, doc, router, httpPair.Request.Payload.RawRequest, httpPair.Response.Payload.RawResponse, contractContent)
baseEntry.ContractStatus = contract.Status
mizuEntry.ContractStatus = contract.Status
mizuEntry.ContractRequestReason = contract.RequestReason
mizuEntry.ContractResponseReason = contract.ResponseReason
mizuEntry.ContractContent = contract.Content
}
var pair tapApi.RequestResponsePair
json.Unmarshal([]byte(mizuEntry.Entry), &pair)
harEntry, err := utils.NewEntry(&pair)
@@ -117,6 +136,7 @@ func startReadingChannel(outputItems <-chan *tapApi.OutputChannelItem, extension
baseEntry.Rules = rules
}
}
database.CreateEntry(mizuEntry)
baseEntryBytes, _ := models.CreateBaseEntryWebSocketMessage(baseEntry)
BroadcastToBrowserClients(baseEntryBytes)
@@ -128,7 +148,7 @@ func resolveIP(connectionInfo *tapApi.ConnectionInfo) (resolvedSource string, re
unresolvedSource := connectionInfo.ClientIP
resolvedSource = k8sResolver.Resolve(unresolvedSource)
if resolvedSource == "" {
rlog.Debugf("Cannot find resolved name to source: %s\n", unresolvedSource)
logger.Log.Debugf("Cannot find resolved name to source: %s\n", unresolvedSource)
if os.Getenv("SKIP_NOT_RESOLVED_SOURCE") == "1" {
return
}
@@ -136,7 +156,7 @@ func resolveIP(connectionInfo *tapApi.ConnectionInfo) (resolvedSource string, re
unresolvedDestination := fmt.Sprintf("%s:%s", connectionInfo.ServerIP, connectionInfo.ServerPort)
resolvedDestination = k8sResolver.Resolve(unresolvedDestination)
if resolvedDestination == "" {
rlog.Debugf("Cannot find resolved name to dest: %s\n", unresolvedDestination)
logger.Log.Debugf("Cannot find resolved name to dest: %s\n", unresolvedDestination)
if os.Getenv("SKIP_NOT_RESOLVED_DEST") == "1" {
return
}

View File

@@ -2,13 +2,14 @@ package api
import (
"errors"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/debounce"
"net/http"
"sync"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
"github.com/up9inc/mizu/shared/debounce"
"github.com/up9inc/mizu/shared/logger"
)
type EventHandlers interface {
@@ -50,7 +51,7 @@ func WebSocketRoutes(app *gin.Engine, eventHandlers EventHandlers) {
func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers EventHandlers, isTapper bool) {
conn, err := websocketUpgrader.Upgrade(w, r, nil)
if err != nil {
rlog.Errorf("Failed to set websocket upgrade: %v", err)
logger.Log.Errorf("Failed to set websocket upgrade: %v", err)
return
}
@@ -71,7 +72,7 @@ func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers Even
for {
_, msg, err := conn.ReadMessage()
if err != nil {
rlog.Errorf("Error reading message, socket id: %d, error: %v", socketId, err)
logger.Log.Errorf("Error reading message, socket id: %d, error: %v", socketId, err)
break
}
eventHandlers.WebSocketMessage(socketId, msg)
@@ -81,7 +82,7 @@ func websocketHandler(w http.ResponseWriter, r *http.Request, eventHandlers Even
func socketCleanup(socketId int, socketConnection *SocketConnection) {
err := socketConnection.connection.Close()
if err != nil {
rlog.Errorf("Error closing socket connection for socket id %d: %v\n", socketId, err)
logger.Log.Errorf("Error closing socket connection for socket id %d: %v\n", socketId, err)
}
websocketIdsLock.Lock()
@@ -92,7 +93,7 @@ func socketCleanup(socketId int, socketConnection *SocketConnection) {
}
var db = debounce.NewDebouncer(time.Second*5, func() {
rlog.Error("Successfully sent to socket")
logger.Log.Error("Successfully sent to socket")
})
func SendToSocket(socketId int, message []byte) error {
@@ -104,7 +105,7 @@ func SendToSocket(socketId int, message []byte) error {
var sent = false
time.AfterFunc(time.Second*5, func() {
if !sent {
rlog.Error("Socket timed out")
logger.Log.Error("Socket timed out")
socketCleanup(socketId, socketObj)
}
})

View File

@@ -10,8 +10,8 @@ import (
tapApi "github.com/up9inc/mizu/tap/api"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
)
var browserClientSocketUUIDs = make([]int, 0)
@@ -28,10 +28,10 @@ func init() {
func (h *RoutesEventHandlers) WebSocketConnect(socketId int, isTapper bool) {
if isTapper {
rlog.Infof("Websocket event - Tapper connected, socket ID: %d", socketId)
logger.Log.Infof("Websocket event - Tapper connected, socket ID: %d", socketId)
providers.TapperAdded()
} else {
rlog.Infof("Websocket event - Browser socket connected, socket ID: %d", socketId)
logger.Log.Infof("Websocket event - Browser socket connected, socket ID: %d", socketId)
socketListLock.Lock()
browserClientSocketUUIDs = append(browserClientSocketUUIDs, socketId)
socketListLock.Unlock()
@@ -40,10 +40,10 @@ func (h *RoutesEventHandlers) WebSocketConnect(socketId int, isTapper bool) {
func (h *RoutesEventHandlers) WebSocketDisconnect(socketId int, isTapper bool) {
if isTapper {
rlog.Infof("Websocket event - Tapper disconnected, socket ID: %d", socketId)
logger.Log.Infof("Websocket event - Tapper disconnected, socket ID: %d", socketId)
providers.TapperRemoved()
} else {
rlog.Infof("Websocket event - Browser socket disconnected, socket ID: %d", socketId)
logger.Log.Infof("Websocket event - Browser socket disconnected, socket ID: %d", socketId)
socketListLock.Lock()
removeSocketUUIDFromBrowserSlice(socketId)
socketListLock.Unlock()
@@ -55,7 +55,7 @@ func BroadcastToBrowserClients(message []byte) {
go func(socketId int) {
err := SendToSocket(socketId, message)
if err != nil {
rlog.Errorf("error sending message to socket ID %d: %v", socketId, err)
logger.Log.Errorf("error sending message to socket ID %d: %v", socketId, err)
}
}(socketId)
}
@@ -65,14 +65,14 @@ func (h *RoutesEventHandlers) WebSocketMessage(_ int, message []byte) {
var socketMessageBase shared.WebSocketMessageMetadata
err := json.Unmarshal(message, &socketMessageBase)
if err != nil {
rlog.Infof("Could not unmarshal websocket message %v\n", err)
logger.Log.Infof("Could not unmarshal websocket message %v\n", err)
} else {
switch socketMessageBase.MessageType {
case shared.WebSocketMessageTypeTappedEntry:
var tappedEntryMessage models.WebSocketTappedEntryMessage
err := json.Unmarshal(message, &tappedEntryMessage)
if err != nil {
rlog.Infof("Could not unmarshal message of message type %s %v\n", socketMessageBase.MessageType, err)
logger.Log.Infof("Could not unmarshal message of message type %s %v\n", socketMessageBase.MessageType, err)
} else {
// NOTE: This is where the message comes back from the intermediate WebSocket to code.
h.SocketOutChannel <- tappedEntryMessage.Data
@@ -81,7 +81,7 @@ func (h *RoutesEventHandlers) WebSocketMessage(_ int, message []byte) {
var statusMessage shared.WebSocketStatusMessage
err := json.Unmarshal(message, &statusMessage)
if err != nil {
rlog.Infof("Could not unmarshal message of message type %s %v\n", socketMessageBase.MessageType, err)
logger.Log.Infof("Could not unmarshal message of message type %s %v\n", socketMessageBase.MessageType, err)
} else {
providers.TapStatus.Pods = statusMessage.TappingStatus.Pods
BroadcastToBrowserClients(message)
@@ -90,12 +90,12 @@ func (h *RoutesEventHandlers) WebSocketMessage(_ int, message []byte) {
var outboundLinkMessage models.WebsocketOutboundLinkMessage
err := json.Unmarshal(message, &outboundLinkMessage)
if err != nil {
rlog.Infof("Could not unmarshal message of message type %s %v\n", socketMessageBase.MessageType, err)
logger.Log.Infof("Could not unmarshal message of message type %s %v\n", socketMessageBase.MessageType, err)
} else {
handleTLSLink(outboundLinkMessage)
}
default:
rlog.Infof("Received socket message of type %s for which no handlers are defined", socketMessageBase.MessageType)
logger.Log.Infof("Received socket message of type %s for which no handlers are defined", socketMessageBase.MessageType)
}
}
}
@@ -116,9 +116,9 @@ func handleTLSLink(outboundLinkMessage models.WebsocketOutboundLinkMessage) {
}
marshaledMessage, err := json.Marshal(outboundLinkMessage)
if err != nil {
rlog.Errorf("Error marshaling outbound link message for broadcasting: %v", err)
logger.Log.Errorf("Error marshaling outbound link message for broadcasting: %v", err)
} else {
rlog.Errorf("Broadcasting outboundlink message %s", string(marshaledMessage))
logger.Log.Errorf("Broadcasting outboundlink message %s", string(marshaledMessage))
BroadcastToBrowserClients(marshaledMessage)
}
}

View File

@@ -3,19 +3,13 @@ package controllers
import (
"encoding/json"
"fmt"
"github.com/gin-gonic/gin"
tapApi "github.com/up9inc/mizu/tap/api"
"mizuserver/pkg/database"
"mizuserver/pkg/models"
"mizuserver/pkg/providers"
"mizuserver/pkg/up9"
"mizuserver/pkg/utils"
"mizuserver/pkg/validation"
"net/http"
"time"
"github.com/google/martian/har"
"github.com/gin-gonic/gin"
tapApi "github.com/up9inc/mizu/tap/api"
)
var extensionsMap map[string]*tapApi.Extension // global
@@ -62,47 +56,6 @@ func GetEntries(c *gin.Context) {
c.JSON(http.StatusOK, baseEntries)
}
func GetFullEntries(c *gin.Context) {
entriesFilter := &models.HarFetchRequestQuery{}
if err := c.BindQuery(entriesFilter); err != nil {
c.JSON(http.StatusBadRequest, err)
}
err := validation.Validate(entriesFilter)
if err != nil {
c.JSON(http.StatusBadRequest, err)
}
var timestampFrom, timestampTo int64
if entriesFilter.From < 0 {
timestampFrom = 0
} else {
timestampFrom = entriesFilter.From
}
if entriesFilter.To <= 0 {
timestampTo = time.Now().UnixNano() / int64(time.Millisecond)
} else {
timestampTo = entriesFilter.To
}
entriesArray := database.GetEntriesFromDb(timestampFrom, timestampTo, nil)
result := make([]har.Entry, 0)
for _, data := range entriesArray {
var pair tapApi.RequestResponsePair
if err := json.Unmarshal([]byte(data.Entry), &pair); err != nil {
continue
}
harEntry, err := utils.NewEntry(&pair)
if err != nil {
continue
}
result = append(result, *harEntry)
}
c.JSON(http.StatusOK, result)
}
func GetEntry(c *gin.Context) {
var entryData tapApi.MizuEntry
database.GetEntriesTable().
@@ -133,30 +86,3 @@ func GetEntry(c *gin.Context) {
IsRulesEnabled: isRulesEnabled,
})
}
func DeleteAllEntries(c *gin.Context) {
database.GetEntriesTable().
Where("1 = 1").
Delete(&tapApi.MizuEntry{})
c.JSON(http.StatusOK, map[string]string{
"msg": "Success",
})
}
func GetGeneralStats(c *gin.Context) {
c.JSON(http.StatusOK, providers.GetGeneralStats())
}
func GetTappingStatus(c *gin.Context) {
c.JSON(http.StatusOK, providers.TapStatus)
}
func AnalyzeInformation(c *gin.Context) {
c.JSON(http.StatusOK, up9.GetAnalyzeInfo())
}
func GetRecentTLSLinks(c *gin.Context) {
c.JSON(http.StatusOK, providers.GetAllRecentTLSAddresses())
}

View File

@@ -1,12 +0,0 @@
package controllers
import (
"github.com/gin-gonic/gin"
"mizuserver/pkg/holder"
"net/http"
)
func GetCurrentResolvingInformation(c *gin.Context) {
c.JSON(http.StatusOK, holder.GetResolver().GetMap())
}

View File

@@ -2,13 +2,16 @@ package controllers
import (
"encoding/json"
"github.com/gin-gonic/gin"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared"
"mizuserver/pkg/api"
"mizuserver/pkg/holder"
"mizuserver/pkg/providers"
"mizuserver/pkg/up9"
"mizuserver/pkg/validation"
"net/http"
"github.com/gin-gonic/gin"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
)
func PostTappedPods(c *gin.Context) {
@@ -21,11 +24,11 @@ func PostTappedPods(c *gin.Context) {
c.JSON(http.StatusBadRequest, err)
return
}
rlog.Infof("[Status] POST request: %d tapped pods", len(tapStatus.Pods))
logger.Log.Infof("[Status] POST request: %d tapped pods", len(tapStatus.Pods))
providers.TapStatus.Pods = tapStatus.Pods
message := shared.CreateWebSocketStatusMessage(*tapStatus)
if jsonBytes, err := json.Marshal(message); err != nil {
rlog.Errorf("Could not Marshal message %v\n", err)
logger.Log.Errorf("Could not Marshal message %v\n", err)
} else {
api.BroadcastToBrowserClients(jsonBytes)
}
@@ -44,3 +47,23 @@ func GetAuthStatus(c *gin.Context) {
c.JSON(http.StatusOK, authStatus)
}
func GetTappingStatus(c *gin.Context) {
c.JSON(http.StatusOK, providers.TapStatus)
}
func AnalyzeInformation(c *gin.Context) {
c.JSON(http.StatusOK, up9.GetAnalyzeInfo())
}
func GetGeneralStats(c *gin.Context) {
c.JSON(http.StatusOK, providers.GetGeneralStats())
}
func GetRecentTLSLinks(c *gin.Context) {
c.JSON(http.StatusOK, providers.GetAllRecentTLSAddresses())
}
func GetCurrentResolvingInformation(c *gin.Context) {
c.JSON(http.StatusOK, holder.GetResolver().GetMap())
}

View File

@@ -1,15 +1,14 @@
package database
import (
"log"
"os"
"strconv"
"time"
"github.com/fsnotify/fsnotify"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/debounce"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/shared/units"
tapApi "github.com/up9inc/mizu/tap/api"
)
@@ -20,13 +19,13 @@ const defaultMaxDatabaseSizeBytes int64 = 200 * 1000 * 1000
func StartEnforcingDatabaseSize() {
watcher, err := fsnotify.NewWatcher()
if err != nil {
log.Fatalf("Error creating filesystem watcher for db size enforcement: %v\n", err)
logger.Log.Fatalf("Error creating filesystem watcher for db size enforcement: %v\n", err)
return
}
maxEntriesDBByteSize, err := getMaxEntriesDBByteSize()
if err != nil {
log.Fatalf("Error parsing max db size: %v\n", err)
logger.Log.Fatalf("Error parsing max db size: %v\n", err)
return
}
@@ -48,14 +47,14 @@ func StartEnforcingDatabaseSize() {
if !ok {
return // closed channel
}
rlog.Errorf("filesystem watcher encountered error:%v", err)
logger.Log.Errorf("filesystem watcher encountered error:%v", err)
}
}
}()
err = watcher.Add(DBPath)
if err != nil {
log.Fatalf("Error adding %s to filesystem watcher for db size enforcement: %v\n", DBPath, err)
logger.Log.Fatalf("Error adding %s to filesystem watcher for db size enforcement: %v\n", DBPath, err)
}
}
@@ -73,7 +72,7 @@ func getMaxEntriesDBByteSize() (int64, error) {
func checkFileSize(maxSizeBytes int64) {
fileStat, err := os.Stat(DBPath)
if err != nil {
rlog.Errorf("Error checking %s file size: %v", DBPath, err)
logger.Log.Errorf("Error checking %s file size: %v", DBPath, err)
} else {
if fileStat.Size() > maxSizeBytes {
pruneOldEntries(fileStat.Size())
@@ -90,7 +89,7 @@ func pruneOldEntries(currentFileSize int64) {
rows, err := GetEntriesTable().Limit(10000).Order("id").Rows()
if err != nil {
rlog.Errorf("Error getting 10000 first db rows: %v", err)
logger.Log.Errorf("Error getting 10000 first db rows: %v", err)
return
}
@@ -103,7 +102,7 @@ func pruneOldEntries(currentFileSize int64) {
var entry tapApi.MizuEntry
err = DB.ScanRows(rows, &entry)
if err != nil {
rlog.Errorf("Error scanning db row: %v", err)
logger.Log.Errorf("Error scanning db row: %v", err)
continue
}
@@ -115,8 +114,8 @@ func pruneOldEntries(currentFileSize int64) {
GetEntriesTable().Where(entryIdsToRemove).Delete(tapApi.MizuEntry{})
// VACUUM causes sqlite to shrink the db file after rows have been deleted, the db file will not shrink without this
DB.Exec("VACUUM")
rlog.Errorf("Removed %d rows and cleared %s", len(entryIdsToRemove), units.BytesToHumanReadable(bytesToBeRemoved))
logger.Log.Errorf("Removed %d rows and cleared %s", len(entryIdsToRemove), units.BytesToHumanReadable(bytesToBeRemoved))
} else {
rlog.Error("Found no rows to remove when pruning")
logger.Log.Error("Found no rows to remove when pruning")
}
}

View File

@@ -22,11 +22,6 @@ type EntriesFilter struct {
Timestamp int64 `form:"timestamp" validate:"required,min=1"`
}
type HarFetchRequestQuery struct {
From int64 `form:"from"`
To int64 `form:"to"`
}
type WebSocketEntryMessage struct {
*shared.WebSocketMessageMetadata
Data *tapApi.BaseEntryDetails `json:"data,omitempty"`

View File

@@ -32,7 +32,7 @@ Now you will be able to import `github.com/up9inc/mizu/resolver` in any `.go` fi
errOut := make(chan error, 100)
k8sResolver, err := resolver.NewFromOutOfCluster("", errOut)
if err != nil {
rlog.Errorf("error creating k8s resolver %s", err)
logger.Log.Errorf("error creating k8s resolver %s", err)
}
ctx, cancel := context.WithCancel(context.Background())
@@ -40,15 +40,15 @@ k8sResolver.Start(ctx)
resolvedName := k8sResolver.Resolve("10.107.251.91") // will always return `nil` in real scenarios as the internal map takes a moment to populate after `Start` is called
if resolvedName != nil {
rlog.Errorf("resolved 10.107.251.91=%s", *resolvedName)
logger.Log.Errorf("resolved 10.107.251.91=%s", *resolvedName)
} else {
rlog.Error("Could not find a resolved name for 10.107.251.91")
logger.Log.Error("Could not find a resolved name for 10.107.251.91")
}
for {
select {
case err := <- errOut:
rlog.Errorf("name resolving error %s", err)
logger.Log.Errorf("name resolving error %s", err)
}
}
```

View File

@@ -5,7 +5,7 @@ import (
"errors"
"fmt"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/logger"
k8serrors "k8s.io/apimachinery/pkg/api/errors"
cmap "github.com/orcaman/concurrent-map"
@@ -157,10 +157,10 @@ func (resolver *Resolver) watchServices(ctx context.Context) error {
func (resolver *Resolver) saveResolvedName(key string, resolved string, eventType watch.EventType) {
if eventType == watch.Deleted {
resolver.nameMap.Remove(key)
rlog.Infof("setting %s=nil\n", key)
logger.Log.Infof("setting %s=nil\n", key)
} else {
resolver.nameMap.Set(key, resolved)
rlog.Infof("setting %s=%s\n", key, resolved)
logger.Log.Infof("setting %s=%s\n", key, resolved)
}
}
@@ -181,7 +181,7 @@ func (resolver *Resolver) infiniteErrorHandleRetryFunc(ctx context.Context, fun
var statusError *k8serrors.StatusError
if errors.As(err, &statusError) {
if statusError.ErrStatus.Reason == metav1.StatusReasonForbidden {
rlog.Infof("Resolver loop encountered permission error, aborting event listening - %v\n", err)
logger.Log.Infof("Resolver loop encountered permission error, aborting event listening - %v\n", err)
return
}
}

View File

@@ -7,17 +7,8 @@ import (
// EntriesRoutes defines the group of har entries routes.
func EntriesRoutes(ginApp *gin.Engine) {
routeGroup := ginApp.Group("/api")
routeGroup := ginApp.Group("/entries")
routeGroup.GET("/entries", controllers.GetEntries) // get entries (base/thin entries)
routeGroup.GET("/entries/:entryId", controllers.GetEntry) // get single (full) entry
routeGroup.GET("/exportEntries", controllers.GetFullEntries)
routeGroup.GET("/resolving", controllers.GetCurrentResolvingInformation)
routeGroup.GET("/resetDB", controllers.DeleteAllEntries) // get single (full) entry
routeGroup.GET("/generalStats", controllers.GetGeneralStats) // get general stats about entries in DB
routeGroup.GET("/tapStatus", controllers.GetTappingStatus) // get tapping status
routeGroup.GET("/analyzeStatus", controllers.AnalyzeInformation)
routeGroup.GET("/recentTLSLinks", controllers.GetRecentTLSLinks)
routeGroup.GET("/", controllers.GetEntries) // get entries (base/thin entries)
routeGroup.GET("/:entryId", controllers.GetEntry) // get single (full) entry
}

View File

@@ -9,8 +9,16 @@ func StatusRoutes(ginApp *gin.Engine) {
routeGroup := ginApp.Group("/status")
routeGroup.POST("/tappedPods", controllers.PostTappedPods)
routeGroup.GET("/tappersCount", controllers.GetTappersCount)
routeGroup.GET("/tap", controllers.GetTappingStatus)
routeGroup.GET("/auth", controllers.GetAuthStatus)
routeGroup.GET("/analyze", controllers.AnalyzeInformation)
routeGroup.GET("/general", controllers.GetGeneralStats) // get general stats about entries in DB
routeGroup.GET("/recentTLSLinks", controllers.GetRecentTLSLinks)
routeGroup.GET("/resolving", controllers.GetCurrentResolvingInformation)
}

View File

@@ -8,7 +8,7 @@ import (
"regexp"
"strings"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/logger"
"github.com/google/martian/har"
"github.com/up9inc/mizu/shared"
@@ -46,7 +46,7 @@ func ValidateService(serviceFromRule string, service string) bool {
func MatchRequestPolicy(harEntry har.Entry, service string) (resultPolicyToSend []RulesMatched, isEnabled bool) {
enforcePolicy, err := shared.DecodeEnforcePolicy(fmt.Sprintf("%s/%s", shared.RulePolicyPath, shared.RulePolicyFileName))
if err == nil {
if err == nil && len(enforcePolicy.Rules) > 0 {
isEnabled = true
}
for _, rule := range enforcePolicy.Rules {
@@ -69,7 +69,7 @@ func MatchRequestPolicy(harEntry har.Entry, service string) (resultPolicyToSend
if err != nil {
continue
}
rlog.Info(matchValue, rule.Value)
logger.Log.Info(matchValue, rule.Value)
} else {
val := fmt.Sprint(out)
matchValue, err = regexp.MatchString(rule.Value, val)

View File

@@ -5,12 +5,7 @@ import (
"compress/zlib"
"encoding/json"
"fmt"
"github.com/google/martian/har"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared"
tapApi "github.com/up9inc/mizu/tap/api"
"io/ioutil"
"log"
"mizuserver/pkg/database"
"mizuserver/pkg/utils"
"net/http"
@@ -18,6 +13,11 @@ import (
"regexp"
"strings"
"time"
"github.com/google/martian/har"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
tapApi "github.com/up9inc/mizu/tap/api"
)
const (
@@ -75,9 +75,6 @@ func getAuthHeader(guestMode bool) string {
func GetTrafficDumpUrl(analyzeDestination string, analyzeModel string) *url.URL {
strUrl := fmt.Sprintf("https://traffic.%s/dumpTrafficBulk/%s", analyzeDestination, analyzeModel)
if strings.HasPrefix(analyzeDestination, "http") {
strUrl = fmt.Sprintf("%s/api/workspace/dumpTrafficBulk", analyzeDestination)
}
postUrl, _ := url.Parse(strUrl)
return postUrl
}
@@ -112,14 +109,14 @@ func GetAnalyzeInfo() *shared.AnalyzeStatus {
}
func SyncEntries(syncEntriesConfig *shared.SyncEntriesConfig) error {
rlog.Infof("Sync entries - started\n")
logger.Log.Infof("Sync entries - started\n")
var (
token, model string
guestMode bool
)
if syncEntriesConfig.Token == "" {
rlog.Infof("Sync entries - creating anonymous token. env %s\n", syncEntriesConfig.Env)
logger.Log.Infof("Sync entries - creating anonymous token. env %s\n", syncEntriesConfig.Env)
guestToken, err := createAnonymousToken(syncEntriesConfig.Env)
if err != nil {
return fmt.Errorf("failed creating anonymous token, err: %v", err)
@@ -132,6 +129,11 @@ func SyncEntries(syncEntriesConfig *shared.SyncEntriesConfig) error {
token = fmt.Sprintf("bearer %s", syncEntriesConfig.Token)
model = syncEntriesConfig.Workspace
guestMode = false
logger.Log.Infof("Sync entries - upserting model. env %s, model %s\n", syncEntriesConfig.Env, model)
if err := upsertModel(token, model, syncEntriesConfig.Env); err != nil {
return fmt.Errorf("failed upserting model, err: %v", err)
}
}
modelRegex, _ := regexp.Compile("[A-Za-z0-9][-A-Za-z0-9_.]*[A-Za-z0-9]+$")
@@ -139,12 +141,37 @@ func SyncEntries(syncEntriesConfig *shared.SyncEntriesConfig) error {
return fmt.Errorf("invalid model name, model name: %s", model)
}
rlog.Infof("Sync entries - syncing. token: %s, model: %s, guest mode: %v\n", token, model, guestMode)
logger.Log.Infof("Sync entries - syncing. token: %s, model: %s, guest mode: %v\n", token, model, guestMode)
go syncEntriesImpl(token, model, syncEntriesConfig.Env, syncEntriesConfig.UploadIntervalSec, guestMode)
return nil
}
func upsertModel(token string, model string, envPrefix string) error {
upsertModelUrl, _ := url.Parse(fmt.Sprintf("https://trcc.%s/models/%s", envPrefix, model))
authHeader := getAuthHeader(false)
req := &http.Request{
Method: http.MethodPost,
URL: upsertModelUrl,
Header: map[string][]string{
authHeader: {token},
},
}
response, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("failed request to upsert model, err: %v", err)
}
// In case the model is not created (not 201) and doesn't exists (not 409)
if response.StatusCode != 201 && response.StatusCode != 409 {
return fmt.Errorf("failed request to upsert model, status code: %v", response.StatusCode)
}
return nil
}
func createAnonymousToken(envPrefix string) (*GuestToken, error) {
tokenUrl := fmt.Sprintf("https://trcc.%s/anonymous/token", envPrefix)
if strings.HasPrefix(envPrefix, "http") {
@@ -152,7 +179,7 @@ func createAnonymousToken(envPrefix string) (*GuestToken, error) {
}
token := &GuestToken{}
if err := getGuestToken(tokenUrl, token); err != nil {
rlog.Infof("Failed to get token, %s", err)
logger.Log.Infof("Failed to get token, %s", err)
return nil, err
}
return token, nil
@@ -164,7 +191,7 @@ func getGuestToken(url string, target *GuestToken) error {
return err
}
defer resp.Body.Close()
rlog.Infof("Got token from the server, starting to json decode... status code: %v", resp.StatusCode)
logger.Log.Infof("Got token from the server, starting to json decode... status code: %v", resp.StatusCode)
return json.NewDecoder(resp.Body).Decode(target)
}
@@ -182,7 +209,7 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
for {
timestampTo := time.Now().UnixNano() / int64(time.Millisecond)
rlog.Infof("Getting entries from %v, to %v\n", timestampFrom, timestampTo)
logger.Log.Infof("Getting entries from %v, to %v\n", timestampFrom, timestampTo)
protocolFilter := "http"
entriesArray := database.GetEntriesFromDb(timestampFrom, timestampTo, &protocolFilter)
@@ -207,13 +234,13 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
result = append(result, *harEntry)
}
rlog.Infof("About to upload %v entries\n", len(result))
logger.Log.Infof("About to upload %v entries\n", len(result))
body, jMarshalErr := json.Marshal(result)
if jMarshalErr != nil {
analyzeInformation.Reset()
rlog.Infof("Stopping sync entries")
log.Fatal(jMarshalErr)
logger.Log.Infof("Stopping sync entries")
logger.Log.Fatal(jMarshalErr)
}
var in bytes.Buffer
@@ -236,17 +263,17 @@ func syncEntriesImpl(token string, model string, envPrefix string, uploadInterva
if _, postErr := http.DefaultClient.Do(req); postErr != nil {
analyzeInformation.Reset()
rlog.Info("Stopping sync entries")
log.Fatal(postErr)
logger.Log.Info("Stopping sync entries")
logger.Log.Fatal(postErr)
}
analyzeInformation.SentCount += len(entriesArray)
rlog.Infof("Finish uploading %v entries to %s\n", len(entriesArray), GetTrafficDumpUrl(envPrefix, model))
logger.Log.Infof("Finish uploading %v entries to %s\n", len(entriesArray), GetTrafficDumpUrl(envPrefix, model))
} else {
rlog.Infof("Nothing to upload")
logger.Log.Infof("Nothing to upload")
}
rlog.Infof("Sleeping for %v...\n", sleepTime)
logger.Log.Infof("Sleeping for %v...\n", sleepTime)
time.Sleep(sleepTime)
timestampFrom = timestampTo
}

View File

@@ -4,12 +4,13 @@ import (
"bytes"
"errors"
"fmt"
"github.com/google/martian/har"
"github.com/romana/rlog"
"github.com/up9inc/mizu/tap/api"
"strconv"
"strings"
"time"
"github.com/google/martian/har"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
// Keep it because we might want cookies in the future
@@ -203,7 +204,7 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
if strings.HasPrefix(mimeType.(string), "application/grpc") {
status, err = strconv.Atoi(_status)
if err != nil {
rlog.Errorf("Failed converting status to int %s (%v,%+v)", err, err, err)
logger.Log.Errorf("Failed converting status to int %s (%v,%+v)", err, err, err)
return nil, errors.New("failed converting response status to int for HAR")
}
}
@@ -224,13 +225,13 @@ func NewResponse(response *api.GenericMessage) (harResponse *har.Response, err e
func NewEntry(pair *api.RequestResponsePair) (*har.Entry, error) {
harRequest, err := NewRequest(&pair.Request)
if err != nil {
rlog.Errorf("Failed converting request to HAR %s (%v,%+v)", err, err, err)
logger.Log.Errorf("Failed converting request to HAR %s (%v,%+v)", err, err, err)
return nil, errors.New("failed converting request to HAR")
}
harResponse, err := NewResponse(&pair.Response)
if err != nil {
rlog.Errorf("Failed converting response to HAR %s (%v,%+v)", err, err, err)
logger.Log.Errorf("Failed converting response to HAR %s (%v,%+v)", err, err, err)
return nil, errors.New("failed converting response to HAR")
}

View File

@@ -3,10 +3,11 @@ package utils
import (
"context"
"fmt"
"github.com/romana/rlog"
"time"
loggerShared "github.com/up9inc/mizu/shared/logger"
"gorm.io/gorm/logger"
"gorm.io/gorm/utils"
"time"
)
// TruncatingLogger implements the gorm logger.Interface interface. Its purpose is to act as gorm's logger while truncating logs to a max of 50 characters to minimise the performance impact
@@ -24,21 +25,21 @@ func (truncatingLogger *TruncatingLogger) Info(_ context.Context, message string
if truncatingLogger.LogLevel < logger.Info {
return
}
rlog.Errorf("gorm info: %.150s", message)
loggerShared.Log.Errorf("gorm info: %.150s", message)
}
func (truncatingLogger *TruncatingLogger) Warn(_ context.Context, message string, __ ...interface{}) {
if truncatingLogger.LogLevel < logger.Warn {
return
}
rlog.Errorf("gorm warning: %.150s", message)
loggerShared.Log.Errorf("gorm warning: %.150s", message)
}
func (truncatingLogger *TruncatingLogger) Error(_ context.Context, message string, __ ...interface{}) {
if truncatingLogger.LogLevel < logger.Error {
return
}
rlog.Errorf("gorm error: %.150s", message)
loggerShared.Log.Errorf("gorm error: %.150s", message)
}
func (truncatingLogger *TruncatingLogger) Trace(ctx context.Context, begin time.Time, fc func() (string, int64), err error) {

View File

@@ -2,8 +2,7 @@ package utils
import (
"context"
"github.com/gin-gonic/gin"
"github.com/romana/rlog"
"fmt"
"net/http"
"net/url"
"os"
@@ -11,6 +10,10 @@ import (
"reflect"
"syscall"
"time"
"github.com/gin-gonic/gin"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
)
// StartServer starts the server with a graceful shutdown
@@ -28,16 +31,16 @@ func StartServer(app *gin.Engine) {
go func() {
_ = <-signals
rlog.Infof("Shutting down...")
logger.Log.Infof("Shutting down...")
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
_ = srv.Shutdown(ctx)
os.Exit(0)
}()
// Run server.
rlog.Infof("Starting the server...")
if err := app.Run(":8899"); err != nil {
rlog.Errorf("Server is not running! Reason: %v", err)
logger.Log.Infof("Starting the server...")
if err := app.Run(fmt.Sprintf(":%d", shared.DefaultApiServerPort)); err != nil {
logger.Log.Errorf("Server is not running! Reason: %v", err)
}
}
@@ -54,14 +57,14 @@ func ReverseSlice(data interface{}) {
func CheckErr(e error) {
if e != nil {
rlog.Errorf("%v", e)
logger.Log.Errorf("%v", e)
}
}
func SetHostname(address, newHostname string) string {
replacedUrl, err := url.Parse(address)
if err != nil {
rlog.Errorf("error replacing hostname to %s in address %s, returning original %v", newHostname, address, err)
logger.Log.Errorf("error replacing hostname to %s in address %s, returning original %v", newHostname, address, err)
return address
}
replacedUrl.Host = newHostname

View File

@@ -4,14 +4,15 @@ import (
"bytes"
"encoding/json"
"fmt"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/shared"
"io/ioutil"
core "k8s.io/api/core/v1"
"net/http"
"net/url"
"time"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
core "k8s.io/api/core/v1"
)
type apiServerProvider struct {
@@ -85,7 +86,7 @@ func (provider *apiServerProvider) GetGeneralStats() (map[string]interface{}, er
if !provider.isReady {
return nil, fmt.Errorf("trying to reach api server when not initialized yet")
}
generalStatsUrl := fmt.Sprintf("%s/api/generalStats", provider.url)
generalStatsUrl := fmt.Sprintf("%s/status/general", provider.url)
response, requestErr := http.Get(generalStatsUrl)
if requestErr != nil {
@@ -108,7 +109,6 @@ func (provider *apiServerProvider) GetGeneralStats() (map[string]interface{}, er
return generalStats, nil
}
func (provider *apiServerProvider) GetVersion() (string, error) {
if !provider.isReady {
return "", fmt.Errorf("trying to reach api server when not initialized yet")

View File

@@ -4,16 +4,17 @@ import (
"context"
"errors"
"fmt"
"github.com/google/uuid"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/uiUtils"
"golang.org/x/oauth2"
"net"
"net/http"
"os"
"time"
"github.com/google/uuid"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared/logger"
"golang.org/x/oauth2"
)
const loginTimeoutInMin = 2

View File

@@ -11,9 +11,9 @@ import (
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/errormessage"
"github.com/up9inc/mizu/cli/kubernetes"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared/logger"
)
func GetApiServerUrl() string {
@@ -46,4 +46,3 @@ func waitForFinish(ctx context.Context, cancel context.CancelFunc) {
cancel()
}
}

View File

@@ -2,13 +2,14 @@ package cmd
import (
"fmt"
"github.com/creasty/defaults"
"github.com/spf13/cobra"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/telemetry"
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared/logger"
)
var configCmd = &cobra.Command{

View File

@@ -2,15 +2,16 @@ package cmd
import (
"context"
"github.com/creasty/defaults"
"github.com/spf13/cobra"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/errormessage"
"github.com/up9inc/mizu/cli/kubernetes"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu/fsUtils"
"github.com/up9inc/mizu/cli/telemetry"
"github.com/up9inc/mizu/shared/logger"
)
var logsCmd = &cobra.Command{

View File

@@ -2,15 +2,16 @@ package cmd
import (
"fmt"
"time"
"github.com/creasty/defaults"
"github.com/spf13/cobra"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/cli/mizu/fsUtils"
"github.com/up9inc/mizu/cli/mizu/version"
"github.com/up9inc/mizu/cli/uiUtils"
"time"
"github.com/up9inc/mizu/shared/logger"
)
var rootCmd = &cobra.Command{
@@ -50,7 +51,7 @@ func Execute() {
if err := fsUtils.EnsureDir(mizu.GetMizuFolderPath()); err != nil {
logger.Log.Errorf("Failed to use mizu folder, %v", err)
}
logger.InitLogger()
logger.InitLogger(fsUtils.GetLogFilePath())
versionChan := make(chan string)
defer printNewVersionIfNeeded(versionChan)

View File

@@ -3,17 +3,18 @@ package cmd
import (
"errors"
"fmt"
"os"
"github.com/creasty/defaults"
"github.com/spf13/cobra"
"github.com/up9inc/mizu/cli/auth"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/errormessage"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/telemetry"
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared"
"os"
"github.com/up9inc/mizu/shared/logger"
)
const uploadTrafficMessageToConfirm = `NOTE: running mizu with --%s flag will upload recorded traffic for further analysis and enriched presentation options.`
@@ -101,4 +102,5 @@ func init() {
tapCmd.Flags().Bool(configStructs.DryRunTapName, defaultTapConfig.DryRun, "Preview of all pods matching the regex, without tapping them")
tapCmd.Flags().StringP(configStructs.WorkspaceTapName, "w", defaultTapConfig.Workspace, "Uploads traffic to your UP9 workspace for further analysis (requires auth)")
tapCmd.Flags().String(configStructs.EnforcePolicyFile, defaultTapConfig.EnforcePolicyFile, "Yaml file path with policy rules")
tapCmd.Flags().String(configStructs.ContractFile, defaultTapConfig.ContractFile, "OAS/Swagger file to validate to monitor the contracts")
}

View File

@@ -3,6 +3,7 @@ package cmd
import (
"context"
"fmt"
"io/ioutil"
"path"
"regexp"
"strings"
@@ -12,12 +13,13 @@ import (
core "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/util/wait"
"github.com/getkin/kin-openapi/openapi3"
"github.com/up9inc/mizu/cli/apiserver"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/errormessage"
"github.com/up9inc/mizu/cli/kubernetes"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/cli/mizu/fsUtils"
"github.com/up9inc/mizu/cli/mizu/goUtils"
@@ -25,6 +27,7 @@ import (
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/debounce"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
@@ -57,6 +60,30 @@ func RunMizuTap() {
}
}
// Read and validate the OAS file
var contract string
if config.Config.Tap.ContractFile != "" {
bytes, err := ioutil.ReadFile(config.Config.Tap.ContractFile)
if err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error reading contract file: %v", errormessage.FormatError(err)))
return
}
contract = string(bytes)
ctx := context.Background()
loader := &openapi3.Loader{Context: ctx}
doc, err := loader.LoadFromData(bytes)
if err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error loading contract file: %v", errormessage.FormatError(err)))
return
}
err = doc.Validate(ctx)
if err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error validating contract file: %v", errormessage.FormatError(err)))
return
}
}
kubernetesProvider, err := kubernetes.NewProvider(config.Config.KubeConfigPath())
if err != nil {
logger.Log.Error(err)
@@ -103,7 +130,7 @@ func RunMizuTap() {
}
defer finishMizuExecution(kubernetesProvider)
if err := createMizuResources(ctx, kubernetesProvider, mizuValidationRules); err != nil {
if err := createMizuResources(ctx, kubernetesProvider, mizuValidationRules, contract); err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Error creating resources: %v", errormessage.FormatError(err)))
return
}
@@ -125,7 +152,7 @@ func readValidationRules(file string) (string, error) {
return string(newContent), nil
}
func createMizuResources(ctx context.Context, kubernetesProvider *kubernetes.Provider, mizuValidationRules string) error {
func createMizuResources(ctx context.Context, kubernetesProvider *kubernetes.Provider, mizuValidationRules string, contract string) error {
if !config.Config.IsNsRestrictedMode() {
if err := createMizuNamespace(ctx, kubernetesProvider); err != nil {
return err
@@ -136,15 +163,15 @@ func createMizuResources(ctx context.Context, kubernetesProvider *kubernetes.Pro
return err
}
if err := createMizuConfigmap(ctx, kubernetesProvider, mizuValidationRules); err != nil {
if err := createMizuConfigmap(ctx, kubernetesProvider, mizuValidationRules, contract); err != nil {
logger.Log.Warningf(uiUtils.Warning, fmt.Sprintf("Failed to create resources required for policy validation. Mizu will not validate policy rules. error: %v\n", errormessage.FormatError(err)))
}
return nil
}
func createMizuConfigmap(ctx context.Context, kubernetesProvider *kubernetes.Provider, data string) error {
err := kubernetesProvider.CreateConfigMap(ctx, config.Config.MizuResourcesNamespace, mizu.ConfigMapName, data)
func createMizuConfigmap(ctx context.Context, kubernetesProvider *kubernetes.Provider, data string, contract string) error {
err := kubernetesProvider.CreateConfigMap(ctx, config.Config.MizuResourcesNamespace, mizu.ConfigMapName, data, contract)
return err
}
@@ -169,15 +196,15 @@ func createMizuApiServer(ctx context.Context, kubernetesProvider *kubernetes.Pro
}
opts := &kubernetes.ApiServerOptions{
Namespace: config.Config.MizuResourcesNamespace,
PodName: mizu.ApiServerPodName,
PodImage: config.Config.AgentImage,
ServiceAccountName: serviceAccountName,
IsNamespaceRestricted: config.Config.IsNsRestrictedMode(),
SyncEntriesConfig: getSyncEntriesConfig(),
MaxEntriesDBSizeBytes: config.Config.Tap.MaxEntriesDBSizeBytes(),
Resources: config.Config.Tap.ApiServerResources,
ImagePullPolicy: config.Config.ImagePullPolicy(),
Namespace: config.Config.MizuResourcesNamespace,
PodName: mizu.ApiServerPodName,
PodImage: config.Config.AgentImage,
ServiceAccountName: serviceAccountName,
IsNamespaceRestricted: config.Config.IsNsRestrictedMode(),
SyncEntriesConfig: getSyncEntriesConfig(),
MaxEntriesDBSizeBytes: config.Config.Tap.MaxEntriesDBSizeBytes(),
Resources: config.Config.Tap.ApiServerResources,
ImagePullPolicy: config.Config.ImagePullPolicy(),
}
_, err = kubernetesProvider.CreateMizuApiServerPod(ctx, opts)
if err != nil {
@@ -295,7 +322,7 @@ func cleanUpMizuResources(ctx context.Context, cancel context.CancelFunc, kubern
}
if len(leftoverResources) > 0 {
errMsg := fmt.Sprintf("Failed to remove the following resources, for more info check logs at %s:", logger.GetLogFilePath())
errMsg := fmt.Sprintf("Failed to remove the following resources, for more info check logs at %s:", fsUtils.GetLogFilePath())
for _, resource := range leftoverResources {
errMsg += "\n- " + resource
}
@@ -562,14 +589,14 @@ func watchApiServerPod(ctx context.Context, kubernetesProvider *kubernetes.Provi
if modifiedPod.Status.Phase == core.PodPending {
if modifiedPod.Status.Conditions[0].Type == core.PodScheduled && modifiedPod.Status.Conditions[0].Status != core.ConditionTrue {
logger.Log.Debugf("Wasn't able to deploy the API server. Reason: \"%s\"", modifiedPod.Status.Conditions[0].Message)
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Wasn't able to deploy the API server, for more info check logs at %s", logger.GetLogFilePath()))
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Wasn't able to deploy the API server, for more info check logs at %s", fsUtils.GetLogFilePath()))
cancel()
break
}
if len(modifiedPod.Status.ContainerStatuses) > 0 && modifiedPod.Status.ContainerStatuses[0].State.Waiting != nil && modifiedPod.Status.ContainerStatuses[0].State.Waiting.Reason == "ErrImagePull" {
logger.Log.Debugf("Wasn't able to deploy the API server. (ErrImagePull) Reason: \"%s\"", modifiedPod.Status.ContainerStatuses[0].State.Waiting.Message)
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Wasn't able to deploy the API server: failed to pull the image, for more info check logs at %v", logger.GetLogFilePath()))
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Wasn't able to deploy the API server: failed to pull the image, for more info check logs at %v", fsUtils.GetLogFilePath()))
cancel()
break
}
@@ -581,7 +608,7 @@ func watchApiServerPod(ctx context.Context, kubernetesProvider *kubernetes.Provi
url := GetApiServerUrl()
if err := apiserver.Provider.InitAndTestConnection(url); err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Couldn't connect to API server, for more info check logs at %s", logger.GetLogFilePath()))
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Couldn't connect to API server, for more info check logs at %s", fsUtils.GetLogFilePath()))
cancel()
break
}

View File

@@ -1,13 +1,14 @@
package cmd
import (
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/telemetry"
"strconv"
"time"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/telemetry"
"github.com/up9inc/mizu/shared/logger"
"github.com/creasty/defaults"
"github.com/spf13/cobra"
"github.com/up9inc/mizu/cli/mizu"

View File

@@ -8,10 +8,11 @@ import (
"github.com/up9inc/mizu/cli/apiserver"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/kubernetes"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/cli/mizu/fsUtils"
"github.com/up9inc/mizu/cli/mizu/version"
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared/logger"
)
func runMizuView() {
@@ -50,7 +51,7 @@ func runMizuView() {
go startProxyReportErrorIfAny(kubernetesProvider, cancel)
if err := apiserver.Provider.InitAndTestConnection(GetApiServerUrl()); err != nil {
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Couldn't connect to API server, for more info check logs at %s", logger.GetLogFilePath()))
logger.Log.Errorf(uiUtils.Error, fmt.Sprintf("Couldn't connect to API server, for more info check logs at %s", fsUtils.GetLogFilePath()))
return
}
}

View File

@@ -3,14 +3,15 @@ package config
import (
"errors"
"fmt"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/shared"
"io/ioutil"
"os"
"reflect"
"strconv"
"strings"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/shared/logger"
"github.com/creasty/defaults"
"github.com/spf13/cobra"
"github.com/spf13/pflag"

View File

@@ -3,8 +3,9 @@ package configStructs
import (
"errors"
"fmt"
"github.com/up9inc/mizu/shared/units"
"regexp"
"github.com/up9inc/mizu/shared/units"
)
const (
@@ -18,6 +19,7 @@ const (
DryRunTapName = "dry-run"
WorkspaceTapName = "workspace"
EnforcePolicyFile = "traffic-validation-file"
ContractFile = "contract"
)
type TapConfig struct {
@@ -34,6 +36,7 @@ type TapConfig struct {
DryRun bool `yaml:"dry-run" default:"false"`
Workspace string `yaml:"workspace"`
EnforcePolicyFile string `yaml:"traffic-validation-file"`
ContractFile string `yaml:"contract"`
ApiServerResources Resources `yaml:"api-server-resources"`
TapperResources Resources `yaml:"tapper-resources"`
}

View File

@@ -5,9 +5,9 @@ go 1.16
require (
github.com/creasty/defaults v1.5.1
github.com/denisbrodbeck/machineid v1.0.1
github.com/getkin/kin-openapi v0.79.0
github.com/google/go-github/v37 v37.0.0
github.com/google/uuid v1.1.2
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
github.com/spf13/cobra v1.1.3
github.com/spf13/pflag v1.0.5
github.com/up9inc/mizu/shared v0.0.0

View File

@@ -113,6 +113,9 @@ github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoD
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
github.com/getkin/kin-openapi v0.79.0 h1:YLZIgIhZLq9z5WFHHIK+oWORRfn6jjwr7qN0xak0xbE=
github.com/getkin/kin-openapi v0.79.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
@@ -140,6 +143,8 @@ github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwds
github.com/go-openapi/jsonpointer v0.18.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M=
github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY=
github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
github.com/go-openapi/jsonreference v0.18.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
@@ -165,6 +170,7 @@ github.com/go-openapi/strfmt v0.19.5/go.mod h1:eftuHTlB/dI8Uq8JJOyRlieZf+WkkxUuk
github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
github.com/go-openapi/swag v0.18.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/swag v0.19.5 h1:lTz6Ys4CmqqCQmZPBlbQENR1/GucA2bzYTE12Pw4tFY=
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/validate v0.18.0/go.mod h1:Uh4HdOzKt19xGIGm1qHf/ofbX1YQ4Y+MYsct2VUrAJ4=
github.com/go-openapi/validate v0.19.2/go.mod h1:1tRCw7m3jtI8eNWEEliiAqUIcBztB2KDnRCRMUi7GTA=
@@ -221,6 +227,7 @@ github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g=
github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
@@ -238,6 +245,7 @@ github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5m
github.com/googleapis/gnostic v0.4.1 h1:DLJCy1n/vrD4HPjOvYcT8aYQXpPIzoRZONaYwyycI+I=
github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
@@ -299,6 +307,7 @@ github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.7.0 h1:aizVhC/NAAcKWb+5QsU1iNOZb4Yws5UO2I+aIprQITM=
github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
github.com/markbates/pkger v0.17.1/go.mod h1:0JoVlrol20BSywW79rN3kdFFsE5xYM+rSCQDXbLhiuI=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
@@ -405,6 +414,7 @@ github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoH
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
@@ -692,6 +702,7 @@ gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -12,10 +12,11 @@ import (
"strconv"
"github.com/up9inc/mizu/cli/config/configStructs"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/shared/logger"
"io"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/shared"
"github.com/up9inc/mizu/tap/api"
@@ -146,15 +147,15 @@ func (provider *Provider) CreateNamespace(ctx context.Context, name string) (*co
}
type ApiServerOptions struct {
Namespace string
PodName string
PodImage string
ServiceAccountName string
IsNamespaceRestricted bool
SyncEntriesConfig *shared.SyncEntriesConfig
MaxEntriesDBSizeBytes int64
Resources configStructs.Resources
ImagePullPolicy core.PullPolicy
Namespace string
PodName string
PodImage string
ServiceAccountName string
IsNamespaceRestricted bool
SyncEntriesConfig *shared.SyncEntriesConfig
MaxEntriesDBSizeBytes int64
Resources configStructs.Resources
ImagePullPolicy core.PullPolicy
}
func (provider *Provider) CreateMizuApiServerPod(ctx context.Context, opts *ApiServerOptions) (*core.Pod, error) {
@@ -193,6 +194,13 @@ func (provider *Provider) CreateMizuApiServerPod(ctx context.Context, opts *ApiS
command = append(command, "--namespace", opts.Namespace)
}
port := intstr.FromInt(shared.DefaultApiServerPort)
debugMode := ""
if config.Config.DumpLogs {
debugMode = "1"
}
pod := &core.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: opts.PodName,
@@ -221,6 +229,10 @@ func (provider *Provider) CreateMizuApiServerPod(ctx context.Context, opts *ApiS
Name: shared.MaxEntriesDBSizeBytesEnvVar,
Value: strconv.FormatInt(opts.MaxEntriesDBSizeBytes, 10),
},
{
Name: shared.DebugModeEnvVar,
Value: debugMode,
},
},
Resources: core.ResourceRequirements{
Limits: core.ResourceList{
@@ -232,6 +244,25 @@ func (provider *Provider) CreateMizuApiServerPod(ctx context.Context, opts *ApiS
"memory": memRequests,
},
},
ReadinessProbe: &core.Probe{
Handler: core.Handler{
TCPSocket: &core.TCPSocketAction{
Port: port,
},
},
InitialDelaySeconds: 5,
PeriodSeconds: 10,
},
LivenessProbe: &core.Probe{
Handler: core.Handler{
HTTPGet: &core.HTTPGetAction{
Path: "/echo",
Port: port,
},
},
InitialDelaySeconds: 5,
PeriodSeconds: 10,
},
},
},
Volumes: []core.Volume{
@@ -260,7 +291,7 @@ func (provider *Provider) CreateService(ctx context.Context, namespace string, s
Namespace: namespace,
},
Spec: core.ServiceSpec{
Ports: []core.ServicePort{{TargetPort: intstr.FromInt(8899), Port: 80}},
Ports: []core.ServicePort{{TargetPort: intstr.FromInt(shared.DefaultApiServerPort), Port: 80}},
Type: core.ServiceTypeClusterIP,
Selector: map[string]string{"app": appLabelValue},
},
@@ -454,13 +485,14 @@ func (provider *Provider) handleRemovalError(err error) error {
return err
}
func (provider *Provider) CreateConfigMap(ctx context.Context, namespace string, configMapName string, data string) error {
if data == "" {
func (provider *Provider) CreateConfigMap(ctx context.Context, namespace string, configMapName string, data string, contract string) error {
if data == "" && contract == "" {
return nil
}
configMapData := make(map[string]string, 0)
configMapData[shared.RulePolicyFileName] = data
configMapData[shared.ContractFileName] = contract
configMap := &core.ConfigMap{
TypeMeta: metav1.TypeMeta{
Kind: "ConfigMap",
@@ -503,6 +535,11 @@ func (provider *Provider) ApplyMizuTapperDaemonSet(ctx context.Context, namespac
"--nodefrag",
}
debugMode := ""
if config.Config.DumpLogs {
debugMode = "1"
}
agentContainer := applyconfcore.Container()
agentContainer.WithName(tapperPodName)
agentContainer.WithImage(podImage)
@@ -510,6 +547,7 @@ func (provider *Provider) ApplyMizuTapperDaemonSet(ctx context.Context, namespac
agentContainer.WithSecurityContext(applyconfcore.SecurityContext().WithPrivileged(true))
agentContainer.WithCommand(mizuCmd...)
agentContainer.WithEnv(
applyconfcore.EnvVar().WithName(shared.DebugModeEnvVar).WithValue(debugMode),
applyconfcore.EnvVar().WithName(shared.HostModeEnvVar).WithValue("1"),
applyconfcore.EnvVar().WithName(shared.TappedAddressesPerNodeDictEnvVar).WithValue(string(nodeToTappedPodIPMapJsonStr)),
applyconfcore.EnvVar().WithName(shared.GoGCEnvVar).WithValue("12800"),

View File

@@ -2,12 +2,13 @@ package kubernetes
import (
"fmt"
"github.com/up9inc/mizu/cli/logger"
"k8s.io/kubectl/pkg/proxy"
"net"
"net/http"
"strings"
"time"
"github.com/up9inc/mizu/shared/logger"
"k8s.io/kubectl/pkg/proxy"
)
const k8sProxyApiPrefix = "/"

View File

@@ -3,10 +3,15 @@ package kubernetes
import (
"context"
"errors"
"fmt"
"github.com/up9inc/mizu/shared/debounce"
"github.com/up9inc/mizu/shared/logger"
"regexp"
"sync"
"time"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/watch"
)
@@ -16,6 +21,7 @@ func FilteredWatch(ctx context.Context, kubernetesProvider *Provider, targetName
removedChan := make(chan *corev1.Pod)
errorChan := make(chan error)
var wg sync.WaitGroup
for _, targetNamespace := range targetNamespaces {
@@ -23,36 +29,33 @@ func FilteredWatch(ctx context.Context, kubernetesProvider *Provider, targetName
go func(targetNamespace string) {
defer wg.Done()
watcher := kubernetesProvider.GetPodWatcher(ctx, targetNamespace)
watchRestartDebouncer := debounce.NewDebouncer(1 * time.Minute, func() {})
for {
watcher := kubernetesProvider.GetPodWatcher(ctx, targetNamespace)
err := startWatchLoop(ctx, watcher, podFilter, addedChan, modifiedChan, removedChan) // blocking
watcher.Stop()
select {
case e := <-watcher.ResultChan():
if e.Object == nil {
errorChan <- errors.New("kubernetes pod watch failed")
return
}
pod, ok := e.Object.(*corev1.Pod)
if !ok {
continue
}
if !podFilter.MatchString(pod.Name) {
continue
}
switch e.Type {
case watch.Added:
addedChan <- pod
case watch.Modified:
modifiedChan <- pod
case watch.Deleted:
removedChan <- pod
}
case <-ctx.Done():
watcher.Stop()
case <- ctx.Done():
return
default:
break
}
if err != nil {
errorChan <- fmt.Errorf("error in k8 watch: %v", err)
break
} else {
if !watchRestartDebouncer.IsOn() {
watchRestartDebouncer.SetOn()
logger.Log.Debug("k8s watch channel closed, restarting watcher")
time.Sleep(time.Second * 5)
continue
} else {
errorChan <- errors.New("k8s watch unstable, closes frequently")
break
}
}
}
}(targetNamespace)
@@ -69,3 +72,39 @@ func FilteredWatch(ctx context.Context, kubernetesProvider *Provider, targetName
return addedChan, modifiedChan, removedChan, errorChan
}
func startWatchLoop(ctx context.Context, watcher watch.Interface, podFilter *regexp.Regexp, addedChan chan *corev1.Pod, modifiedChan chan *corev1.Pod, removedChan chan *corev1.Pod) error {
resultChan := watcher.ResultChan()
for {
select {
case e, isChannelOpen := <-resultChan:
if !isChannelOpen {
return nil
}
if e.Type == watch.Error {
return apierrors.FromObject(e.Object)
}
pod, ok := e.Object.(*corev1.Pod)
if !ok {
continue
}
if !podFilter.MatchString(pod.Name) {
continue
}
switch e.Type {
case watch.Added:
addedChan <- pod
case watch.Modified:
modifiedChan <- pod
case watch.Deleted:
removedChan <- pod
}
case <-ctx.Done():
return nil
}
}
}

View File

@@ -4,14 +4,20 @@ import (
"archive/zip"
"context"
"fmt"
"os"
"path"
"regexp"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/kubernetes"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"os"
"regexp"
"github.com/up9inc/mizu/shared/logger"
)
func GetLogFilePath() string {
return path.Join(mizu.GetMizuFolderPath(), "mizu_cli.log")
}
func DumpLogs(ctx context.Context, provider *kubernetes.Provider, filePath string) error {
podExactRegex := regexp.MustCompile("^" + mizu.MizuResourcesPrefix)
pods, err := provider.ListAllPodsMatchingRegex(ctx, podExactRegex, []string{config.Config.MizuResourcesNamespace})
@@ -66,10 +72,10 @@ func DumpLogs(ctx context.Context, provider *kubernetes.Provider, filePath strin
logger.Log.Debugf("Successfully added file %s", config.Config.ConfigFilePath)
}
if err := AddFileToZip(zipWriter, logger.GetLogFilePath()); err != nil {
if err := AddFileToZip(zipWriter, GetLogFilePath()); err != nil {
logger.Log.Debugf("Failed write file, %v", err)
} else {
logger.Log.Debugf("Successfully added file %s", logger.GetLogFilePath())
logger.Log.Debugf("Successfully added file %s", GetLogFilePath())
}
logger.Log.Infof("You can find the zip file with all logs in %s\n", filePath)

View File

@@ -3,11 +3,12 @@ package fsUtils
import (
"archive/zip"
"fmt"
"github.com/up9inc/mizu/cli/logger"
"io"
"os"
"path/filepath"
"strings"
"github.com/up9inc/mizu/shared/logger"
)
func AddFileToZip(zipWriter *zip.Writer, filename string) error {

View File

@@ -1,9 +1,10 @@
package goUtils
import (
"github.com/up9inc/mizu/cli/logger"
"reflect"
"runtime/debug"
"github.com/up9inc/mizu/shared/logger"
)
func HandleExcWrapper(fn interface{}, params ...interface{}) (result []reflect.Value) {

View File

@@ -3,15 +3,16 @@ package version
import (
"context"
"fmt"
"github.com/up9inc/mizu/cli/apiserver"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"io/ioutil"
"net/http"
"runtime"
"strings"
"time"
"github.com/up9inc/mizu/cli/apiserver"
"github.com/up9inc/mizu/cli/mizu"
"github.com/up9inc/mizu/shared/logger"
"github.com/google/go-github/v37/github"
"github.com/up9inc/mizu/cli/uiUtils"
"github.com/up9inc/mizu/shared/semver"

View File

@@ -4,12 +4,13 @@ import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"github.com/denisbrodbeck/machineid"
"github.com/up9inc/mizu/cli/apiserver"
"github.com/up9inc/mizu/cli/config"
"github.com/up9inc/mizu/cli/logger"
"github.com/up9inc/mizu/cli/mizu"
"net/http"
"github.com/up9inc/mizu/shared/logger"
)
const telemetryUrl = "https://us-east4-up9-prod.cloudfunctions.net/mizu-telemetry"

View File

@@ -3,9 +3,10 @@ package uiUtils
import (
"bufio"
"fmt"
"log"
"os"
"strings"
"github.com/up9inc/mizu/shared/logger"
)
func AskForConfirmation(s string) bool {
@@ -15,7 +16,7 @@ func AskForConfirmation(s string) bool {
response, err := reader.ReadString('\n')
if err != nil {
log.Fatal(err)
logger.Log.Fatalf("Error while reading confirmation string, err: %v", err)
}
response = strings.ToLower(strings.TrimSpace(response))
if response == "" || response == "y" || response == "yes" {

View File

@@ -2,9 +2,10 @@ package uiUtils
import (
"fmt"
"github.com/up9inc/mizu/cli/logger"
"os/exec"
"runtime"
"github.com/up9inc/mizu/shared/logger"
)
func OpenBrowser(url string) {

172
docs/CONTRACT_MONITORING.md Normal file
View File

@@ -0,0 +1,172 @@
# OpenAPI Specification (OAS) Contract Monitoring
An OAS/Swagger file can contain schemas under `parameters` and `responses` fields. With `--contract catalogue.yaml`
CLI option, you can pass your API description to Mizu and the traffic will automatically be validated
against the contracts.
Below is an example of an OAS/Swagger file from [Sock Shop](https://microservices-demo.github.io/) microservice demo
that contains a bunch contracts:
```yaml
openapi: 3.0.1
info:
title: Catalogue resources
version: 1.0.0
description: ""
license:
name: MIT
url: http://github.com/gruntjs/grunt/blob/master/LICENSE-MIT
paths:
/catalogue:
get:
description: Catalogue API
operationId: List catalogue
responses:
200:
description: ""
content:
application/json;charset=UTF-8:
schema:
type: array
items:
$ref: '#/components/schemas/Listresponse'
/catalogue/{id}:
get:
operationId: Get an item
parameters:
- name: id
in: path
required: true
schema:
type: string
example: a0a4f044-b040-410d-8ead-4de0446aec7e
responses:
200:
description: ""
content:
application/json; charset=UTF-8:
schema:
$ref: '#/components/schemas/Getanitemresponse'
/catalogue/size:
get:
operationId: Get size
responses:
200:
description: ""
content:
application/json;charset=UTF-8:
schema:
$ref: '#/components/schemas/Getsizeresponse'
/tags:
get:
operationId: List_
responses:
200:
description: ""
content:
application/json;charset=UTF-8:
schema:
$ref: '#/components/schemas/Listresponse3'
components:
schemas:
Listresponse:
title: List response
required:
- count
- description
- id
- imageUrl
- name
- price
- tag
type: object
properties:
id:
type: string
name:
type: string
description:
type: string
imageUrl:
type: array
items:
type: string
price:
type: number
format: double
count:
type: integer
format: int32
tag:
type: array
items:
type: string
Getanitemresponse:
title: Get an item response
required:
- count
- description
- id
- imageUrl
- name
- price
- tag
type: object
properties:
id:
type: string
name:
type: string
description:
type: string
imageUrl:
type: array
items:
type: string
price:
type: number
format: double
count:
type: integer
format: int32
tag:
type: array
items:
type: string
Getsizeresponse:
title: Get size response
required:
- size
type: object
properties:
size:
type: integer
format: int32
Listresponse3:
title: List response3
required:
- tags
type: object
properties:
tags:
type: array
items:
type: string
```
Pass it to Mizu through the CLI option: `mizu tap -n sock-shop --contract catalogue.yaml`
Now Mizu will monitor the traffic against these contracts.
If an entry fails to comply with the contract, it's marked with `Breach` notice in the UI.
The reason of the failure can be seen under the `CONTRACT` tab in the details layout.
### Notes
Make sure that you;
- specified the `openapi` version
- specified the `info.version` version in the YAML
- and removed `servers` field from the YAML
Otherwise the OAS file cannot be recognized. (see [this issue](https://github.com/getkin/kin-openapi/issues/356))

View File

@@ -9,5 +9,8 @@ const (
MaxEntriesDBSizeBytesEnvVar = "MAX_ENTRIES_DB_BYTES"
RulePolicyPath = "/app/enforce-policy/"
RulePolicyFileName = "enforce-policy.yaml"
ContractFileName = "contract-oas.yaml"
GoGCEnvVar = "GOGC"
DefaultApiServerPort = 8899
DebugModeEnvVar = "MIZU_DEBUG"
)

View File

@@ -52,3 +52,7 @@ func (d *Debouncer) SetOn() error {
d.timer = time.AfterFunc(d.timeout, d.callback)
return nil
}
func (d *Debouncer) IsOn() bool {
return d.running
}

View File

@@ -5,5 +5,6 @@ go 1.16
require (
github.com/docker/go-units v0.4.0
github.com/golang-jwt/jwt/v4 v4.1.0
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
)

View File

@@ -2,6 +2,8 @@ github.com/docker/go-units v0.4.0 h1:3uh0PgVws3nIA0Q+MwDC8yjEPf9zjRfZZWXZYDct3Tw
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/golang-jwt/jwt/v4 v4.1.0 h1:XUgk2Ex5veyVFVeLm0xhusUTQybEbexJXrvPNOKkSY0=
github.com/golang-jwt/jwt/v4 v4.1.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 h1:lDH9UUVJtmYCjyT0CI4q8xvlXPxeZ0gYCVvWbmPlp88=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=

View File

@@ -1,24 +1,18 @@
package logger
import (
"github.com/op/go-logging"
"github.com/up9inc/mizu/cli/mizu"
"os"
"path"
"github.com/op/go-logging"
)
var Log = logging.MustGetLogger("mizu_cli")
var Log = logging.MustGetLogger("mizu")
var format = logging.MustStringFormatter(
`%{time} %{level:.5s} ▶ %{pid} %{shortfile} %{shortfunc} ▶ %{message}`,
)
func GetLogFilePath() string {
return path.Join(mizu.GetMizuFolderPath(), "mizu_cli.log")
}
func InitLogger() {
logPath := GetLogFilePath()
func InitLogger(logPath string) {
f, err := os.OpenFile(logPath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
Log.Infof("Failed to open mizu log file: %v, err %v", logPath, err)
@@ -33,7 +27,12 @@ func InitLogger() {
backend1Leveled.SetLevel(logging.INFO, "")
logging.SetBackend(backend1Leveled, backend2Formatter)
Log.Debugf("\n\n\n")
Log.Debugf("Running mizu version %v", mizu.SemVer)
}
func InitLoggerStderrOnly(level logging.Level) {
backend := logging.NewLogBackend(os.Stderr, "", 0)
backendFormatter := logging.NewBackendFormatter(backend, format)
logging.SetBackend(backendFormatter)
logging.SetLevel(level, "")
}

View File

@@ -2,9 +2,17 @@ package api
import (
"bufio"
"bytes"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net/http"
"plugin"
"sync"
"time"
"github.com/google/martian/har"
)
type Protocol struct {
@@ -104,32 +112,36 @@ type MizuEntry struct {
ID uint `gorm:"primarykey"`
CreatedAt time.Time
UpdatedAt time.Time
ProtocolName string `json:"protocolName" gorm:"column:protocolName"`
ProtocolLongName string `json:"protocolLongName" gorm:"column:protocolLongName"`
ProtocolAbbreviation string `json:"protocolAbbreviation" gorm:"column:protocolAbbreviation"`
ProtocolVersion string `json:"protocolVersion" gorm:"column:protocolVersion"`
ProtocolBackgroundColor string `json:"protocolBackgroundColor" gorm:"column:protocolBackgroundColor"`
ProtocolForegroundColor string `json:"protocolForegroundColor" gorm:"column:protocolForegroundColor"`
ProtocolFontSize int8 `json:"protocolFontSize" gorm:"column:protocolFontSize"`
ProtocolReferenceLink string `json:"protocolReferenceLink" gorm:"column:protocolReferenceLink"`
Entry string `json:"entry,omitempty" gorm:"column:entry"`
EntryId string `json:"entryId" gorm:"column:entryId"`
Url string `json:"url" gorm:"column:url"`
Method string `json:"method" gorm:"column:method"`
Status int `json:"status" gorm:"column:status"`
RequestSenderIp string `json:"requestSenderIp" gorm:"column:requestSenderIp"`
Service string `json:"service" gorm:"column:service"`
Timestamp int64 `json:"timestamp" gorm:"column:timestamp"`
ElapsedTime int64 `json:"elapsedTime" gorm:"column:elapsedTime"`
Path string `json:"path" gorm:"column:path"`
ResolvedSource string `json:"resolvedSource,omitempty" gorm:"column:resolvedSource"`
ResolvedDestination string `json:"resolvedDestination,omitempty" gorm:"column:resolvedDestination"`
SourceIp string `json:"sourceIp,omitempty" gorm:"column:sourceIp"`
DestinationIp string `json:"destinationIp,omitempty" gorm:"column:destinationIp"`
SourcePort string `json:"sourcePort,omitempty" gorm:"column:sourcePort"`
DestinationPort string `json:"destinationPort,omitempty" gorm:"column:destinationPort"`
IsOutgoing bool `json:"isOutgoing,omitempty" gorm:"column:isOutgoing"`
EstimatedSizeBytes int `json:"-" gorm:"column:estimatedSizeBytes"`
ProtocolName string `json:"protocolName" gorm:"column:protocolName"`
ProtocolLongName string `json:"protocolLongName" gorm:"column:protocolLongName"`
ProtocolAbbreviation string `json:"protocolAbbreviation" gorm:"column:protocolAbbreviation"`
ProtocolVersion string `json:"protocolVersion" gorm:"column:protocolVersion"`
ProtocolBackgroundColor string `json:"protocolBackgroundColor" gorm:"column:protocolBackgroundColor"`
ProtocolForegroundColor string `json:"protocolForegroundColor" gorm:"column:protocolForegroundColor"`
ProtocolFontSize int8 `json:"protocolFontSize" gorm:"column:protocolFontSize"`
ProtocolReferenceLink string `json:"protocolReferenceLink" gorm:"column:protocolReferenceLink"`
Entry string `json:"entry,omitempty" gorm:"column:entry"`
EntryId string `json:"entryId" gorm:"column:entryId"`
Url string `json:"url" gorm:"column:url"`
Method string `json:"method" gorm:"column:method"`
Status int `json:"status" gorm:"column:status"`
RequestSenderIp string `json:"requestSenderIp" gorm:"column:requestSenderIp"`
Service string `json:"service" gorm:"column:service"`
Timestamp int64 `json:"timestamp" gorm:"column:timestamp"`
ElapsedTime int64 `json:"elapsedTime" gorm:"column:elapsedTime"`
Path string `json:"path" gorm:"column:path"`
ResolvedSource string `json:"resolvedSource,omitempty" gorm:"column:resolvedSource"`
ResolvedDestination string `json:"resolvedDestination,omitempty" gorm:"column:resolvedDestination"`
SourceIp string `json:"sourceIp,omitempty" gorm:"column:sourceIp"`
DestinationIp string `json:"destinationIp,omitempty" gorm:"column:destinationIp"`
SourcePort string `json:"sourcePort,omitempty" gorm:"column:sourcePort"`
DestinationPort string `json:"destinationPort,omitempty" gorm:"column:destinationPort"`
IsOutgoing bool `json:"isOutgoing,omitempty" gorm:"column:isOutgoing"`
ContractStatus ContractStatus `json:"contractStatus,omitempty" gorm:"column:contractStatus"`
ContractRequestReason string `json:"contractRequestReason,omitempty" gorm:"column:contractRequestReason"`
ContractResponseReason string `json:"contractResponseReason,omitempty" gorm:"column:contractResponseReason"`
ContractContent string `json:"contractContent,omitempty" gorm:"column:contractContent"`
EstimatedSizeBytes int `json:"-" gorm:"column:estimatedSizeBytes"`
}
type MizuEntryWrapper struct {
@@ -159,6 +171,7 @@ type BaseEntryDetails struct {
IsOutgoing bool `json:"isOutgoing,omitempty"`
Latency int64 `json:"latency"`
Rules ApplicableRules `json:"rules,omitempty"`
ContractStatus ContractStatus `json:"contractStatus"`
}
type ApplicableRules struct {
@@ -167,6 +180,15 @@ type ApplicableRules struct {
NumberOfRules int `json:"numberOfRules,omitempty"`
}
type ContractStatus int
type Contract struct {
Status ContractStatus `json:"status"`
RequestReason string `json:"requestReason"`
ResponseReason string `json:"responseReason"`
Content string `json:"content"`
}
type DataUnmarshaler interface {
UnmarshalData(*MizuEntry) error
}
@@ -184,14 +206,20 @@ func (bed *BaseEntryDetails) UnmarshalData(entry *MizuEntry) error {
}
bed.Id = entry.EntryId
bed.Url = entry.Url
bed.RequestSenderIp = entry.RequestSenderIp
bed.Service = entry.Service
bed.Path = entry.Path
bed.Summary = entry.Path
bed.StatusCode = entry.Status
bed.Method = entry.Method
bed.Timestamp = entry.Timestamp
bed.RequestSenderIp = entry.RequestSenderIp
bed.SourceIp = entry.SourceIp
bed.DestinationIp = entry.DestinationIp
bed.SourcePort = entry.SourcePort
bed.DestinationPort = entry.DestinationPort
bed.IsOutgoing = entry.IsOutgoing
bed.Latency = entry.ElapsedTime
bed.ContractStatus = entry.ContractStatus
return nil
}
@@ -199,3 +227,109 @@ const (
TABLE string = "table"
BODY string = "body"
)
const (
TypeHttpRequest = iota
TypeHttpResponse
)
type HTTPPayload struct {
Type uint8
Data interface{}
}
type HTTPPayloader interface {
MarshalJSON() ([]byte, error)
}
type HTTPWrapper struct {
Method string `json:"method"`
Url string `json:"url"`
Details interface{} `json:"details"`
RawRequest *HTTPRequestWrapper `json:"rawRequest"`
RawResponse *HTTPResponseWrapper `json:"rawResponse"`
}
func (h HTTPPayload) MarshalJSON() ([]byte, error) {
switch h.Type {
case TypeHttpRequest:
harRequest, err := har.NewRequest(h.Data.(*http.Request), true)
if err != nil {
return nil, errors.New("Failed converting request to HAR")
}
return json.Marshal(&HTTPWrapper{
Method: harRequest.Method,
Url: "",
Details: harRequest,
RawRequest: &HTTPRequestWrapper{Request: h.Data.(*http.Request)},
})
case TypeHttpResponse:
harResponse, err := har.NewResponse(h.Data.(*http.Response), true)
if err != nil {
return nil, errors.New("Failed converting response to HAR")
}
return json.Marshal(&HTTPWrapper{
Method: "",
Url: "",
Details: harResponse,
RawResponse: &HTTPResponseWrapper{Response: h.Data.(*http.Response)},
})
default:
panic(fmt.Sprintf("HTTP payload cannot be marshaled: %s\n", h.Type))
}
}
type HTTPWrapperTricky struct {
Method string `json:"method"`
Url string `json:"url"`
Details interface{} `json:"details"`
RawRequest *http.Request `json:"rawRequest"`
RawResponse *http.Response `json:"rawResponse"`
}
type HTTPMessage struct {
IsRequest bool `json:"isRequest"`
CaptureTime time.Time `json:"captureTime"`
Payload HTTPWrapperTricky `json:"payload"`
}
type HTTPRequestResponsePair struct {
Request HTTPMessage `json:"request"`
Response HTTPMessage `json:"response"`
}
type HTTPRequestWrapper struct {
*http.Request
}
func (r *HTTPRequestWrapper) MarshalJSON() ([]byte, error) {
body, _ := ioutil.ReadAll(r.Request.Body)
r.Request.Body = ioutil.NopCloser(bytes.NewBuffer(body))
return json.Marshal(&struct {
Body string `json:"Body,omitempty"`
GetBody string `json:"GetBody,omitempty"`
Cancel string `json:"Cancel,omitempty"`
*http.Request
}{
Body: string(body),
Request: r.Request,
})
}
type HTTPResponseWrapper struct {
*http.Response
}
func (r *HTTPResponseWrapper) MarshalJSON() ([]byte, error) {
body, _ := ioutil.ReadAll(r.Response.Body)
r.Response.Body = ioutil.NopCloser(bytes.NewBuffer(body))
return json.Marshal(&struct {
Body string `json:"Body,omitempty"`
GetBody string `json:"GetBody,omitempty"`
Cancel string `json:"Cancel,omitempty"`
*http.Response
}{
Body: string(body),
Response: r.Response,
})
}

View File

@@ -1,3 +1,5 @@
module github.com/up9inc/mizu/tap/api
go 1.16
require github.com/google/martian v2.1.0+incompatible

2
tap/api/go.sum Normal file
View File

@@ -0,0 +1,2 @@
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=

View File

@@ -5,7 +5,7 @@ import (
"time"
"github.com/google/gopacket/reassembly"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
@@ -28,7 +28,7 @@ func (cl *Cleaner) clean() {
startCleanTime := time.Now()
cl.assemblerMutex.Lock()
rlog.Debugf("Assembler Stats before cleaning %s", cl.assembler.Dump())
logger.Log.Debugf("Assembler Stats before cleaning %s", cl.assembler.Dump())
flushed, closed := cl.assembler.FlushCloseOlderThan(startCleanTime.Add(-cl.connectionTimeout))
cl.assemblerMutex.Unlock()
@@ -38,7 +38,7 @@ func (cl *Cleaner) clean() {
}
cl.statsMutex.Lock()
rlog.Debugf("Assembler Stats after cleaning %s", cl.assembler.Dump())
logger.Log.Debugf("Assembler Stats after cleaning %s", cl.assembler.Dump())
cl.stats.flushed += flushed
cl.stats.closed += closed
cl.statsMutex.Unlock()

View File

@@ -2,8 +2,6 @@ module github.com/up9inc/mizu/tap/extensions/amqp
go 1.16
require (
github.com/up9inc/mizu/tap/api v0.0.0
)
require github.com/up9inc/mizu/tap/api v0.0.0
replace github.com/up9inc/mizu/tap/api v0.0.0 => ../../api

View File

@@ -0,0 +1,2 @@
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=

View File

@@ -3,9 +3,7 @@ module github.com/up9inc/mizu/tap/extensions/http
go 1.16
require (
github.com/beevik/etree v1.1.0 // indirect
github.com/google/martian v2.1.0+incompatible
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7
github.com/beevik/etree v1.1.0
github.com/up9inc/mizu/tap/api v0.0.0
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
golang.org/x/text v0.3.5 // indirect

View File

@@ -2,8 +2,6 @@ github.com/beevik/etree v1.1.0 h1:T0xke/WvNtMoCqgzPhkX2r4rjY3GDZFi+FjpRZY2Jbs=
github.com/beevik/etree v1.1.0/go.mod h1:r8Aw8JqVegEf0w2fDnATrX9VpkMcyFeM0FhwO62wh+A=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7 h1:jkvpcEatpwuMF5O5LVxTnehj6YZ/aEZN4NWD/Xml4pI=
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7/go.mod h1:KTrHyWpO1sevuXPZwyeZc72ddWRFqNSKDFl7uVWKpg0=
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7 h1:OgUuv8lsRpBibGNbSizVwKWlysjaNzmC9gYMhPVfqFM=
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=

View File

@@ -8,8 +8,6 @@ import (
"io/ioutil"
"net/http"
"github.com/romana/rlog"
"github.com/up9inc/mizu/tap/api"
)
@@ -85,22 +83,12 @@ func handleHTTP2Stream(grpcAssembler *GrpcAssembler, tcpID *api.TcpID, superTime
func handleHTTP1ClientStream(b *bufio.Reader, tcpID *api.TcpID, counterPair *api.CounterPair, superTimer *api.SuperTimer, emitter api.Emitter, options *api.TrafficFilteringOptions) error {
req, err := http.ReadRequest(b)
if err != nil {
// log.Println("Error reading stream:", err)
return err
}
counterPair.Request++
body, err := ioutil.ReadAll(req.Body)
req.Body = io.NopCloser(bytes.NewBuffer(body)) // rewind
s := len(body)
if err != nil {
rlog.Debugf("[HTTP-request-body] stream %s Got body err: %s", tcpID.Ident, err)
}
if err := req.Body.Close(); err != nil {
rlog.Debugf("[HTTP-request-body-close] stream %s Failed to close request body: %s", tcpID.Ident, err)
}
encoding := req.Header["Content-Encoding"]
rlog.Tracef(1, "HTTP/1 Request: %s %s %s (Body:%d) -> %s", tcpID.Ident, req.Method, req.URL, s, encoding)
ident := fmt.Sprintf(
"%s->%s %s->%s %d",
@@ -127,30 +115,12 @@ func handleHTTP1ClientStream(b *bufio.Reader, tcpID *api.TcpID, counterPair *api
func handleHTTP1ServerStream(b *bufio.Reader, tcpID *api.TcpID, counterPair *api.CounterPair, superTimer *api.SuperTimer, emitter api.Emitter, options *api.TrafficFilteringOptions) error {
res, err := http.ReadResponse(b, nil)
if err != nil {
// log.Println("Error reading stream:", err)
return err
}
counterPair.Response++
body, err := ioutil.ReadAll(res.Body)
res.Body = io.NopCloser(bytes.NewBuffer(body)) // rewind
s := len(body)
if err != nil {
rlog.Debugf("[HTTP-response-body] HTTP/%s: failed to get body(parsed len:%d): %s", tcpID.Ident, s, err)
}
if err := res.Body.Close(); err != nil {
rlog.Debugf("[HTTP-response-body-close] HTTP/%s: failed to close body(parsed len:%d): %s", tcpID.Ident, s, err)
}
sym := ","
if res.ContentLength > 0 && res.ContentLength != int64(s) {
sym = "!="
}
contentType, ok := res.Header["Content-Type"]
if !ok {
contentType = []string{http.DetectContentType(body)}
}
encoding := res.Header["Content-Encoding"]
rlog.Tracef(1, "HTTP/1 Response: %s %s (%d%s%d%s) -> %s", tcpID.Ident, res.Status, res.ContentLength, sym, s, contentType, encoding)
ident := fmt.Sprintf(
"%s->%s %s->%s %d",

View File

@@ -10,8 +10,6 @@ import (
"net/url"
"time"
"github.com/romana/rlog"
"github.com/up9inc/mizu/tap/api"
)
@@ -62,19 +60,11 @@ func (d dissecting) Ping() {
}
func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, counterPair *api.CounterPair, superTimer *api.SuperTimer, superIdentifier *api.SuperIdentifier, emitter api.Emitter, options *api.TrafficFilteringOptions) error {
ident := fmt.Sprintf("%s->%s:%s->%s", tcpID.SrcIP, tcpID.DstIP, tcpID.SrcPort, tcpID.DstPort)
isHTTP2, err := checkIsHTTP2Connection(b, isClient)
if err != nil {
rlog.Debugf("[HTTP/2-Prepare-Connection] stream %s Failed to check if client is HTTP/2: %s (%v,%+v)", ident, err, err, err)
// Do something?
}
var grpcAssembler *GrpcAssembler
if isHTTP2 {
err := prepareHTTP2Connection(b, isClient)
if err != nil {
rlog.Debugf("[HTTP/2-Prepare-Connection-After-Check] stream %s error: %s (%v,%+v)", ident, err, err, err)
}
prepareHTTP2Connection(b, isClient)
grpcAssembler = createGrpcAssembler(b)
}
@@ -89,7 +79,6 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
if err == io.EOF || err == io.ErrUnexpectedEOF {
break
} else if err != nil {
rlog.Debugf("[HTTP/2] stream %s error: %s (%v,%+v)", ident, err, err, err)
continue
}
dissected = true
@@ -98,7 +87,6 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
if err == io.EOF || err == io.ErrUnexpectedEOF {
break
} else if err != nil {
rlog.Debugf("[HTTP-request] stream %s Request error: %s (%v,%+v)", ident, err, err, err)
continue
}
dissected = true
@@ -107,7 +95,6 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
if err == io.EOF || err == io.ErrUnexpectedEOF {
break
} else if err != nil {
rlog.Debugf("[HTTP-response], stream %s Response error: %s (%v,%+v)", ident, err, err, err)
continue
}
dissected = true
@@ -124,7 +111,6 @@ func (d dissecting) Dissect(b *bufio.Reader, isClient bool, tcpID *api.TcpID, co
func SetHostname(address, newHostname string) string {
replacedUrl, err := url.Parse(address)
if err != nil {
log.Printf("error replacing hostname to %s in address %s, returning original %v", newHostname, address, err)
return address
}
replacedUrl.Host = newHostname

View File

@@ -7,8 +7,6 @@ import (
"sync"
"time"
"github.com/romana/rlog"
"github.com/up9inc/mizu/tap/api"
)
@@ -31,7 +29,7 @@ func (matcher *requestResponseMatcher) registerRequest(ident string, request *ht
requestHTTPMessage := api.GenericMessage{
IsRequest: true,
CaptureTime: captureTime,
Payload: HTTPPayload{
Payload: api.HTTPPayload{
Type: TypeHttpRequest,
Data: request,
},
@@ -41,15 +39,12 @@ func (matcher *requestResponseMatcher) registerRequest(ident string, request *ht
// Type assertion always succeeds because all of the map's values are of api.GenericMessage type
responseHTTPMessage := response.(*api.GenericMessage)
if responseHTTPMessage.IsRequest {
rlog.Debugf("[Request-Duplicate] Got duplicate request with same identifier")
return nil
}
rlog.Tracef(1, "Matched open Response for %s", key)
return matcher.preparePair(&requestHTTPMessage, responseHTTPMessage)
}
matcher.openMessagesMap.Store(key, &requestHTTPMessage)
rlog.Tracef(1, "Registered open Request for %s", key)
return nil
}
@@ -60,7 +55,7 @@ func (matcher *requestResponseMatcher) registerResponse(ident string, response *
responseHTTPMessage := api.GenericMessage{
IsRequest: false,
CaptureTime: captureTime,
Payload: HTTPPayload{
Payload: api.HTTPPayload{
Type: TypeHttpResponse,
Data: response,
},
@@ -70,15 +65,12 @@ func (matcher *requestResponseMatcher) registerResponse(ident string, response *
// Type assertion always succeeds because all of the map's values are of api.GenericMessage type
requestHTTPMessage := request.(*api.GenericMessage)
if !requestHTTPMessage.IsRequest {
rlog.Debugf("[Response-Duplicate] Got duplicate response with same identifier")
return nil
}
rlog.Tracef(1, "Matched open Request for %s", key)
return matcher.preparePair(requestHTTPMessage, &responseHTTPMessage)
}
matcher.openMessagesMap.Store(key, &responseHTTPMessage)
rlog.Tracef(1, "Registered open Response for %s", key)
return nil
}

View File

@@ -12,7 +12,6 @@ import (
"strings"
"github.com/beevik/etree"
"github.com/romana/rlog"
"github.com/up9inc/mizu/tap/api"
)
@@ -30,7 +29,7 @@ func IsIgnoredUserAgent(item *api.OutputChannelItem, options *api.TrafficFilteri
return false
}
request := item.Pair.Request.Payload.(HTTPPayload).Data.(*http.Request)
request := item.Pair.Request.Payload.(api.HTTPPayload).Data.(*http.Request)
for headerKey, headerValues := range request.Header {
if strings.ToLower(headerKey) == "user-agent" {
@@ -50,8 +49,8 @@ func IsIgnoredUserAgent(item *api.OutputChannelItem, options *api.TrafficFilteri
}
func FilterSensitiveData(item *api.OutputChannelItem, options *api.TrafficFilteringOptions) {
request := item.Pair.Request.Payload.(HTTPPayload).Data.(*http.Request)
response := item.Pair.Response.Payload.(HTTPPayload).Data.(*http.Response)
request := item.Pair.Request.Payload.(api.HTTPPayload).Data.(*http.Request)
response := item.Pair.Response.Payload.(api.HTTPPayload).Data.(*http.Response)
filterHeaders(&request.Header)
filterHeaders(&response.Header)
@@ -64,7 +63,6 @@ func filterRequestBody(request *http.Request, options *api.TrafficFilteringOptio
contenType := getContentTypeHeaderValue(request.Header)
body, err := ioutil.ReadAll(request.Body)
if err != nil {
rlog.Debugf("Filtering error reading body: %v", err)
return
}
filteredBody, err := filterHttpBody([]byte(body), contenType, options)
@@ -79,7 +77,6 @@ func filterResponseBody(response *http.Response, options *api.TrafficFilteringOp
contentType := getContentTypeHeaderValue(response.Header)
body, err := ioutil.ReadAll(response.Body)
if err != nil {
rlog.Debugf("Filtering error reading body: %v", err)
return
}
filteredBody, err := filterHttpBody([]byte(body), contentType, options)

View File

@@ -1,55 +0,0 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"github.com/google/martian/har"
"github.com/romana/rlog"
)
type HTTPPayload struct {
Type uint8
Data interface{}
}
type HTTPPayloader interface {
MarshalJSON() ([]byte, error)
}
type HTTPWrapper struct {
Method string `json:"method"`
Url string `json:"url"`
Details interface{} `json:"details"`
}
func (h HTTPPayload) MarshalJSON() ([]byte, error) {
switch h.Type {
case TypeHttpRequest:
harRequest, err := har.NewRequest(h.Data.(*http.Request), true)
if err != nil {
rlog.Debugf("convert-request-to-har", "Failed converting request to HAR %s (%v,%+v)", err, err, err)
return nil, errors.New("Failed converting request to HAR")
}
return json.Marshal(&HTTPWrapper{
Method: harRequest.Method,
Url: "",
Details: harRequest,
})
case TypeHttpResponse:
harResponse, err := har.NewResponse(h.Data.(*http.Response), true)
if err != nil {
rlog.Debugf("convert-response-to-har", "Failed converting response to HAR %s (%v,%+v)", err, err, err)
return nil, errors.New("Failed converting response to HAR")
}
return json.Marshal(&HTTPWrapper{
Method: "",
Url: "",
Details: harResponse,
})
default:
panic(fmt.Sprintf("HTTP payload cannot be marshaled: %s\n", h.Type))
}
}

View File

@@ -7,6 +7,8 @@ github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/klauspost/compress v1.9.8 h1:VMAMUUOh+gaxKTMk+zqbjsSjsIcUcL/LF4o63i82QyA=
github.com/klauspost/compress v1.9.8/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=

View File

@@ -2,8 +2,6 @@ module github.com/up9inc/mizu/tap/extensions/redis
go 1.16
require (
github.com/up9inc/mizu/tap/api v0.0.0
)
require github.com/up9inc/mizu/tap/api v0.0.0
replace github.com/up9inc/mizu/tap/api v0.0.0 => ../../api

View File

@@ -0,0 +1,2 @@
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=

View File

@@ -4,7 +4,6 @@ import (
"bufio"
"errors"
"fmt"
"log"
"math"
"reflect"
"strconv"
@@ -315,7 +314,6 @@ func (p *RedisProtocol) Read() (packet *RedisPacket, err error) {
}
default:
msg := fmt.Sprintf("Unrecognized element in Redis array: %v\n", reflect.TypeOf(array[0]))
log.Printf(msg)
err = errors.New(msg)
return
}
@@ -336,7 +334,6 @@ func (p *RedisProtocol) Read() (packet *RedisPacket, err error) {
packet.Value = fmt.Sprintf("%d", x.(int64))
default:
msg := fmt.Sprintf("Unrecognized Redis data type: %v\n", reflect.TypeOf(x))
log.Printf(msg)
err = errors.New(msg)
return
}

View File

@@ -3,15 +3,14 @@ module github.com/up9inc/mizu/tap
go 1.16
require (
github.com/bradleyfalzon/tlsx v0.0.0-20170624122154-28fd0e59bac4 // indirect
github.com/bradleyfalzon/tlsx v0.0.0-20170624122154-28fd0e59bac4
github.com/google/gopacket v1.1.19
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7
github.com/up9inc/mizu/shared v0.0.0
github.com/up9inc/mizu/tap/api v0.0.0
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d
golang.org/x/text v0.3.5
golang.org/x/tools v0.0.0-20210106214847-113979e3529a
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7 // indirect
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073 // indirect
)
replace github.com/up9inc/mizu/tap/api v0.0.0 => ./api
replace github.com/up9inc/mizu/shared v0.0.0 => ../shared

View File

@@ -1,45 +1,32 @@
github.com/bradleyfalzon/tlsx v0.0.0-20170624122154-28fd0e59bac4 h1:NJOOlc6ZJjix0A1rAU+nxruZtR8KboG1848yqpIUo4M=
github.com/bradleyfalzon/tlsx v0.0.0-20170624122154-28fd0e59bac4/go.mod h1:DQPxZS994Ld1Y8uwnJT+dRL04XPD0cElP/pHH/zEBHM=
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/golang-jwt/jwt/v4 v4.1.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg=
github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=
github.com/google/gopacket v1.1.19/go.mod h1:iJ8V8n6KS+z2U1A8pUwu8bW5SyEMkXJB8Yo/Vo+TKTo=
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7 h1:jkvpcEatpwuMF5O5LVxTnehj6YZ/aEZN4NWD/Xml4pI=
github.com/romana/rlog v0.0.0-20171115192701-f018bc92e7d7/go.mod h1:KTrHyWpO1sevuXPZwyeZc72ddWRFqNSKDFl7uVWKpg0=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 h1:lDH9UUVJtmYCjyT0CI4q8xvlXPxeZ0gYCVvWbmPlp88=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7 h1:OgUuv8lsRpBibGNbSizVwKWlysjaNzmC9gYMhPVfqFM=
golang.org/x/net v0.0.0-20210224082022-3d97a244fca7/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073 h1:8qxJSnu+7dRq6upnbntrmriWByIakBuct5OM/MdQC1M=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d h1:SZxvLBoTP5yHO3Frd4z4vrF+DBX9vMVanchswa69toE=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4 h1:0YWbFKbhXG/wIiuHDSKpS0Iy7FSA+u45VtBMfQcFTTc=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5 h1:i6eZZ+zk0SOf0xgBpEpPD18qWcJda6q1sxt3S0kzyUQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a h1:CB3a9Nez8M13wwlr/E2YtwoU+qYHKfC+JrDa45RXXoQ=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -14,7 +14,6 @@ import (
"flag"
"fmt"
"io"
"log"
"os"
"os/signal"
"runtime"
@@ -25,14 +24,13 @@ import (
"sync"
"time"
"github.com/romana/rlog"
"github.com/google/gopacket"
"github.com/google/gopacket/examples/util"
"github.com/google/gopacket/ip4defrag"
"github.com/google/gopacket/layers" // pulls in all layers decoders
"github.com/google/gopacket/pcap"
"github.com/google/gopacket/reassembly"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
@@ -102,7 +100,7 @@ const baseStreamChannelTimeoutMs int = 5000 * 100
/* minOutputLevel: Error will be printed only if outputLevel is above this value
* t: key for errorsMap (counting errors)
* s, a: arguments log.Printf
* s, a: arguments logger.Log.Infof
* Note: Too bad for perf that a... is evaluated
*/
func logError(minOutputLevel int, t string, s string, a ...interface{}) {
@@ -114,7 +112,7 @@ func logError(minOutputLevel int, t string, s string, a ...interface{}) {
if outputLevel >= minOutputLevel {
formatStr := fmt.Sprintf("%s: %s", t, s)
rlog.Errorf(formatStr, a...)
logger.Log.Errorf(formatStr, a...)
}
}
func Error(t string, s string, a ...interface{}) {
@@ -124,10 +122,7 @@ func SilentError(t string, s string, a ...interface{}) {
logError(2, t, s, a...)
}
func Debug(s string, a ...interface{}) {
rlog.Debugf(s, a...)
}
func Trace(s string, a ...interface{}) {
rlog.Tracef(1, s, a...)
logger.Log.Debugf(s, a...)
}
func inArrayInt(arr []int, valueToCheck int) bool {
@@ -188,11 +183,11 @@ func startMemoryProfiler() {
}
}
rlog.Info("Profiling is on, results will be written to %s", dumpPath)
logger.Log.Info("Profiling is on, results will be written to %s", dumpPath)
go func() {
if _, err := os.Stat(dumpPath); os.IsNotExist(err) {
if err := os.Mkdir(dumpPath, 0777); err != nil {
log.Fatal("could not create directory for profile: ", err)
logger.Log.Fatal("could not create directory for profile: ", err)
}
}
@@ -201,15 +196,15 @@ func startMemoryProfiler() {
filename := fmt.Sprintf("%s/%s__mem.prof", dumpPath, t.Format("15_04_05"))
rlog.Infof("Writing memory profile to %s\n", filename)
logger.Log.Infof("Writing memory profile to %s\n", filename)
f, err := os.Create(filename)
if err != nil {
log.Fatal("could not create memory profile: ", err)
logger.Log.Fatal("could not create memory profile: ", err)
}
runtime.GC() // get up-to-date statistics
if err := pprof.WriteHeapProfile(f); err != nil {
log.Fatal("could not write memory profile: ", err)
logger.Log.Fatal("could not write memory profile: ", err)
}
_ = f.Close()
time.Sleep(time.Second * time.Duration(timeInterval))
@@ -229,7 +224,7 @@ func closeTimedoutTcpStreamChannels() {
if !stream.isClosed && time.Now().After(streamWrapper.createdAt.Add(TcpStreamChannelTimeoutMs)) {
stream.Close()
appStats.IncDroppedTcpStreams()
rlog.Debugf("Dropped an unidentified TCP stream because of timeout. Total dropped: %d Total Goroutines: %d Timeout (ms): %d\n", appStats.DroppedTcpStreams, runtime.NumGoroutine(), TcpStreamChannelTimeoutMs/1000000)
logger.Log.Debugf("Dropped an unidentified TCP stream because of timeout. Total dropped: %d Total Goroutines: %d Timeout (ms): %d\n", appStats.DroppedTcpStreams, runtime.NumGoroutine(), TcpStreamChannelTimeoutMs/1000000)
}
} else {
if !stream.superIdentifier.IsClosedOthers {
@@ -254,7 +249,6 @@ func closeTimedoutTcpStreamChannels() {
}
func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
log.SetFlags(log.LstdFlags | log.LUTC | log.Lshortfile)
go closeTimedoutTcpStreamChannels()
defer util.Run()()
@@ -269,8 +263,8 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
if localhostIPs, err := getLocalhostIPs(); err != nil {
// TODO: think this over
rlog.Info("Failed to get self IP addresses")
rlog.Errorf("Getting-Self-Address", "Error getting self ip address: %s (%v,%+v)", err, err, err)
logger.Log.Info("Failed to get self IP addresses")
logger.Log.Errorf("Getting-Self-Address", "Error getting self ip address: %s (%v,%+v)", err, err, err)
ownIps = make([]string, 0)
} else {
ownIps = localhostIPs
@@ -280,7 +274,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
var err error
if *fname != "" {
if handle, err = pcap.OpenOffline(*fname); err != nil {
log.Fatalf("PCAP OpenOffline error: %v", err)
logger.Log.Fatalf("PCAP OpenOffline error: %v", err)
}
} else {
// This is a little complicated because we want to allow all possible options
@@ -288,33 +282,33 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
// just call pcap.OpenLive if you want a simple handle.
inactive, err := pcap.NewInactiveHandle(*iface)
if err != nil {
log.Fatalf("could not create: %v", err)
logger.Log.Fatalf("could not create: %v", err)
}
defer inactive.CleanUp()
if err = inactive.SetSnapLen(*snaplen); err != nil {
log.Fatalf("could not set snap length: %v", err)
logger.Log.Fatalf("could not set snap length: %v", err)
} else if err = inactive.SetPromisc(*promisc); err != nil {
log.Fatalf("could not set promisc mode: %v", err)
logger.Log.Fatalf("could not set promisc mode: %v", err)
} else if err = inactive.SetTimeout(time.Second); err != nil {
log.Fatalf("could not set timeout: %v", err)
logger.Log.Fatalf("could not set timeout: %v", err)
}
if *tstype != "" {
if t, err := pcap.TimestampSourceFromString(*tstype); err != nil {
log.Fatalf("Supported timestamp types: %v", inactive.SupportedTimestamps())
logger.Log.Fatalf("Supported timestamp types: %v", inactive.SupportedTimestamps())
} else if err := inactive.SetTimestampSource(t); err != nil {
log.Fatalf("Supported timestamp types: %v", inactive.SupportedTimestamps())
logger.Log.Fatalf("Supported timestamp types: %v", inactive.SupportedTimestamps())
}
}
if handle, err = inactive.Activate(); err != nil {
log.Fatalf("PCAP Activate error: %v", err)
logger.Log.Fatalf("PCAP Activate error: %v", err)
}
defer handle.Close()
}
if len(flag.Args()) > 0 {
bpffilter := strings.Join(flag.Args(), " ")
rlog.Infof("Using BPF filter %q", bpffilter)
logger.Log.Infof("Using BPF filter %q", bpffilter)
if err = handle.SetBPFFilter(bpffilter); err != nil {
log.Fatalf("BPF filter error: %v", err)
logger.Log.Fatalf("BPF filter error: %v", err)
}
}
@@ -325,12 +319,12 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
decoderName = fmt.Sprintf("%s", handle.LinkType())
}
if dec, ok = gopacket.DecodersByLayerName[decoderName]; !ok {
log.Fatalln("No decoder named", decoderName)
logger.Log.Fatal("No decoder named", decoderName)
}
source := gopacket.NewPacketSource(handle, dec)
source.Lazy = *lazy
source.NoCopy = true
rlog.Info("Starting to read packets")
logger.Log.Info("Starting to read packets")
appStats.SetStartTime(time.Now())
defragger := ip4defrag.NewIPv4Defragmenter()
@@ -347,7 +341,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
maxBufferedPagesTotal := GetMaxBufferedPagesPerConnection()
maxBufferedPagesPerConnection := GetMaxBufferedPagesTotal()
rlog.Infof("Assembler options: maxBufferedPagesTotal=%d, maxBufferedPagesPerConnection=%d", maxBufferedPagesTotal, maxBufferedPagesPerConnection)
logger.Log.Infof("Assembler options: maxBufferedPagesTotal=%d, maxBufferedPagesPerConnection=%d", maxBufferedPagesTotal, maxBufferedPagesPerConnection)
assembler.AssemblerOptions.MaxBufferedPagesTotal = maxBufferedPagesTotal
assembler.AssemblerOptions.MaxBufferedPagesPerConnection = maxBufferedPagesPerConnection
@@ -377,7 +371,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
errorMapLen := len(errorsMap)
errorsSummery := fmt.Sprintf("%v", errorsMap)
errorsMapMutex.Unlock()
log.Printf("%v (errors: %v, errTypes:%v) - Errors Summary: %s",
logger.Log.Infof("%v (errors: %v, errTypes:%v) - Errors Summary: %s",
time.Since(appStats.StartTime),
nErrors,
errorMapLen,
@@ -387,7 +381,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
// At this moment
memStats := runtime.MemStats{}
runtime.ReadMemStats(&memStats)
log.Printf(
logger.Log.Infof(
"mem: %d, goroutines: %d",
memStats.HeapAlloc,
runtime.NumGoroutine(),
@@ -395,7 +389,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
// Since the last print
cleanStats := cleaner.dumpStats()
log.Printf(
logger.Log.Infof(
"cleaner - flushed connections: %d, closed connections: %d, deleted messages: %d",
cleanStats.flushed,
cleanStats.closed,
@@ -403,7 +397,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
)
currentAppStats := appStats.DumpStats()
appStatsJSON, _ := json.Marshal(currentAppStats)
log.Printf("app stats - %v", string(appStatsJSON))
logger.Log.Infof("app stats - %v", string(appStatsJSON))
}
}()
@@ -416,15 +410,15 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
if err == io.EOF {
break
} else if err != nil {
rlog.Debugf("Error:", err)
logger.Log.Debugf("Error: %v", err)
continue
}
packetsCount := appStats.IncPacketsCount()
rlog.Debugf("PACKET #%d", packetsCount)
logger.Log.Debugf("PACKET #%d", packetsCount)
data := packet.Data()
appStats.UpdateProcessedBytes(uint64(len(data)))
if *hexdumppkt {
rlog.Debugf("Packet content (%d/0x%x) - %s", len(data), len(data), hex.Dump(data))
logger.Log.Debugf("Packet content (%d/0x%x) - %s", len(data), len(data), hex.Dump(data))
}
// defrag the IPv4 packet if required
@@ -437,17 +431,17 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
l := ip4.Length
newip4, err := defragger.DefragIPv4(ip4)
if err != nil {
log.Fatalln("Error while de-fragmenting", err)
logger.Log.Fatal("Error while de-fragmenting", err)
} else if newip4 == nil {
rlog.Debugf("Fragment...")
logger.Log.Debugf("Fragment...")
continue // packet fragment, we don't have whole packet yet.
}
if newip4.Length != l {
stats.ipdefrag++
rlog.Debugf("Decoding re-assembled packet: %s", newip4.NextLayerType())
logger.Log.Debugf("Decoding re-assembled packet: %s", newip4.NextLayerType())
pb, ok := packet.(gopacket.PacketBuilder)
if !ok {
log.Panic("Not a PacketBuilder")
logger.Log.Panic("Not a PacketBuilder")
}
nextDecoder := newip4.NextLayerType()
_ = nextDecoder.Decode(newip4.Payload, pb)
@@ -461,14 +455,14 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
if *checksum {
err := tcp.SetNetworkLayerForChecksum(packet.NetworkLayer())
if err != nil {
log.Fatalf("Failed to set network layer for checksum: %s\n", err)
logger.Log.Fatalf("Failed to set network layer for checksum: %s\n", err)
}
}
c := Context{
CaptureInfo: packet.Metadata().CaptureInfo,
}
stats.totalsz += len(tcp.Payload)
rlog.Debugf("%s : %v -> %s : %v", packet.NetworkLayer().NetworkFlow().Src(), tcp.SrcPort, packet.NetworkLayer().NetworkFlow().Dst(), tcp.DstPort)
logger.Log.Debugf("%s : %v -> %s : %v", packet.NetworkLayer().NetworkFlow().Src(), tcp.SrcPort, packet.NetworkLayer().NetworkFlow().Dst(), tcp.DstPort)
assemblerMutex.Lock()
assembler.AssembleWithContext(packet.NetworkLayer().NetworkFlow(), tcp, &c)
assemblerMutex.Unlock()
@@ -479,7 +473,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
errorsMapMutex.Lock()
errorMapLen := len(errorsMap)
errorsMapMutex.Unlock()
log.Printf("Processed %v packets (%v bytes) in %v (errors: %v, errTypes:%v)",
logger.Log.Infof("Processed %v packets (%v bytes) in %v (errors: %v, errTypes:%v)",
appStats.PacketsCount,
appStats.ProcessedBytes,
time.Since(appStats.StartTime),
@@ -488,7 +482,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
}
select {
case <-signalChan:
log.Printf("Caught SIGINT: aborting")
logger.Log.Infof("Caught SIGINT: aborting")
done = true
default:
// NOP: continue
@@ -501,7 +495,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
assemblerMutex.Lock()
closed := assembler.FlushAll()
assemblerMutex.Unlock()
rlog.Debugf("Final flush: %d closed", closed)
logger.Log.Debugf("Final flush: %d closed", closed)
if outputLevel >= 2 {
streamPool.Dump()
}
@@ -509,7 +503,7 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
if *memprofile != "" {
f, err := os.Create(*memprofile)
if err != nil {
log.Fatal(err)
logger.Log.Fatal(err)
}
_ = pprof.WriteHeapProfile(f)
_ = f.Close()
@@ -517,29 +511,29 @@ func startPassiveTapper(outputItems chan *api.OutputChannelItem) {
streamFactory.WaitGoRoutines()
assemblerMutex.Lock()
rlog.Debugf("%s", assembler.Dump())
logger.Log.Debugf("%s", assembler.Dump())
assemblerMutex.Unlock()
if !*nodefrag {
log.Printf("IPdefrag:\t\t%d", stats.ipdefrag)
logger.Log.Infof("IPdefrag:\t\t%d", stats.ipdefrag)
}
log.Printf("TCP stats:")
log.Printf(" missed bytes:\t\t%d", stats.missedBytes)
log.Printf(" total packets:\t\t%d", stats.pkt)
log.Printf(" rejected FSM:\t\t%d", stats.rejectFsm)
log.Printf(" rejected Options:\t%d", stats.rejectOpt)
log.Printf(" reassembled bytes:\t%d", stats.sz)
log.Printf(" total TCP bytes:\t%d", stats.totalsz)
log.Printf(" conn rejected FSM:\t%d", stats.rejectConnFsm)
log.Printf(" reassembled chunks:\t%d", stats.reassembled)
log.Printf(" out-of-order packets:\t%d", stats.outOfOrderPackets)
log.Printf(" out-of-order bytes:\t%d", stats.outOfOrderBytes)
log.Printf(" biggest-chunk packets:\t%d", stats.biggestChunkPackets)
log.Printf(" biggest-chunk bytes:\t%d", stats.biggestChunkBytes)
log.Printf(" overlap packets:\t%d", stats.overlapPackets)
log.Printf(" overlap bytes:\t\t%d", stats.overlapBytes)
log.Printf("Errors: %d", nErrors)
logger.Log.Infof("TCP stats:")
logger.Log.Infof(" missed bytes:\t\t%d", stats.missedBytes)
logger.Log.Infof(" total packets:\t\t%d", stats.pkt)
logger.Log.Infof(" rejected FSM:\t\t%d", stats.rejectFsm)
logger.Log.Infof(" rejected Options:\t%d", stats.rejectOpt)
logger.Log.Infof(" reassembled bytes:\t%d", stats.sz)
logger.Log.Infof(" total TCP bytes:\t%d", stats.totalsz)
logger.Log.Infof(" conn rejected FSM:\t%d", stats.rejectConnFsm)
logger.Log.Infof(" reassembled chunks:\t%d", stats.reassembled)
logger.Log.Infof(" out-of-order packets:\t%d", stats.outOfOrderPackets)
logger.Log.Infof(" out-of-order bytes:\t%d", stats.outOfOrderBytes)
logger.Log.Infof(" biggest-chunk packets:\t%d", stats.biggestChunkPackets)
logger.Log.Infof(" biggest-chunk bytes:\t%d", stats.biggestChunkBytes)
logger.Log.Infof(" overlap packets:\t%d", stats.overlapPackets)
logger.Log.Infof(" overlap bytes:\t\t%d", stats.overlapBytes)
logger.Log.Infof("Errors: %d", nErrors)
for e := range errorsMap {
log.Printf(" %s:\t\t%d", e, errorsMap[e])
logger.Log.Infof(" %s:\t\t%d", e, errorsMap[e])
}
log.Printf("AppStats: %v", GetStats())
logger.Log.Infof("AppStats: %v", GetStats())
}

View File

@@ -9,7 +9,7 @@ import (
"time"
"github.com/bradleyfalzon/tlsx"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
)
@@ -79,7 +79,7 @@ func (h *tcpReader) Read(p []byte) (int, error) {
clientHello := tlsx.ClientHello{}
err := clientHello.Unmarshall(msg.bytes)
if err == nil {
rlog.Debugf("Detected TLS client hello with SNI %s\n", clientHello.SNI)
logger.Log.Debugf("Detected TLS client hello with SNI %s\n", clientHello.SNI)
// TODO: Throws `panic: runtime error: invalid memory address or nil pointer dereference` error.
// numericPort, _ := strconv.Atoi(h.tcpID.DstPort)
// h.outboundLinkWriter.WriteOutboundLink(h.tcpID.SrcIP, h.tcpID.DstIP, numericPort, clientHello.SNI, TLSProtocol)

View File

@@ -108,7 +108,7 @@ func (t *tcpStream) ReassembledSG(sg reassembly.ScatterGather, ac reassembly.Ass
} else {
ident = fmt.Sprintf("%v %v(%s): ", t.net.Reverse(), t.transport.Reverse(), dir)
}
Trace("%s: SG reassembled packet with %d bytes (start:%v,end:%v,skip:%d,saved:%d,nb:%d,%d,overlap:%d,%d)", ident, length, start, end, skip, saved, sgStats.Packets, sgStats.Chunks, sgStats.OverlapBytes, sgStats.OverlapPackets)
Debug("%s: SG reassembled packet with %d bytes (start:%v,end:%v,skip:%d,saved:%d,nb:%d,%d,overlap:%d,%d)", ident, length, start, end, skip, saved, sgStats.Packets, sgStats.Chunks, sgStats.OverlapBytes, sgStats.OverlapPackets)
if skip == -1 && *allowmissinginit {
// this is allowed
} else if skip != 0 {
@@ -127,7 +127,7 @@ func (t *tcpStream) ReassembledSG(sg reassembly.ScatterGather, ac reassembly.Ass
}
dnsSize := binary.BigEndian.Uint16(data[:2])
missing := int(dnsSize) - len(data[2:])
Trace("dnsSize: %d, missing: %d", dnsSize, missing)
Debug("dnsSize: %d, missing: %d", dnsSize, missing)
if missing > 0 {
Debug("Missing some bytes: %d", missing)
sg.KeepFrom(0)
@@ -138,7 +138,7 @@ func (t *tcpStream) ReassembledSG(sg reassembly.ScatterGather, ac reassembly.Ass
if err != nil {
SilentError("DNS-parser", "Failed to decode DNS: %v", err)
} else {
Trace("DNS: %s", gopacket.LayerDump(dns))
Debug("DNS: %s", gopacket.LayerDump(dns))
}
if len(data) > 2+int(dnsSize) {
sg.KeepFrom(2 + int(dnsSize))
@@ -173,7 +173,7 @@ func (t *tcpStream) ReassembledSG(sg reassembly.ScatterGather, ac reassembly.Ass
}
func (t *tcpStream) ReassemblyComplete(ac reassembly.AssemblerContext) bool {
Trace("%s: Connection closed", t.ident)
Debug("%s: Connection closed", t.ident)
if t.isTapTarget && !t.isClosed {
t.Close()
}

View File

@@ -5,7 +5,7 @@ import (
"sync"
"time"
"github.com/romana/rlog"
"github.com/up9inc/mizu/shared/logger"
"github.com/up9inc/mizu/tap/api"
"github.com/google/gopacket"
@@ -33,7 +33,7 @@ var streams *sync.Map = &sync.Map{} // global
var streamId int64 = 0
func (factory *tcpStreamFactory) New(net, transport gopacket.Flow, tcp *layers.TCP, ac reassembly.AssemblerContext) reassembly.Stream {
rlog.Debugf("* NEW: %s %s", net, transport)
logger.Log.Debugf("* NEW: %s %s", net, transport)
fsmOptions := reassembly.TCPSimpleFSMOptions{
SupportMissingEstablishment: *allowmissinginit,
}
@@ -123,21 +123,21 @@ func (factory *tcpStreamFactory) WaitGoRoutines() {
func (factory *tcpStreamFactory) getStreamProps(srcIP string, srcPort string, dstIP string, dstPort string) *streamProps {
if hostMode {
if inArrayString(gSettings.filterAuthorities, fmt.Sprintf("%s:%s", dstIP, dstPort)) {
rlog.Debugf("getStreamProps %s", fmt.Sprintf("+ host1 %s:%s", dstIP, dstPort))
logger.Log.Debugf("getStreamProps %s", fmt.Sprintf("+ host1 %s:%s", dstIP, dstPort))
return &streamProps{isTapTarget: true, isOutgoing: false}
} else if inArrayString(gSettings.filterAuthorities, dstIP) {
rlog.Debugf("getStreamProps %s", fmt.Sprintf("+ host2 %s", dstIP))
logger.Log.Debugf("getStreamProps %s", fmt.Sprintf("+ host2 %s", dstIP))
return &streamProps{isTapTarget: true, isOutgoing: false}
} else if inArrayString(gSettings.filterAuthorities, fmt.Sprintf("%s:%s", srcIP, srcPort)) {
rlog.Debugf("getStreamProps %s", fmt.Sprintf("+ host3 %s:%s", srcIP, srcPort))
logger.Log.Debugf("getStreamProps %s", fmt.Sprintf("+ host3 %s:%s", srcIP, srcPort))
return &streamProps{isTapTarget: true, isOutgoing: true}
} else if inArrayString(gSettings.filterAuthorities, srcIP) {
rlog.Debugf("getStreamProps %s", fmt.Sprintf("+ host4 %s", srcIP))
logger.Log.Debugf("getStreamProps %s", fmt.Sprintf("+ host4 %s", srcIP))
return &streamProps{isTapTarget: true, isOutgoing: true}
}
return &streamProps{isTapTarget: false, isOutgoing: false}
} else {
rlog.Debugf("getStreamProps %s", fmt.Sprintf("+ notHost3 %s:%s -> %s:%s", srcIP, srcPort, dstIP, dstPort))
logger.Log.Debugf("getStreamProps %s", fmt.Sprintf("+ notHost3 %s:%s -> %s:%s", srcIP, srcPort, dstIP, dstPort))
return &streamProps{isTapTarget: true}
}
}

View File

@@ -35,7 +35,7 @@ const App = () => {
try {
const recentTLSLinks = await api.getRecentTLSLinks();
if (recentTLSLinks?.length > 0) {
setAddressesWithTLS(new Set([...addressesWithTLS, ...recentTLSLinks]));
setAddressesWithTLS(new Set(recentTLSLinks));
setShowTLSWarning(true);
}
const auth = await api.getAuthStatus();
@@ -45,7 +45,7 @@ const App = () => {
}
})();
},[]);
}, []);
const onTLSDetected = (destAddress: string) => {
addressesWithTLS.add(destAddress);

View File

@@ -71,7 +71,7 @@ export const EntryDetailed: React.FC<EntryDetailedProps> = ({entryData}) => {
/>
{entryData.data && <EntrySummary data={entryData.data}/>}
<>
{entryData.data && <EntryViewer representation={entryData.representation} isRulesEnabled={entryData.isRulesEnabled} rulesMatched={entryData.rulesMatched} elapsedTime={entryData.data.elapsedTime} color={entryData.protocol.backgroundColor}/>}
{entryData.data && <EntryViewer representation={entryData.representation} isRulesEnabled={entryData.isRulesEnabled} rulesMatched={entryData.rulesMatched} contractStatus={entryData.data.contractStatus} requestReason={entryData.data.contractRequestReason} responseReason={entryData.data.contractResponseReason} contractContent={entryData.data.contractContent} elapsedTime={entryData.data.elapsedTime} color={entryData.protocol.backgroundColor}/>}
</>
</>
};

View File

@@ -150,8 +150,6 @@ export const EntryTableSection: React.FC<EntrySectionProps> = ({title, color, ar
</React.Fragment>
}
interface EntryPolicySectionProps {
title: string,
color: string,
@@ -159,7 +157,6 @@ interface EntryPolicySectionProps {
arrayToIterate: any[],
}
interface EntryPolicySectionCollapsibleTitleProps {
label: string;
matched: string;
@@ -253,3 +250,28 @@ export const EntryTablePolicySection: React.FC<EntryPolicySectionProps> = ({titl
}
</React.Fragment>
}
interface EntryContractSectionProps {
color: string,
requestReason: string,
responseReason: string,
contractContent: string,
}
export const EntryContractSection: React.FC<EntryContractSectionProps> = ({color, requestReason, responseReason, contractContent}) => {
return <React.Fragment>
{requestReason && <EntrySectionContainer title="Request" color={color}>
{requestReason}
</EntrySectionContainer>}
{responseReason && <EntrySectionContainer title="Response" color={color}>
{responseReason}
</EntrySectionContainer>}
{contractContent && <EntrySectionContainer title="Contract" color={color}>
<SyntaxHighlighter
isWrapped={false}
code={contractContent}
language={"yaml"}
/>
</EntrySectionContainer>}
</React.Fragment>
}

View File

@@ -1,7 +1,7 @@
import React, {useState} from 'react';
import styles from './EntryViewer.module.sass';
import Tabs from "../UI/Tabs";
import {EntryTableSection, EntryBodySection, EntryTablePolicySection} from "./EntrySections";
import {EntryTableSection, EntryBodySection, EntryTablePolicySection, EntryContractSection} from "./EntrySections";
enum SectionTypes {
SectionTable = "table",
@@ -33,7 +33,7 @@ const SectionsRepresentation: React.FC<any> = ({data, color}) => {
return <>{sections}</>;
}
const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rulesMatched, elapsedTime, color}) => {
const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color}) => {
var TABS = [
{
tab: 'Request'
@@ -50,6 +50,7 @@ const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rule
var responseTabIndex = 0;
var rulesTabIndex = 0;
var contractTabIndex = 0;
if (response) {
TABS.push(
@@ -69,11 +70,19 @@ const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rule
rulesTabIndex = TABS.length - 1;
}
if (contractStatus !== 0 && contractContent) {
TABS.push(
{
tab: 'Contract',
}
);
contractTabIndex = TABS.length - 1;
}
return <div className={styles.Entry}>
{<div className={styles.body}>
<div className={styles.bodyHeader}>
<Tabs tabs={TABS} currentTab={currentTab} color={color} onChange={setCurrentTab} leftAligned/>
{request?.url && <a className={styles.endpointURL} href={request.payload.url} target='_blank' rel="noreferrer">{request.payload.url}</a>}
</div>
{currentTab === TABS[0].tab && <React.Fragment>
<SectionsRepresentation data={request} color={color}/>
@@ -84,6 +93,9 @@ const AutoRepresentation: React.FC<any> = ({representation, isRulesEnabled, rule
{isRulesEnabled && currentTab === TABS[rulesTabIndex].tab && <React.Fragment>
<EntryTablePolicySection title={'Rule'} color={color} latency={elapsedTime} arrayToIterate={rulesMatched ? rulesMatched : []}/>
</React.Fragment>}
{contractStatus !== 0 && contractContent && currentTab === TABS[contractTabIndex].tab && <React.Fragment>
<EntryContractSection color={color} requestReason={requestReason} responseReason={responseReason} contractContent={contractContent}/>
</React.Fragment>}
</div>}
</div>;
}
@@ -92,12 +104,26 @@ interface Props {
representation: any;
isRulesEnabled: boolean;
rulesMatched: any;
contractStatus: number;
requestReason: string;
responseReason: string;
contractContent: string;
color: string;
elapsedTime: number;
}
const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatched, elapsedTime, color}) => {
return <AutoRepresentation representation={representation} isRulesEnabled={isRulesEnabled} rulesMatched={rulesMatched} elapsedTime={elapsedTime} color={color}/>
const EntryViewer: React.FC<Props> = ({representation, isRulesEnabled, rulesMatched, contractStatus, requestReason, responseReason, contractContent, elapsedTime, color}) => {
return <AutoRepresentation
representation={representation}
isRulesEnabled={isRulesEnabled}
rulesMatched={rulesMatched}
contractStatus={contractStatus}
requestReason={requestReason}
responseReason={responseReason}
contractContent={contractContent}
elapsedTime={elapsedTime}
color={color}
/>
};
export default EntryViewer;

View File

@@ -38,6 +38,7 @@
.ruleNumberText
font-size: 12px
font-weight: 600
white-space: nowrap
.ruleNumberTextFailure
color: #DB2156
@@ -72,12 +73,17 @@
padding-left: 10px
flex-grow: 1
.directionContainer
.separatorRight
display: flex
border-right: 1px solid $data-background-color
padding: 4px
padding-right: 12px
.separatorLeft
display: flex
padding: 4px
padding-left: 12px
.port
font-size: 12px
color: $secondary-font-color

View File

@@ -26,6 +26,7 @@ interface Entry {
isOutgoing?: boolean;
latency: number;
rules: Rules;
contractStatus: number,
}
interface Rules {
@@ -64,7 +65,7 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
}
}
let additionalRulesProperties = "";
let ruleSuccess: boolean;
let ruleSuccess = true;
let rule = 'latency' in entry.rules
if (rule) {
if (entry.rules.latency !== -1) {
@@ -91,11 +92,32 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
}
}
}
var contractEnabled = true;
var contractText = "";
switch (entry.contractStatus) {
case 0:
contractEnabled = false;
break;
case 1:
additionalRulesProperties = styles.ruleSuccessRow
ruleSuccess = true
contractText = "No Breaches"
break;
case 2:
additionalRulesProperties = styles.ruleFailureRow
ruleSuccess = false
contractText = "Breach"
break;
default:
break;
}
return <>
<div
id={entry.id}
className={`${styles.row}
${isSelected && !rule ? styles.rowSelected : additionalRulesProperties}`}
${isSelected && !rule && !contractEnabled ? styles.rowSelected : additionalRulesProperties}`}
onClick={() => setFocusedEntryId(entry.id)}
style={{
border: isSelected ? `1px ${entry.protocol.backgroundColor} solid` : "1px transparent solid",
@@ -117,12 +139,19 @@ export const EntryItem: React.FC<EntryProps> = ({entry, setFocusedEntryId, isSel
</div>
{
rule ?
<div className={`${styles.ruleNumberText} ${ruleSuccess ? styles.ruleNumberTextSuccess : styles.ruleNumberTextFailure}`}>
<div className={`${styles.ruleNumberText} ${ruleSuccess ? styles.ruleNumberTextSuccess : styles.ruleNumberTextFailure} ${rule && contractEnabled ? styles.separatorRight : ""}`}>
{`Rules (${numberOfRules})`}
</div>
: ""
}
<div className={styles.directionContainer}>
{
contractEnabled ?
<div className={`${styles.ruleNumberText} ${ruleSuccess ? styles.ruleNumberTextSuccess : styles.ruleNumberTextFailure} ${rule && contractEnabled ? styles.separatorLeft : ""}`}>
{contractText}
</div>
: ""
}
<div className={styles.separatorRight}>
<span className={styles.port} title="Source Port">{entry.sourcePort}</span>
{entry.isOutgoing ?
<img src={outgoingIcon} alt="Ingoing traffic" title="Ingoing"/>

View File

@@ -22,27 +22,27 @@ export default class Api {
}
tapStatus = async () => {
const response = await this.client.get("/api/tapStatus");
const response = await this.client.get("/status/tap");
return response.data;
}
analyzeStatus = async () => {
const response = await this.client.get("/api/analyzeStatus");
const response = await this.client.get("/status/analyze");
return response.data;
}
getEntry = async (entryId) => {
const response = await this.client.get(`/api/entries/${entryId}`);
const response = await this.client.get(`/entries/${entryId}`);
return response.data;
}
fetchEntries = async (operator, timestamp) => {
const response = await this.client.get(`/api/entries?limit=50&operator=${operator}&timestamp=${timestamp}`);
const response = await this.client.get(`/entries?limit=50&operator=${operator}&timestamp=${timestamp}`);
return response.data;
}
getRecentTLSLinks = async () => {
const response = await this.client.get("/api/recentTLSLinks");
const response = await this.client.get("/status/recentTLSLinks");
return response.data;
}