78 Commits

Author SHA1 Message Date
RamiBerm
2da7c0f0ed TRA-4065 fix mizu tapping (#561)
* Update main.go and passive_tapper.go

* Update main.go
2021-12-27 15:11:36 +02:00
RamiBerm
cfe9e863b7 TRA-4065 support inflight tap target update (#556)
* WIP

* WIP

* Update main.go

* Update main.go and passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go

* Update passive_tapper.go
2021-12-27 11:50:34 +02:00
M. Mert Yıldıran
e358aa4c8f Remove TCP related logging to reduce the amount of logs on DEBUG level (#553) 2021-12-26 15:10:02 +03:00
David Levanon
a75bac181d support linkerd (#547)
* support linkerd - initial commit

* renaming readEnvironmentVariable
2021-12-20 13:57:58 +02:00
M. Mert Yıldıran
23332639d0 Fix a JavaScript error in case of null attribute and an interface conversion error in the API server (#540) 2021-12-17 17:26:36 +03:00
David Levanon
ab029f4394 Send pod info to tapper (#532) 2021-12-16 10:51:03 +02:00
M. Mert Yıldıran
4e17ac5654 Remove unnecessary fields and split service into src.name and dst.name (#525)
* Remove unnecessary fields and split `service` into `src.name` and `dst.name`

* Don't fall back to IP address but instead display `[Unresolved]` text

* Fix the CSS issues in the plus icon position and replace the separator `->` text with `SwapHorizIcon`
2021-12-14 11:36:02 +03:00
M. Mert Yıldıran
a62842ac9f Add HTTP2 Over Cleartext (H2C) support (#510)
* Add HTTP2 Over Cleartext (H2C) support

* Remove a parameter which is a remnant of debugging
2021-11-25 20:36:13 +03:00
M. Mert Yıldıran
e667597e6e Rename URL field to Target URI in the UI to prevent confusion (#509) 2021-11-25 20:15:43 +03:00
David Levanon
01d6005a7b minor logging changes (#499)
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2021-11-23 14:21:53 +02:00
M. Mert Yıldıran
d66c7445e6 Remove SetHostname method in HTTP extension (#496) 2021-11-22 19:30:06 +03:00
M. Mert Yıldıran
12ca3d8779 Make the gRPC and HTTP/2 distinction (#492)
* Remove the extra negation on `nodefrag` flag's value

* Support IPv4 fragmentation and IPv6 at the same time

* Set `Method` and `StatusCode` fields correctly for `HTTP/2`

* Replace unnecessary `grpc` naming with `http2`

* Make the `gRPC` and `HTTP/2` distinction

* Fix the macros of `http` extension

* Fix the macros of other protocol extensions

* Update the method signature of `Represent`

* Fix the `HTTP/2` support

* Fix some minor issues

* Upgrade Basenine version from `0.2.10` to `0.2.11`

Sorts macros before expanding them and prioritize the long macros.

* Don't regex split the gRPC method name

* Re-enable `nodefrag` flag
2021-11-22 17:46:35 +03:00
M. Mert Yıldıran
02a125bb86 Disable IPv4 defragmentation and support IPv6 (#487)
* Remove the extra negation on `nodefrag` flag's value

* Support IPv4 fragmentation and IPv6 at the same time

* Re-enable `nodefrag` flag
2021-11-22 17:35:17 +03:00
M. Mert Yıldıran
08d7fa988e Remove tap/tester/ directory (#489)
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2021-11-22 17:32:38 +03:00
David Levanon
6caa94f08f Add support to auto discover envoy processes (#459)
* discover envoy pids using cluster ips

* add istio flag to cli + rename mtls flag to istio

* add istio.md to docs

* Fixing typos

* Fix minor typos and grammer in docs

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-11-21 15:45:07 +02:00
RoyUP9
763f72a640 remove newline in logs, fixed logs time format (#469) 2021-11-16 12:07:48 +02:00
Igor Gov
a6ec246dd1 Stop reduction of user agent header (#468) 2021-11-16 11:33:31 +02:00
M. Mert Yıldıran
a6bf39fad5 Prevent elapsedTime to be negative (#467)
Also fix the `elapsedTime` for Redis.
2021-11-16 02:52:48 +03:00
M. Mert Yıldıran
58a1eac247 Set response.bodySize to 0 if it's negative (#466) 2021-11-16 01:58:22 +03:00
M. Mert Yıldıran
618cb3a409 Optimize UI entry feed performance (#452)
* Optimize the React code for feeding the entries

By building `EntryItem` only once and updating the `entries` state on meta query messages.

* Upgrade `react-scrollable-feed-virtualized` version from `1.4.3` to `1.4.8`

* Fix the `isSelected` state

* Set the query text before deciding the background to prevent lags while typing

* Upgrade Basenine version from `0.2.6` to `0.2.7`

* Set the query background color only if the query is same after the HTTP request and use `useEffect` instead

* Upgrade Basenine version from `0.2.7` to `0.2.8`

* Use `CancelToken` of `axios` instead of trying to check the query state

* Turn `updateQuery` function into a state hook

* Update the macro for `http`

* Do the `source.cancel()` call in `axios.CancelToken`

* Reduce client-side logging
2021-11-15 17:32:05 +03:00
M. Mert Yıldıran
d2fe3f6620 Migrate from SQLite to Basenine and introduce a new filtering syntax (#279)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`

* Write the client and integrate to the new real-time database

* Refactor the WebSocket implementaiton for `/ws`

* Adapt the UI to the new filtering system

* Fix the rest of the issues in the UI

* Increase the buffer of the scanner

* Implement accessing single records

* Increase the buffer of another scanner

* Populate `Request` and `Response` fields of `MizuEntry`

* Add syntax highlighting for the query

* Add database to `Dockerfile`

* Fix some issues

* Update the `realtime_dbms` Git module commit hash

* Upgrade Gin version and print the query string

* Revert "Upgrade Gin version and print the query string"

This reverts commit aa09f904ee.

* Use WebSocket's itself to query instead of the query string

* Fix some errors related to conversion to HAR

* Fix the issues caused by the latest merge

* Fix the build error

* Fix PR validation GitHub workflow

* Replace the git submodule with latest Basenine version `0.1.0`

Remove `realtime_client.go` and use the official client library `github.com/up9inc/basenine/client/go` instead.

* Move Basenine host and port constants to `shared` module

* Reliably execute and wait for Basenine to become available

* Upgrade Basenine version

* Properly close WebSocket and data channel

* Fix the issues caused by the recent merge commit

* Clean up the TypeScript code

* Update `.gitignore`

* Limit the database size

* Add `Macros` method signature to `Dissector` interface and set the macros provided by the protocol extensions

* Run `go mod tidy` on `agent`

* Upgrade `github.com/up9inc/basenine/client/go` version

* Implement a mechanism to update the query using click events in the UI and use it for protocol macros

* Update the query on click to timestamps

* Fix some issues in the WebSocket and channel handling

* Update the query on clicks to status code

* Update the query on clicks to method, path and service

* Update the query on clicks to is outgoing, source and destination ports

* Add an API endpoint to validate the query against syntax errors

* Move the query background color state into `TrafficPage`

* Fix the logic in `setQuery`

* Display a toast message in case of a syntax error in the query

* Remove a call to `fmt.Printf`

* Upgrade Basenine version to `0.1.3`

* Fix an issue related to getting `MAX_ENTRIES_DB_BYTES` environment variable

* Have the `path` key in request details, in HTTP

* Rearrange the HTTP headers for the querying

* Do the same thing for `cookies` and `queryString`

* Update the query on click to table elements

Add the selectors for `TABLE` type representations in HTTP extension.

* Update the query on click to `bodySize` and `elapsedTime` in `EntryTitle`

* Add the selectors for `TABLE` type representations in AMQP extension

* Add the selectors for `TABLE` type representations in Kafka extension

* Add the selectors for `TABLE` type representations in Redis extension

* Define a struct in `tap/api.go` for the section representation data

* Add the selectors for `BODY` type representations

* Add `request.path` to the HTTP request details

* Change the summary string's field name from `path` to `summary`

* Introduce `queryable` CSS class for queryable UI elements and underline them on hover

* Instead of `N requests` at the bottom, make it `Displaying N results (queried X/Y)` and live update the values

Upgrade Basenine version to `0.2.0`.

* Verify the sha256sum of Basenine executable inside `Dockerfile`

* Pass the start time to web UI through WebSocket and always show the `EntriesList` footer

* Pipe the `stderr` of Basenine as well

* Fix the layout issues related to `CodeEditor` in the UI

* Use the correct `shasum` command in `Dockerfile`

* Upgrade Basenine version to `0.2.1`

* Limit the height of `CodeEditor` container

* Remove `Paused` enum `ConnectionStatus` in UI

* Fix the issue caused by the recent merge

* Add the filtering guide (cheatsheet)

* Update open cheatsheet button's title

* Update cheatsheet content

* Remove the old SQLite code, adapt the `--analyze` related code to Basenine

* Change the method signature of `NewEntry`

* Change the method signature of `Represent`

* Introduce `HTTPPair` field in `MizuEntry` specific to HTTP

* Remove `Entry`, `EntryId` and `EstimatedSizeBytes` fields from `MizuEntry`

Also remove the `getEstimatedEntrySizeBytes` method.

* Remove `gorm.io/gorm` dependency

* Remove unused `sensitiveDataFiltering` folder

* Increase the left margin of open cheatsheet button

* Add `overflow: auto` to the cheatsheet `Modal`

* Fix `GetEntry` method

* Fix the macro for gRPC

* Fix an interface conversion in case of AMQP

* Fix two more interface conversion errors in AMQP

* Make the `syncEntriesImpl` method blocking

* Fix a grammar mistake in the cheatsheet

* Adapt to the changes in the recent merge commit

* Improve the cheatsheet text

* Always display the timestamp in `en-US`

* Upgrade Basenine version to `0.2.2`

* Fix the order of closing Basenine connections and channels

* Don't close the Basenine channels at all

* Upgrade Basenine version to `0.2.3`

* Set the initial filter to `rlimit(100)`

* Make Basenine persistent

* Upgrade Basenine version to `0.2.4`

* Update `debug.Dockerfile`

* Fix a failing test

* Upgrade Basenine version to `0.2.5`

* Revert "Do not show play icon when disconnected (#428)"

This reverts commit 8af2e562f8.

* Upgrade Basenine version to `0.2.6`

* Make all non-informative things informative

* Make `100` a constant

* Use `===` in JavaScript no matter what

* Remove a forgotten `console.log`

* Add a comment and update the `query` in `syncEntriesImpl`

* Don't call `panic` in `GetEntry`

* Replace `panic` calls in `startBasenineServer` with `logger.Log.Panicf`

* Remove unnecessary `\n` characters in the logs
2021-11-09 19:54:48 +03:00
David Levanon
8a90f02161 Add support of listening to multiple netns (#418)
* multiple netns listen - initial commit

* multiple netns listen - actual work

* remove redundant log line

* map /proc of host to tapper

* changing kubernetes provider again after big conflict

* revert node-sass version back to 5.0.0

* Rename host_source to hostSource

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* PR fixes - adding comment + typos + naming conventions

* go fmt + making procfs read only

* setns back to the original value after packet source initialized

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
2021-11-07 16:00:59 +02:00
David Levanon
e635b97d11 remove duplicated tcp packet source (#405) 2021-10-26 18:16:33 +03:00
David Levanon
e696851261 Tapper Refactor (#396)
* introduce tcp_assembler and tcp_packet_source - the motivation is to … (#380)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* fix a typo

* remove unused pid param

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* extract stats functions out of the main tapping function (#381)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Feature/tapper refactor i/internal tapper stats (#384)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* introduce internal tapper stats

* minor fixes for errors map and internal stats

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Feature/tapper refactor i/diagnose package (#386)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* introduce internal tapper stats

* minor fixes for errors map and internal stats

* move errors map + app stats + internal stats + periodic tasks to diagnose package

* initialize tapper internal stats

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move tcp packet source to its packet (#387)

* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

* introduce tcp_assembler and tcp_packet_source - the motivation is to split the actual packet sources from the assembler, so we can have a single thread for the assembly which is separated from packet source threads

* make struts private at this point - planning to move some packages to their own package so we can utilize encapsulation

* extract stats functions out of the main tapping function

* move context to tcp_assembly + fix error check of tcp source packet

* use param instead of gloab flag for ipdefrag

* introduce internal tapper stats

* minor fixes for errors map and internal stats

* move errors map + app stats + internal stats + periodic tasks to diagnose package

* move tcp packet source to its packet

* initialize tapper internal stats

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* Fix coding style

* Remove `tap/internal_stats.go`

* make channel between input and assembler blocking - to preserve the same behaviour we have before the refactor

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
2021-10-25 13:59:06 +03:00
David Levanon
e2e69a3dc4 remove main dir (#385) 2021-10-20 13:48:53 +03:00
David Levanon
2944493e2d passive-tapper refactor - first phase
* add passive-tapper main tester (#353)

* add passive-tapper main tester

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* rename main to tester

* build extenssions as part of the tester launch

* add a README to the tester

* solving go.mod and .sum conflicts with addition of go-errors

* trivial warning fixes (#354)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* disable host mode for tester - to avoid filterAuthorities

* tcp streams map (#355)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* change rlog to mizu logger

* errors map (#356)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* change int to uint - errorsmap

* change from int to uint

* Change errorsMap.nErrors to uint.

* change errors map to mizu logger instead of rlog

* init mizu logger in tester + fix errormap declaration

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>

* move own ips to tcp stream factory (#358)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* Feature/tapper refactor i/move own ips to tcp stream factory (#379)

* add passive-tapper main tester

* trivial warning fixes

* add errors to go.sum of mizu agent

* tcp streams map

* disable host mode for tester - to avoid filterAuthorities

* set tcp streams map for tcp stream factory

* errors map

* move own ips to tcp stream factory

* fix ownips compilation issue

Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
2021-10-20 11:15:22 +03:00
RoyUP9
47f2e69b7e fixed entries port when fetching data from db (#376) 2021-10-19 17:37:57 +03:00
M. Mert Yıldıran
6240d85377 Bring back the lines that didn't meant to be removed (#375) 2021-10-19 16:22:47 +03:00
M. Mert Yıldıran
29ba963c48 Remove github.com/romana/rlog dependency completely (#374)
* Remove `github.com/romana/rlog` dependency completely

* Comment out all the unnecessary logging in the protocol extensions

* Remove commented out all the unnecessary logging lines

* Remove two more lines related to logging
2021-10-19 16:13:03 +03:00
M. Mert Yıldıran
145e7cda01 Add OAS contract monitoring support (#325)
* Add OAS contract monitoring support

* Pass the contract failure reason to UI

* Fix the issues related to contract validation

* Fix rest of the issues in the UI

* Add documentation related to contract monitoring feature

* Fix a typo in the docs

* Unmarshal to `HTTPRequestResponsePair` only if the OAS validation is enabled

* Fix an issue caused by the merge commit

* Slightly change the logic in the `validateOAS` method

Change the `contractText` value to `No Breaches` or `Breach` and make the text `white-space: nowrap`.

* Retrieve and display the failure reason for both request and response

Also display the content of the contract/OAS file in the UI.

* Display the OAS under `CONTRACT` tab with syntax highlighting

Also fix the styling in the entry feed.

* Remove `EnforcePolicyFileDeprecated` constant

* Log the other errors as well

* Get context from caller instead

* Define a type for the contract status and make its values enum-like

* Remove an unnecessary `if` statement

* Validate OAS in the CLI before passing it to Agent

* Get rid of the `github.com/ghodss/yaml` dependency in `loadOAS` by using `LoadFromData`

* Fix an artifact from the merge conflict
2021-10-19 14:24:22 +03:00
M. Mert Yıldıran
b7ff076571 Set the default log level for Agent to INFO and raise it to DEBUG if dump-logs=true is provided (#373)
* Set the default log level for Agent to `INFO` and change it to `DEBUG` if `dump-logs=true` is provided

* Remove `Trace` method and replace its calls with `Debug`

* Export logging levels from `logger` by defining functions

* Revert "Export logging levels from `logger` by defining functions"

This reverts commit e554e40f4a.

* Run `go mod tidy` on agent

* Define a method named `determineLogLevel`
2021-10-19 14:22:20 +03:00
M. Mert Yıldıran
58e9363fda Replace all rlog occurrences with the shared logger in tap (#369) 2021-10-18 16:35:42 +03:00
M. Mert Yıldıran
70982c2844 Fix the interface conversion errors in Redis (#334) 2021-10-10 08:34:47 +03:00
M. Mert Yıldıran
eb4a541376 Fix the interface conversion errors in Kafka (#333) 2021-10-08 07:35:20 +03:00
RoyUP9
8636a4731e fixed ignored user agents (#322) 2021-10-06 17:16:47 +03:00
Igor Gov
fd48cc6d87 Renaming ignored user agents var (#320) 2021-10-06 13:52:30 +03:00
M. Mert Yıldıran
26efaa101d Fix a 500 error caused by an interface conversion in Redis (#308) 2021-09-29 14:37:50 +03:00
M. Mert Yıldıran
352567c56e Fix the typo in protocolAbbreviation field of MizuEntry (#307) 2021-09-28 16:45:04 +03:00
M. Mert Yıldıran
cdf1c39a52 Omit the RULES tab if the policy rules feature is inactive (#303)
* Omit the `RULES` tab if the policy rules feature is inactive (WIP)

* Propagate the boolean value `isRulesEnabled` from file read error to UI

* Remove the debug log
2021-09-25 18:15:54 +03:00
M. Mert Yıldıran
7b333556d0 Add Redis Serialization Protocol support (#290)
* Bring in the files

* Add request-response pair matcher for Redis

* Implement the `Represent` method

* Update `representGeneric` method signature

* Don't export `IntToByteArr`

* Remove unused `newRedisInputStream` method

* Return the errors as string

* Adapt to the latest change in the `develop`
2021-09-23 17:09:00 +03:00
M. Mert Yıldıran
8ba96acf05 Don't omit the Latency field in BaseEntryDetails (#302) 2021-09-23 14:58:20 +03:00
M. Mert Yıldıran
649b733ba1 Don't redact :authority pseudo-header field (#300) 2021-09-23 09:12:04 +03:00
M. Mert Yıldıran
5392475486 Fix the issues related to sensitive data filtering feature (#285)
* Run acceptance tests on pull request

* Take `options.DisableRedaction` into account

* Log `defaultTapConfig`

* Pass the `SENSITIVE_DATA_FILTERING_OPTIONS` to tapper daemon set too

* Revert "Run acceptance tests on pull request"

This reverts commit ad79f1418f.
2021-09-19 13:33:34 +03:00
M. Mert Yıldıran
65bb262652 Fix the OOMKilled error by calling debug.FreeOSMemory periodically (#277)
* Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically

* Remove `MAX_NUMBER_OF_GOROUTINES` environment variable

* Change the line

* Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`
2021-09-19 12:31:09 +03:00
M. Mert Yıldıran
9fa9b67328 Bring back the sensitive data filtering feature (#280)
* Bring back the sensitive data filtering feature

* Add `// global` comment
2021-09-18 20:13:59 +03:00
Selton Fiuza
6337b75f0e [TRA-3659] Fix rules (#271)
* Fix rules

* Not reay, error on running

* Empty dissector Rules()

* almost working

* Finally, fixed

* undo changes on agent/pkg/utils/har.go

* fix not showing service on rules detail

* Update tap/api/api.go

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* Update agent/pkg/controllers/entries_controller.go

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* Update agent/pkg/controllers/entries_controller.go

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>

* unwrap Data

* Fix bug off using more than one latency rule that always get the first.

* fix json type, decoding base64 before unmarshal

* Run `go mod tidy` on `cli/go.sum`

* Fix the linting issues

* Remove a `FIXME` comment

* Remove a CSS rule

* Adapt `ruleNumberText` CSS class to the design language of the UI

* Fix an issue in the UI related to `rule.Latency` slipping out

* Removed unecessary codes.

Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
2021-09-18 14:02:18 -03:00
M. Mert Yıldıran
7dca1ad889 Move stats_tracker.go into the extension API and increment MatchedPairs from inside the Emit method (#272)
* Move `stats_tracker.go` into the extension API and increment `MatchedPairs` from inside the `Emit` method

* Replace multiple `sync.Mutex`(es) with low-level atomic memory primitives
2021-09-13 16:22:16 +03:00
M. Mert Yıldıran
616eccb2cf Make ScrollableFeed virtualized by replacing react-scrollable-feed with react-scrollable-feed-virtualized (#268)
* Make `ScrollableFeed` virtualized by replacing `react-scrollable-feed` with `react-scrollable-feed-virtualized`

* fix get new entries button

* Fix the not populated `Protocol` struct in case of `GetEntries` endpoint is called

Co-authored-by: Liraz Yehezkel <lirazy@up9.com>
2021-09-13 16:18:43 +03:00
M. Mert Yıldıran
30f07479cb Fix the JSON style to camelCase and rename CONTRIBUTE.md to CONTRIBUTING.md (#274)
* Fix the JSON style to camelCase and rename `CONTRIBUTE.md` to `CONTRIBUTING.md`

* Move `CONTRIBUTING.md` to `.github/` directory
2021-09-13 10:55:46 +03:00
M. Mert Yıldıran
858a64687d Stop the hanging Goroutines by dropping the old, unidentified TCP streams (#260)
* Close the hanging TCP message channels after a dynamically aligned timeout (base `10000` milliseconds)

* Bring back `source.Lazy`

* Add a one more `sync.Map.Delete` call

* Improve the formula by taking base Goroutine count into account

* Reduce duplication

* Include the dropped TCP streams count into the stats tracker and print a debug log whenever it happens

* Add `superIdentifier` field to `tcpStream` to check if it has identified

Also stop the other protocol dissectors if a TCP stream identified by a protocol.

* Take one step forward in fixing the channel closing issue (WIP)

Add `sync.Mutex` to `tcpReader` and make the loops reference based.

* Fix the channel closing issue

* Improve the accuracy of the formula, log better and multiply `baseStreamChannelTimeoutMs` by 100

* Remove `fmt.Printf`

* Replace `runtime.Gosched()` with `time.Sleep(1 * time.Millisecond)`

* Close the channels of other protocols in case of an identification

* Simplify the logic

* Replace the formula with hard timeout 5000 milliseconds and 4000 maximum number of Goroutines
2021-09-12 08:26:48 +03:00