* Run `go generate tls_tapper.go`
* Add `golang_uprobes.c`
* Add Golang hooks and offsets
* Add `golangConnection` struct and implement `pollGolangReadWrite` method
* Upgrade `github.com/cilium/ebpf` version to `v0.8.1`
* Fix the linter error
* Move map related stuff to `maps.h` and run `go generate tls_tapper.go`
* Remove unused parameter
* Add an environment variable to test Golang locally
* Replace `Libssl` occurrences with `Ssllib` for consistency
* Fix exe path finding
* Temporarily disable OpenSSL
* Fix the mixed offsets and dissection preparation
* Change the read symbol from `net/http.(*persistConn).Read` to `crypto/tls.(*Conn).Read`
* Remove `len` and `cap` fields
* Fix the indent
* Fix the read data address
* Make `golang_dial_writes` key `__u64` and include the PID
* Fix the read data address one more time
* Temporarily disable the PCAP capture
* Add a uprobe for `net/http.(*gzipReader).Read` to read chunked HTTP response body
* Cancel `golang_crypto_tls_read_uprobe` if it's a gzip read
* Make hash map names more meaningful
* Pass the connection address from `write` to `gzip` through a common address between `gzip` and `dial`
* Fix the probed line number links
* Add `golangReader` struct and implement its `Read` method
* Have a single counter pair and request response matcher per Golang connection
* Add `MIZU_GLOBAL_GOLANG_PATH` environment variable
* `NULL` terminate the bytes with `unix.ByteSliceToString`
* Temporarily reject the gzip chunks
* Add malformed TODOs
* Revert "`NULL` terminate the bytes with `unix.ByteSliceToString`"
This reverts commit 7ee7ef7e44.
* Bring back `len` and `cap` fields
* Set `len` and `cap` in `golang_net_http_gzipreader_read_uprobe` as well
* Remove two `TODO`s
* Fix the `key_gzip` offsets
* Compress if it's gzip chunk (probably wrong!)
* Revert "Compress if it's gzip chunk (probably wrong!)"
This reverts commit 094a7c3da4.
* Remove `golang_net_http_gzipreader_read_uprobe`
* Read constant 4KiB
* Use constant read length
* Get the correct len of bytes (saw the second entry)
* Set all buffer sizes to `CHUNK_SIZE`
* Remove a `TODO`
* Revert "Temporarily disable the PCAP capture"
This reverts commit a2da15ef2d.
* Update `golang_crypto_tls_read_uprobe`
* Set the `reader` field of `tlsStream` to fix a `nil pointer dereference` error
* Don't export any fields of `golangConnection`
* Close the reader when we drop the connection
* Add a tracepoint for `sys_enter_close` to detect socket closes
* Rename `socket` struct to `golang_socket`
* Call `should_tap` in Golang uprobes
* Add `log_error` calls
* Revert "Temporarily disable OpenSSL"
This reverts commit f54d9a453f.
* Fix linter
* Revert "Revert "Temporarily disable OpenSSL""
This reverts commit 2433d867af.
* Change `golang_read_writes` map type from `BPF_RINGBUF` to `BPF_PERF_OUTPUT`
* Rename `golang_read_write` to `golang_event`
* Define an error
* Add comments
* Revert "Revert "Revert "Temporarily disable OpenSSL"""
This reverts commit e5a1de9c71.
* Fix `pollGolang`
* Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""
This reverts commit 6e1bd5d4f3.
* Fix `panic: send on closed channel`
* Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""
This reverts commit 57d0584655.
* Use `findLibraryByPid`
* Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL""""""
This reverts commit 46f3d290b0.
* Revert "Revert "Revert "Revert "Revert "Revert "Revert "Temporarily disable OpenSSL"""""""
This reverts commit 775c833c06.
* Log tapping Golang
* Fix `Poll`
* Refactor `golang_net_http_dialconn_uprobe`
* Remove an excess error check
* Fix `can only use path@version syntax with 'go get' and 'go install' in module-aware mode` error in `tap/tlstapper/bpf-builder/build.sh`
* Unify Golang and OpenSSL under a single perf event buffer and `tls_chunk` struct
* Generate `tlsTapperChunkType` type (enum) as well
* Use kernel page size for the `sys_closes` perf buffer
* Fix the linter error
* Fix `MIZU_GLOBAL_GOLANG_PID` environment variable's functionality
* Rely on tracepoints for file descriptor retrieval in Golang implementation
* Remove the unnecessary changes
* Move common functions into `common.c`
* Declare `lookup_ssl_info` function to reduce duplication
* Fix linter
* Add comments and TODOs
* Remove `MIZU_GLOBAL_GOLANG_PATH` environment variable
* Update the object files
* Fix indentation
* Update object files
* Add `go_abi_internal.h`
* Fix `lookup_ssl_info`
* Convert indentation to spaces
* Add header guard comment
* Add more comments
* Find the `ret` instructions using Capstone Engine and `uprobe` the `return` statements
* Implement `get_fd_from_tcp_conn` function
* Separate SSL contexts to OpenSSL and Go
* Move `get_count_bytes` from `common.c` to `openssl_uprobes.c`
* Rename everything contains Golang to Go
* Reduce duplication in `go_uprobes.c`
* Update the comments
* Install Capstone in CI and Docker native builds
* Update `devops/install-capstone.sh`
* Add Capstone to AArch64 cross-compilation target
* Fix some of the issues on ARM64
* Delete the map element in `_ex_urpobe`
* Remove an unsued `LOG_` macro
* Rename `aquynh` to `capstone-engine`
* Add comment
* Revert "Fix some of the issues on ARM64"
This reverts commit 0b3eceddf4.
* Revert "Revert "Fix some of the issues on ARM64""
This reverts commit 681534ada1.
* Update object files
* Remove unnecessary return
* Increase timeout
* #run_acceptance_tests
* #run_acceptance_tests
* Fix the `arm64v8` sourced builds
* #run_acceptance_tests
* Call `SetProtocol` in AMQP faster and remove `GetProtocol` method
* #run_acceptance_tests
* Remove the unused fields from the test mocks #run_acceptance_tests
* Add gin-contrib/pprof dependency
* Run pprof server on agent with --profiler flag
* Add --profiler flag to cli
* Fix error message
* Print cpu usage percentage
* measure cpu of current pid instead of globaly on the system
* Add scripts to plot performance
* Plot packetsCount in analysis
* Concat to DataFrame
* Plot in turbo colorscheme
* Make COLORMAP const
* Fix rss units
* Reduce code repetition by adding function for plotting
* Allow grouping based on filenames
* Temporary: Marked with comments where to disable code for experiments
* Add newline at end of file
* Add tap.cpuprofile flag. Change memprofile flag to tap.memprofile
* create tapper modes for debugging using env vars
* Fix rss plot units (MB instead of bytes)
* Remove comment
* Add info to plot script
* Remove tap.cpumemprofile. Rename tap.memprofile to memprofile
* Remove unused import
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Remove whitespaces
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* Rename debug env vars
* Create package for debug env vars, read each env var once
* Run go mod tidy
* Increment MatchedPairs before emitting
* Only count cores once
* Count virtual and physical cores
* Add dbgctl replace in cli
* Fix lint: Check return values
* Add tap/dbgctl to test-lint make rule
* Replace tap/dbgctl in all modules
* #run_acceptance_tests
* Copy dbgctl module to docker image
* Debug/profile tapper benchmark (#1093)
* add mizu debug env to avoid all extensions
* add readme + run_tapper_benchmark.sh
* temporary change branch name
* fix readme
* fix MIZU_BENCHMARK_CLIENTS_COUNT env
* change tap target to tcp stream
* track live tcp streams
* pr fixes
* rename tapperPacketsCount to ignored_packets_count
* change mizu tapper to mizu debugg
Co-authored-by: David Levanon <dvdlevanon@gmail.com>
Co-authored-by: M. Mert Yıldıran <mehmet@up9.com>
* stop tapping self tapper traffic
* run go mod tidy
* allow to explicitly ignore ports
* remove unused code
* remove shared from tap + go mod tidy
* move ignroe ports to tapper
* rename TapperPacketsCount to IgnoredPacketsCount
* don't check null - go is smart
* remove nil check
* Fix a `nil pointer dereference` error that occurs in `tcpReader`'s `Read` method
* #run_acceptance_tests
* #run_acceptance_tests
* Revert "Fix a `nil pointer dereference` error that occurs in `tcpReader`'s `Read` method"
This reverts commit ccef6cb393.
* Fix the race condition using locks #run_acceptance_tests
* Spawn only two Goroutines per TCP stream
* Fix the linter error
* Use `isProtocolIdentified` method instead
* Fix the `Read` method of `tcpReader`
* Remove unnecessary `append`
* Copy to buffer only a message is received
* Remove `exhaustBuffer` field and add `rewind` function
* Rename `buffer` field to `pastData`
* Update tap/tcp_reader.go
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* Use `copy` instead of assignment
* No lint
* #run_acceptance_tests
* Fix `rewind` #run_acceptance_tests
* Fix the buffering algorithm #run_acceptance_tests
* Add `TODO`
* Fix the problems in AMQP and Kafka #run_acceptance_tests
* Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests
* Have a single `*bytes.Buffer`
* Revert "Have a single `*bytes.Buffer`"
This reverts commit fad96a288a.
* Revert "Use `*bytes.Buffer` instead of `[]api.TcpReaderDataMsg` #run_acceptance_tests"
This reverts commit 0fc70bffe2.
* Fix the early timing out issue #run_acceptance_tests
* Remove `NewBytes()` method
* Update the `NewTcpReader` method signature #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
* #run_acceptance_tests
Co-authored-by: Nimrod Gilboa Markevich <59927337+nimrod-up9@users.noreply.github.com>
* avoid chunks with invalid address
* tls tapper should distict between pids
* prettfy tls verbose log and tls key
* support tls from multi threads + duplicate calls to the same target
* introduce fdCache and user address pair as tls key
* remove unused comment
* fix merge conflicts
* use lru for fdcache
* pr fixes - renaming
* fix conflict issue
* Add GraphQL detection
* Change the abbreviation to `GQL`
* Fix the `TestMacros` test case
* Try to fix the security issues reported by Snyk
* Run `go mod tidy` in `agent` directory
* Upgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Downgrade `golang.org/x/crypto`
* Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2`
* Run `go mod tidy`
* Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns
* Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6`
* Don't upgrade `golang.org/x/net`
* Fix `panic: interface conversion: api.RequestResponseMatcher is nil, not *http.requestResponseMatcher` error
Also fix the request-response matcher maps iteration in `clean()` method.
* Fix the mocks in the unit tests
* Remove unnecessary fields from `tlsPoller` and implement `SetProtocol` method
* Use concrete types in `tap` package
* Share the streams map with the TLS tapper
* Check interface conversion error
* Remove `tcpStreamWrapper` struct
* Refactor `tap` module and move some of the code to `tap/api` module
* Move `TrafficFilteringOptions` struct to `shared` module
* Change the `Dissect` method signature to have `*TcpReader` as an argument
* Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors
* Run `go mod tidy` in `cli` module
* Rename `SuperIdentifier` struct to `ProtoIdentifier`
* Remove `SuperTimer` struct
* Bring back `CloseTimedoutTcpStreamChannels` method
* Run `go mod tidy` everywhere
* Remove `GOGC` environment variable from tapper
* Fix the tests
* Bring back `debug.FreeOSMemory()` call
* Make `CloseOtherProtocolDissectors` method mutexed
* Revert "Remove `GOGC` environment variable from tapper"
This reverts commit cfc2484bbb.
* Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags
* Define a bunch of interfaces and don't export any new structs from `tap/api`
* Keep the interfaces in `tap/api` but move the structs to `tap/tcp`
* Fix the unit tests by depending on `github.com/up9inc/mizu/tap`
* Use the modified `tlsEmitter`
* Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`
* Remove unused fields in `tlsReader`
* Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct
Such that make `tap/api` don't depend on `gopacket`
* Remove the unused fields
* Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method
* Remove unused fields from `tlsPoller`
* Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface
* Revert "Revert "Remove `GOGC` environment variable from tapper""
This reverts commit ab2b9a803b.
* Revert "Bring back `debug.FreeOSMemory()` call"
This reverts commit 1cce863bbb.
* Remove excess comment
* Fix acceptance tests (`logger` module) #run_acceptance_tests
* Bring back `github.com/patrickmn/go-cache`
* Fix `NewTcpStream` method signature
* Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency
* Fix AMQP tests
* Revert 960ba644cd
* Revert `go.mod` and `go.sum` files in protocol dissectors
* Fix the comment position
* Revert `AppStatsInst` change
* Fix indent
* Fix CLI build
* Fix linter error
* Fix error msg
* Revert some of the changes in `chunk.go`
* Remove non-critical TLS detected log that causes `slice bounds out of range` error
* Remove all non-functional `OutboundLink` code that was providing `/status/recentTLSLinks` endpoint
* Fix more unused code
* Increase the OOM risk in exchange of less idle CPU usage
* Read the interval from an environment variable named `CLOSE_TIMEDOUT_TCP_STREAM_CHANNELS_INTERVAL_MS`
* Log the `getCloseTimedoutTcpChannelsInterval` return value
* Basenine MongoDB mess
* Fix more
* Fix the `mongodb` container arguments
* Add Basenine ARM64 binary
* Make the changes related to `leftOff` becoming a string value
* Make `leftOffTop` state string
* Handle `CloseConnection` in `Fetch`
* Upgrade Basenine to `0.7.0`
* Revert the changes in `package.json` and `package-lock.json`
* Fix the `Dockerfile`
* Remove the binaries
* Increase the Basenine up deadline to 20 seconds
* Revert the changes in `shared/kubernetes/provider.go`
* Fix the OAS generator tests
* Protect from race condition
* Fix mutexes
* Fix unlock
* Fix logging data types
* Try to stabilize the tests
* Remove the `replace` statement
* revert the change the done in 2899414f2b to not change the leftOff
* Change `leftOffBottom` empty string default value to `latest`
* Upgrade Basenine to `0.7.1`
* Handle the Basenine client library errors better
* Use `DEFAULT_QUERY` constant
* Remove `min=-1`
* Replace some `Errorf`s with `Panicf`s
* Remove the closure in `runGenerator` method
* Remove an unnecessary check
Co-authored-by: M. Mert Yildiran <mehmet@up9.com>
Co-authored-by: Andrey Pokhilko <apc4@ya.ru>
Co-authored-by: undera <undera@undera-old-desktop.home>
Co-authored-by: AmitUp9 <96980485+AmitUp9@users.noreply.github.com>
* Remove gRPC related modifications
* Remove gRPC status text related modifications as well
* Fixing gRPC vertical image
detect grpc when content type is 'application/grpc' as well (and not only from the grpc-status)
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Update tappers via websocket instead of by env var. This way the DaemonSet doesn't have to be applied just to notify the tappers that the tap targets changed. The number of tapper restarts is reduced. The DaemonSet still gets applied when there is a need to add/remove a tapper from a node.
* Define `ReadProgress` struct and update `Dissector` interface such that the `bufio.Reader` progress can be learned on item emitting
* Display the `requestSize` and `responseSize` fields in the UI
* Update the tests
* publish ui-common version 1.0.130 and bump to this version in ui/package.json file
Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>
Co-authored-by: Roee Gadot <roee.gadot@up9.com>
* Define `Capture` type and expect it as an argument in `Dissect` method
* Add an indicator for the eBPF sourced entries
* Fix the Go lint error
* Fix the logic in the UI
* Update the expected JSONs
* Add TODOs
* Add `UndefinedCapture` constant
* Define `CaptureTypes` enum
* Always derive the summary and method fields from the entry in the database on read
* Update the expected JSONs in the protocol unit tests
* Add test cases for `Summarize` method
* Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method
* Temporarily enable the acceptance tests
* Temporarily disable Slack notification on failure
* Update the Cypress tests
* Fix an issue in Redis
* Fix a typo and the Cypress tests
* Revert "Temporarily disable Slack notification on failure"
This reverts commit cad1901ea4.
* Revert "Temporarily enable the acceptance tests"
This reverts commit bad7706c9b.
