mirror of
https://github.com/kubeshark/kubeshark.git
synced 2026-04-19 00:49:30 +00:00
366c1d0c6c
* Separate HTTP related code into `extensions/http` as a Go plugin * Move `extensions` folder into `tap` folder * Move HTTP files into `tap/extensions/lib` for now * Replace `orcaman/concurrent-map` with `sync.Map` * Remove `grpc_assembler.go` * Remove `github.com/up9inc/mizu/tap/extensions/http/lib` * Add a build script to automatically build extensions from a known path and load them * Start to define the extension API * Implement the `run()` function for the TCP stream * Add support of defining multiple ports to the extension API * Set the extension name inside the extension * Declare the `Dissect` function in the extension API * Dissect HTTP request from inside the HTTP extension * Make the distinction of outbound and inbound ports * Dissect HTTP response from inside the HTTP extension * Bring back the HTTP request-response pair matcher * Return a `*api.RequestResponsePair` from the dissection * Bring back the gRPC-HTTP/2 parser * Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream` * Call a function pointer to emit dissected data back to the `tap` package * roee changes - trying to fix agent to work with the "api" object) - ***still not working*** * small mistake in the conflicts * Fix the issues that are introduced by the merge conflict * Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel` * Fix the `HTTP1` handlers * Set `ConnectionInfo` in HTTP handlers * Fix the `Dockerfile` to build the extensions * remove some unwanted code * no message * Re-enable `getStreamProps` function * Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly` * Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol * Read `socketHarOutChannel` instead of `filteredHarChannel` * Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again * Add `.env.example` to React app * Marshal and unmarshal `*http.Request`, `*http.Response` pairs * Move `loadExtensions` into `main.go` and map extensions into `extensionsMap` * Add `Summarize()` method to the `Dissector` interface * Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API * Add `Protocol` struct and make it effect the UI * Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI * Display the protocol name inside the details layout * Add `Represent` method to the `Dissector` interface and manipulate the UI through this method * Make the protocol color affect the details layout color and write protocol abbreviation vertically * Remove everything HTTP related from the `tap` package and make the extension system fully functional * Fix the TypeScript warnings * Bring in the files related AMQP into `amqp` directory * Add `--nodefrag` flag to the tapper and bring in the main AMQP code * Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing * Implement `representBasicPublish` method * Fix several minor issues * Implement the AMQP `BasicDeliver` * Implement the AMQP `QueueDeclare` * Implement the AMQP `ExchangeDeclare` * Implement the AMQP `ConnectionStart` * Implement the AMQP `ConnectionClose` * Implement the AMQP `QueueBind` * Implement the AMQP `BasicConsume` * Fix an issue in `ConnectionStart` * Fix a linter error * Bring in the files related Kafka into `kafka` directory * Fix the build errors in Kafka Go files * Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream * Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method * Implement `Analyze`, `Summarize` methods of Kafka * Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader` * Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method * Implement the representations for Kafka `ApiVersions` * Implement the representations for Kafka `Produce` * Implement the representations for Kafka `Fetch` * Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics` * Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body * Remove the unnecessary logging * Remove more logging * Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2 * Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol * Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol * Fix the read errors that freezes the sniffer in HTTP and Kafka * Fix the issues in HTTP POST data * Fix one more issue in HTTP POST data * Fix an infinite loop in Kafka * Fix another freezing issue in Kafka * Revert "UI Infra - Support multiple entry types + refactoring (#211)" This reverts commitf74a52d4dc. * Fix more issues that are introduced by the merge * Fix the status code in the summary section * adding the cleaner again (why we removed it?). add TODO: on the extension loop . * fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists * fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared) * Fix an issue in the UI summary section * Refactor the protocol payload structs * Fix a log message in the passive tapper * Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' ` * Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists" This reverts commit4f514ae1f4. * Bring in the necessary changes fromf74a52d4dc* Open the API server URL in the web browser as soon as Mizu is ready * Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding * Swap `TcpID` without overwriting it * Sort extension by priority * Try to dissect with looping through all the extensions * fix getStreamProps function. (it should be passed from CLI as it was before). * Turn TCP reader back into two Goroutines (client and server) * typo * Learn `isClient` from the TCP stream * Set `viewer` style `overflow: "auto"` * Fix the memory leaks in AMQP and Kafka dissectors * Revert some of the changes inbe7c65eb6d* Remove `allExtensionPorts` since it's no longer needed * Remove `APP_PORTS` since it's no longer needed * Fix all of the minor issues in the React code * Check Kafka header size and fail-fast * Break the dissectors loop upon a successful dissection * Don't break the dissector loop. Protocols might collide * Improve the HTTP request-response counter (still not perfect) * Make the HTTP request-response counter perfect * Revert "Revert some of the changes in be7c65eb6d3fb657a059707da3ca559937e59739" This reverts commit08e7d786d8. * Bring back `filterItems` and `isHealthCheckByUserAgent` functions * Remove some development artifacts * remove unused and commented lines that are not relevant * Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension * Change a log to debug * Make `*api.CounterPair` a field of `tcpReader` * Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems * Remove a variable that's only used for logging even though not introduced by this branch * Bring back the `NumberOfRules` field of `ApplicableRules` struct * Remove the unused `NewEntry` function * Move `k8sResolver == nil` check to a more appropriate place * default healthChecksUserAgentHeaders should be empty array (like the default config value) * remove spam console.log * Rules button cause app to crash (access the service via incorrect property) * Ignore all .env* files in docker build. * Better caching in dockerfile: only copy go.mod before go mod download. * Check for errors while loading an extension * Add a comment about why `Protocol` is not a pointer * Bring back the call to `deleteOlderThan` * Remove the `nil` check * Reduce the maximum allowed AMQP message from 128MB to 1MB * Fix an error that only occurs when a Kafka broker is initiating * Revert the change inb2abd7b990* Fix the service name resolution in all protocols * Remove the `anydirection` flag and fix the issue in `filterAuthorities` * Pass `sync.Map` by reference to `deleteOlderThan` method * Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection` * Temporarily resolve the memory exhaustion in AMQP * Fix a nil pointer dereference error * Fix the CLI build error * Fix a memory leak that's identified by `pprof` Co-authored-by: Roee Gadot <roee.gadot@up9.com> Co-authored-by: Nimrod Gilboa Markevich <nimrod@up9.com>
174 lines
5.6 KiB
Markdown
174 lines
5.6 KiB
Markdown
|
|
|
|
# The API Traffic Viewer for Kubernetes
|
|
|
|
A simple-yet-powerful API traffic viewer for Kubernetes enabling you to view all API communication between microservices to help your debug and troubleshoot regressions.
|
|
|
|
Think TCPDump and Chrome Dev Tools combined.
|
|
|
|
|
|
|
|
## Features
|
|
|
|
- Simple and powerful CLI
|
|
- Real-time view of all HTTP requests, REST and gRPC API calls
|
|
- No installation or code instrumentation
|
|
- Works completely on premises
|
|
|
|
## Download
|
|
|
|
Download Mizu for your platform and operating system
|
|
|
|
### Latest Stable Release
|
|
|
|
* for MacOS - Intel
|
|
```
|
|
curl -Lo mizu \
|
|
https://github.com/up9inc/mizu/releases/latest/download/mizu_darwin_amd64 \
|
|
&& chmod 755 mizu
|
|
```
|
|
|
|
* for Linux - Intel 64bit
|
|
```
|
|
curl -Lo mizu \
|
|
https://github.com/up9inc/mizu/releases/latest/download/mizu_linux_amd64 \
|
|
&& chmod 755 mizu
|
|
```
|
|
|
|
SHA256 checksums are available on the [Releases](https://github.com/up9inc/mizu/releases) page
|
|
|
|
### Development (unstable) Build
|
|
Pick one from the [Releases](https://github.com/up9inc/mizu/releases) page
|
|
|
|
## Kubeconfig & Permissions
|
|
While `mizu`most often works out of the box, you can influence its behavior:
|
|
|
|
1. [OPTIONAL] Set `KUBECONFIG` environment variable to your Kubernetes configuration. If this is not set, Mizu assumes that configuration is at `${HOME}/.kube/config`
|
|
2. `mizu` assumes user running the command has permissions to create resources (such as pods, services, namespaces) on your Kubernetes cluster (no worries - `mizu` resources are cleaned up upon termination)
|
|
|
|
For detailed list of k8s permissions see [PERMISSIONS](PERMISSIONS.md) document
|
|
|
|
|
|
## How to Run
|
|
|
|
1. Find pods you'd like to tap to in your Kubernetes cluster
|
|
2. Run `mizu tap` or `mizu tap PODNAME`
|
|
3. Open browser on `http://localhost:8899/mizu` **or** as instructed in the CLI
|
|
4. Watch the API traffic flowing
|
|
5. Type ^C to stop
|
|
|
|
## Examples
|
|
|
|
Run `mizu help` for usage options
|
|
|
|
To tap all pods in current namespace -
|
|
```
|
|
$ kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
carts-66c77f5fbb-fq65r 2/2 Running 0 20m
|
|
catalogue-5f4cb7cf5-7zrmn 2/2 Running 0 20m
|
|
front-end-649fc5fd6-kqbtn 2/2 Running 0 20m
|
|
..
|
|
|
|
$ mizu tap
|
|
+carts-66c77f5fbb-fq65r
|
|
+catalogue-5f4cb7cf5-7zrmn
|
|
+front-end-649fc5fd6-kqbtn
|
|
Web interface is now available at http://localhost:8899
|
|
^C
|
|
```
|
|
|
|
|
|
To tap specific pod -
|
|
```bash
|
|
$ kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
front-end-649fc5fd6-kqbtn 2/2 Running 0 7m
|
|
..
|
|
|
|
$ mizu tap front-end-649fc5fd6-kqbtn
|
|
+front-end-649fc5fd6-kqbtn
|
|
Web interface is now available at http://localhost:8899
|
|
^C
|
|
```
|
|
|
|
To tap multiple pods using regex -
|
|
```bash
|
|
$ kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
carts-66c77f5fbb-fq65r 2/2 Running 0 20m
|
|
catalogue-5f4cb7cf5-7zrmn 2/2 Running 0 20m
|
|
front-end-649fc5fd6-kqbtn 2/2 Running 0 20m
|
|
..
|
|
|
|
$ mizu tap "^ca.*"
|
|
+carts-66c77f5fbb-fq65r
|
|
+catalogue-5f4cb7cf5-7zrmn
|
|
Web interface is now available at http://localhost:8899
|
|
^C
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Mizu can work with config file which should be stored in ${HOME}/.mizu/config.yaml (macOS: ~/.mizu/config.yaml) <br />
|
|
In case no config file found, defaults will be used <br />
|
|
In case of partial configuration defined, all other fields will be used with defaults <br />
|
|
You can always override the defaults or config file with CLI flags
|
|
|
|
To get the default config params run `mizu config` <br />
|
|
To generate a new config file with default values use `mizu config -r`
|
|
|
|
### Telemetry
|
|
|
|
By default, mizu reports usage telemetry. It can be disabled by adding a line of `telemetry: false` in the `${HOME}/.mizu/config.yaml` file
|
|
|
|
|
|
## Advanced Usage
|
|
|
|
### Namespace-Restricted Mode
|
|
|
|
Some users have permission to only manage resources in one particular namespace assigned to them
|
|
By default `mizu tap` creates a new namespace `mizu` for all of its Kubernetes resources. In order to instead install
|
|
Mizu in an existing namespace, set the `mizu-resources-namespace` config option
|
|
|
|
If `mizu-resources-namespace` is set to a value other than the default `mizu`, Mizu will operate in a
|
|
Namespace-Restricted mode. It will only tap pods in `mizu-resources-namespace`. This way Mizu only requires permissions
|
|
to the namespace set by `mizu-resources-namespace`. The user must set the tapped namespace to the same namespace by
|
|
using the `--namespace` flag or by setting `tap.namespaces` in the config file
|
|
|
|
Setting `mizu-resources-namespace=mizu` resets Mizu to its default behavior
|
|
|
|
### User agent filtering
|
|
|
|
User-agent filtering (like health checks) - can be configured using command-line options:
|
|
|
|
```shell
|
|
$ mizu tap "^ca.*" --set ignored-user-agents=kube-probe --set ignored-user-agents=prometheus
|
|
+carts-66c77f5fbb-fq65r
|
|
+catalogue-5f4cb7cf5-7zrmn
|
|
Web interface is now available at http://localhost:8899
|
|
^C
|
|
|
|
```
|
|
Any request that contains `User-Agent` header with one of the specified values (`kube-probe` or `prometheus`) will not be captured
|
|
|
|
### API Rules validation
|
|
|
|
This feature allows you to define set of simple rules, and test the API against them.
|
|
Such validation may test response for specific JSON fields, headers, etc.
|
|
|
|
Please see [API RULES](docs/POLICY_RULES.md) page for more details and syntax.
|
|
|
|
|
|
## How to Run local UI
|
|
|
|
- run from mizu/agent `go run main.go --hars-read --hars-dir <folder>`
|
|
|
|
- copy Har files into the folder from last command
|
|
|
|
- change `MizuWebsocketURL` and `apiURL` in `api.js` file
|
|
|
|
- run from mizu/ui - `npm run start`
|
|
|
|
- open browser on `localhost:3000`
|