Feature/fix tls not listening (#1046 )

* avoid chunks with invalid address * tls tapper should distict between pids * prettfy tls verbose log and tls key * support tls from multi threads + duplicate calls to the same target * introduce fdCache and user address pair as tls key * remove unused comment * fix merge conflicts * use lru for fdcache * pr fixes - renaming * fix conflict issue
Bring back debug.FreeOSMemory() call to prevent OOMKilled(s) (#1063 )
2026-02-16 19:10:17 +00:00 · 2022-05-02 21:33:26 +03:00 · 2022-05-02 14:13:04 +03:00 · 2022-05-02 13:24:28 +03:00 · 2022-05-02 11:16:05 +03:00 · 2022-05-02 11:06:01 +03:00
40 changed files with 397 additions and 142 deletions
--- a/.github/workflows/static_code_analysis.yml
+++ b/.github/workflows/static_code_analysis.yml
@@ -10,7 +10,7 @@ permissions:
  contents: read

 jobs:
-  golangci:
+  go-lint:
    name: Go lint
    runs-on: ubuntu-latest
    steps:
@@ -141,7 +141,26 @@ jobs:
        with:
          version: latest
          working-directory: tap/extensions/redis
-      
+
+      - name: Check logger modified files
+        id: logger_modified_files
+        run: devops/check_modified_files.sh logger/
+
+      - name: Go lint - logger
+        uses: golangci/golangci-lint-action@v2
+        if: steps.logger_modified_files.outputs.matched == 'true'
+        with:
+          version: latest
+          working-directory: logger
+
+  es-lint:
+    name: ES lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 2
+
      - uses: actions/setup-node@v2
        with:
          node-version: 16
--- a/10
+++ b/10
@@ -44,11 +44,18 @@ ENV CGO_ENABLED=1 GOOS=linux
 ENV GOARCH=arm64 CGO_CFLAGS="-I/work/libpcap"


+### Builder image for AArch64 to x86-64 cross-compilation
+FROM up9inc/linux-x86_64-musl-go-libpcap AS builder-from-arm64v8-to-amd64
+ENV CGO_ENABLED=1 GOOS=linux
+ENV GOARCH=amd64 CGO_CFLAGS="-I/libpcap"
+
+
 ### Final builder image where the build happens
 # Possible build strategies:
 # BUILDARCH=amd64 TARGETARCH=amd64
 # BUILDARCH=arm64v8 TARGETARCH=arm64v8
 # BUILDARCH=amd64 TARGETARCH=arm64v8
+# BUILDARCH=arm64v8 TARGETARCH=amd64
 ARG BUILDARCH=amd64
 ARG TARGETARCH=amd64
 FROM builder-from-${BUILDARCH}-to-${TARGETARCH} AS builder
@@ -66,9 +73,6 @@ COPY tap/extensions/http/go.mod ../tap/extensions/http/
 COPY tap/extensions/kafka/go.mod ../tap/extensions/kafka/
 COPY tap/extensions/redis/go.mod ../tap/extensions/redis/
 RUN go mod download
-# cheap trick to make the build faster (as long as go.mod did not change)
-RUN go get github.com/patrickmn/go-cache
-RUN go list -f '{{.Path}}@{{.Version}}' -m all | sed 1d | grep -e 'go-cache' | xargs go get

 # Copy and build agent code
 COPY shared ../shared
--- a/agent/go.mod
+++ b/agent/go.mod
@@ -76,6 +76,7 @@ require (
 	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
 	github.com/googleapis/gnostic v0.5.5 // indirect
 	github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
+	github.com/hashicorp/golang-lru v0.5.4 // indirect
 	github.com/imdario/mergo v0.3.12 // indirect
 	github.com/inconshreveable/mousetrap v1.0.0 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
@@ -85,6 +86,7 @@ require (
 	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
 	github.com/mattn/go-isatty v0.0.14 // indirect
+	github.com/mertyildiran/gqlparser/v2 v2.4.6 // indirect
 	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
 	github.com/moby/spdystream v0.2.0 // indirect
 	github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 // indirect
@@ -110,7 +112,7 @@ require (
 	github.com/vishvananda/netns v0.0.0-20211101163701-50045581ed74 // indirect
 	github.com/xlab/treeprint v1.1.0 // indirect
 	go.starlark.net v0.0.0-20220203230714-bb14e151c28f // indirect
-	golang.org/x/crypto v0.0.0-20220208050332-20e1d8d225ab // indirect
+	golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b // indirect
 	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect
 	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
 	golang.org/x/sys v0.0.0-20220207234003-57398862261d // indirect
--- a/agent/go.sum
+++ b/agent/go.sum
@@ -83,6 +83,7 @@ github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tN
 github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
 github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
 github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
@@ -404,6 +405,7 @@ github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/b
 github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
 github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
 github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
@@ -493,6 +495,8 @@ github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27k
 github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
+github.com/mertyildiran/gqlparser/v2 v2.4.6 h1:enAq4F5PYgW/rYExPNzYt7IYrrZnzrfqdywMA1QdFtM=
+github.com/mertyildiran/gqlparser/v2 v2.4.6/go.mod h1:XZId58F+XqRSmoLrdsOLgqA918oNvBzuOORruJWBjDo=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
 github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso=
 github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI=
@@ -751,8 +755,9 @@ golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5y
 golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
-golang.org/x/crypto v0.0.0-20220208050332-20e1d8d225ab h1:lnZ4LoV0UMdibeCUfIB2a4uFwRu491WX/VB2reB8xNc=
 golang.org/x/crypto v0.0.0-20220208050332-20e1d8d225ab/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b h1:Qwe1rC8PSniVfAFPFJeyUkB+zcysC3RgJBAGk7eqBEU=
+golang.org/x/crypto v0.0.0-20220314234659-1baeb1ce4c0b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
--- a/agent/pkg/resolver/go.sum
+++ b/agent/pkg/resolver/go.sum
@@ -219,8 +219,6 @@ github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v1.7.0 h1:XPnZz8VVBHjVsy1vzJmRwIcSwiUO+JFfrv/xGiigmME=
 github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
-github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
-github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
 github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
--- a/devops/linux-x86_64-musl-go-libpcap/Dockerfile
+++ b/devops/linux-x86_64-musl-go-libpcap/Dockerfile
@@ -0,0 +1,31 @@
+FROM messense/rust-musl-cross:x86_64-musl AS builder-from-arm64v8-to-amd64
+
+ENV CROSS_TRIPLE x86_64-unknown-linux-musl
+ENV CROSS_ROOT /usr/local/musl
+
+ENV AS=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-as \
+    AR=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-ar \
+    CC=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-gcc \
+    CPP=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-cpp \
+    CXX=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-g++ \
+    LD=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-ld \
+    FC=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-gfortran
+
+# Install Go
+WORKDIR /
+RUN curl https://go.dev/dl/go1.17.6.linux-arm64.tar.gz -Lo ./go.linux-arm64.tar.gz \
+ && curl https://go.dev/dl/go1.17.6.linux-arm64.tar.gz.asc -Lo ./go.linux-arm64.tar.gz.asc \
+ && curl https://dl.google.com/dl/linux/linux_signing_key.pub  -Lo linux_signing_key.pub \
+ && gpg --import linux_signing_key.pub && gpg --verify ./go.linux-arm64.tar.gz.asc ./go.linux-arm64.tar.gz \
+ && rm -rf /usr/local/go && tar -C /usr/local -xzf go.linux-arm64.tar.gz
+ENV PATH "$PATH:/usr/local/go/bin"
+
+# Compile libpcap from source
+RUN curl https://www.tcpdump.org/release/libpcap-1.10.1.tar.gz -Lo ./libpcap.tar.gz \
+ && curl https://www.tcpdump.org/release/libpcap-1.10.1.tar.gz.sig -Lo ./libpcap.tar.gz.sig \
+ && curl https://www.tcpdump.org/release/signing-key.asc -Lo ./signing-key.asc \
+ && gpg --import signing-key.asc && gpg --verify libpcap.tar.gz.sig libpcap.tar.gz \
+ && tar -xzf libpcap.tar.gz && mv ./libpcap-* ./libpcap
+WORKDIR /libpcap
+RUN ./configure --host=x86_64 && make \
+ && cp /libpcap/libpcap.a /usr/local/musl/lib/gcc/x86_64-unknown-linux-musl/*/
--- a/devops/linux-x86_64-musl-go-libpcap/build-push.sh
+++ b/devops/linux-x86_64-musl-go-libpcap/build-push.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+set -e
+
+docker build . -t up9inc/linux-x86_64-musl-go-libpcap && docker push up9inc/linux-x86_64-musl-go-libpcap
--- a/tap/api/api.go
+++ b/tap/api/api.go
@@ -426,7 +426,7 @@ type TcpStream interface {
 	SetProtocol(protocol *Protocol)
 	GetOrigin() Capture
 	GetProtoIdentifier() *ProtoIdentifier
-	GetReqResMatcher() RequestResponseMatcher
+	GetReqResMatchers() []RequestResponseMatcher
 	GetIsTapTarget() bool
 	GetIsClosed() bool
 }
--- a/tap/api/go.mod
+++ b/tap/api/go.mod
@@ -3,7 +3,3 @@ module github.com/up9inc/mizu/tap/api
 go 1.17

 require github.com/google/martian v2.1.0+incompatible
-
-replace github.com/up9inc/mizu/logger v0.0.0 => ../../logger
-
-replace github.com/up9inc/mizu/shared v0.0.0 => ../../shared
--- a/tap/cleaner.go
+++ b/tap/cleaner.go
@@ -34,12 +34,14 @@ func (cl *Cleaner) clean() {
 	cl.assemblerMutex.Unlock()

 	cl.streamsMap.Range(func(k, v interface{}) bool {
-		reqResMatcher := v.(api.TcpStream).GetReqResMatcher()
-		if reqResMatcher == nil {
-			return true
+		reqResMatchers := v.(api.TcpStream).GetReqResMatchers()
+		for _, reqResMatcher := range reqResMatchers {
+			if reqResMatcher == nil {
+				continue
+			}
+			deleted := deleteOlderThan(reqResMatcher.GetMap(), startCleanTime.Add(-cl.connectionTimeout))
+			cl.stats.deleted += deleted
 		}
-		deleted := deleteOlderThan(reqResMatcher.GetMap(), startCleanTime.Add(-cl.connectionTimeout))
-		cl.stats.deleted += deleted
 		return true
 	})

--- a/tap/extensions/amqp/go.mod
+++ b/tap/extensions/amqp/go.mod
@@ -15,5 +15,3 @@ require (
 )

 replace github.com/up9inc/mizu/tap/api v0.0.0 => ../../api
-
-replace github.com/up9inc/mizu/logger v0.0.0 => ../../../logger
--- a/tap/extensions/amqp/tcp_stream_mock_test.go
+++ b/tap/extensions/amqp/tcp_stream_mock_test.go
@@ -11,7 +11,7 @@ type tcpStream struct {
 	protoIdentifier *api.ProtoIdentifier
 	isTapTarget     bool
 	origin          api.Capture
-	reqResMatcher   api.RequestResponseMatcher
+	reqResMatchers  []api.RequestResponseMatcher
 	sync.Mutex
 }

@@ -32,8 +32,8 @@ func (t *tcpStream) GetProtoIdentifier() *api.ProtoIdentifier {
 	return t.protoIdentifier
 }

-func (t *tcpStream) GetReqResMatcher() api.RequestResponseMatcher {
-	return t.reqResMatcher
+func (t *tcpStream) GetReqResMatchers() []api.RequestResponseMatcher {
+	return t.reqResMatchers
 }

 func (t *tcpStream) GetIsTapTarget() bool {
--- a/tap/extensions/http/go.mod
+++ b/tap/extensions/http/go.mod
@@ -4,6 +4,7 @@ go 1.17

 require (
 	github.com/beevik/etree v1.1.0
+	github.com/mertyildiran/gqlparser/v2 v2.4.6
 	github.com/stretchr/testify v1.7.0
 	github.com/up9inc/mizu/tap/api v0.0.0
 	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
@@ -18,5 +19,3 @@ require (
 )

 replace github.com/up9inc/mizu/tap/api v0.0.0 => ../../api
-
-replace github.com/up9inc/mizu/logger v0.0.0 => ../../../logger
--- a/tap/extensions/http/go.sum
+++ b/tap/extensions/http/go.sum
@@ -1,3 +1,4 @@
+github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM=
 github.com/beevik/etree v1.1.0 h1:T0xke/WvNtMoCqgzPhkX2r4rjY3GDZFi+FjpRZY2Jbs=
 github.com/beevik/etree v1.1.0/go.mod h1:r8Aw8JqVegEf0w2fDnATrX9VpkMcyFeM0FhwO62wh+A=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -5,6 +6,13 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/mertyildiran/gqlparser/v2 v2.4.6 h1:enAq4F5PYgW/rYExPNzYt7IYrrZnzrfqdywMA1QdFtM=
+github.com/mertyildiran/gqlparser/v2 v2.4.6/go.mod h1:XZId58F+XqRSmoLrdsOLgqA918oNvBzuOORruJWBjDo=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -18,8 +26,10 @@ golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuX
 golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
+gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo=
 gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
--- a/tap/extensions/http/graphql.go
+++ b/tap/extensions/http/graphql.go
@@ -0,0 +1,31 @@
+package http
+
+import (
+	"encoding/json"
+
+	"github.com/mertyildiran/gqlparser/v2/ast"
+	"github.com/mertyildiran/gqlparser/v2/parser"
+)
+
+func isGraphQL(request map[string]interface{}) bool {
+	if postData, ok := request["postData"].(map[string]interface{}); ok {
+		if postData["mimeType"] == "application/json" {
+			if text, ok := postData["text"].(string); ok {
+				var data map[string]interface{}
+				if err := json.Unmarshal([]byte(text), &data); err != nil {
+					return false
+				}
+
+				if query, ok := data["query"].(string); ok {
+
+					_, err := parser.ParseQuery(&ast.Source{Name: "ff", Input: query})
+					if err == nil {
+						return true
+					}
+				}
+			}
+		}
+	}
+
+	return false
+}
--- a/tap/extensions/http/main.go
+++ b/tap/extensions/http/main.go
@@ -71,6 +71,34 @@ var grpcProtocol api.Protocol = api.Protocol{
 	Priority:        0,
 }

+var graphQL1Protocol api.Protocol = api.Protocol{
+	Name:            "http",
+	LongName:        "Hypertext Transfer Protocol -- HTTP/1.1 [ GraphQL over HTTP/1.1 ]",
+	Abbreviation:    "GQL",
+	Macro:           "gql",
+	Version:         "1.1",
+	BackgroundColor: "#e10098",
+	ForegroundColor: "#ffffff",
+	FontSize:        12,
+	ReferenceLink:   "https://graphql.org/learn/serving-over-http/",
+	Ports:           []string{"80", "443", "8080"},
+	Priority:        0,
+}
+
+var graphQL2Protocol api.Protocol = api.Protocol{
+	Name:            "http",
+	LongName:        "Hypertext Transfer Protocol Version 2 (HTTP/2) [ GraphQL over HTTP/2 ]",
+	Abbreviation:    "GQL",
+	Macro:           "gql",
+	Version:         "2.0",
+	BackgroundColor: "#e10098",
+	ForegroundColor: "#ffffff",
+	FontSize:        12,
+	ReferenceLink:   "https://graphql.org/learn/serving-over-http/",
+	Ports:           []string{"80", "443", "8080", "50051"},
+	Priority:        0,
+}
+
 const (
 	TypeHttpRequest = iota
 	TypeHttpResponse
@@ -208,6 +236,14 @@ func (d dissecting) Analyze(item *api.OutputChannelItem, resolvedSource string,
 		}
 	}

+	if isGraphQL(reqDetails) {
+		if item.Protocol.Version == "2.0" {
+			item.Protocol = graphQL2Protocol
+		} else {
+			item.Protocol = graphQL1Protocol
+		}
+	}
+
 	if resDetails["bodySize"].(float64) < 0 {
 		resDetails["bodySize"] = 0
 	}
@@ -481,6 +517,7 @@ func (d dissecting) Macros() map[string]string {
 		`http`:  fmt.Sprintf(`proto.name == "%s" and proto.version.startsWith("%c")`, http11protocol.Name, http11protocol.Version[0]),
 		`http2`: fmt.Sprintf(`proto.name == "%s" and proto.version == "%s"`, http11protocol.Name, http2Protocol.Version),
 		`grpc`:  fmt.Sprintf(`proto.name == "%s" and proto.version == "%s" and proto.macro == "%s"`, http11protocol.Name, grpcProtocol.Version, grpcProtocol.Macro),
+		`gql`:   fmt.Sprintf(`proto.name == "%s" and proto.macro == "%s"`, graphQL1Protocol.Name, graphQL1Protocol.Macro),
 	}
 }

--- a/tap/extensions/http/main_test.go
+++ b/tap/extensions/http/main_test.go
@@ -47,6 +47,7 @@ func TestMacros(t *testing.T) {
 		"http":  `proto.name == "http" and proto.version.startsWith("1")`,
 		"http2": `proto.name == "http" and proto.version == "2.0"`,
 		"grpc":  `proto.name == "http" and proto.version == "2.0" and proto.macro == "grpc"`,
+		"gql":   `proto.name == "http" and proto.macro == "gql"`,
 	}
 	dissector := NewDissector()
 	macros := dissector.Macros()
--- a/tap/extensions/http/tcp_stream_mock_test.go
+++ b/tap/extensions/http/tcp_stream_mock_test.go
@@ -11,7 +11,7 @@ type tcpStream struct {
 	protoIdentifier *api.ProtoIdentifier
 	isTapTarget     bool
 	origin          api.Capture
-	reqResMatcher   api.RequestResponseMatcher
+	reqResMatchers  []api.RequestResponseMatcher
 	sync.Mutex
 }

@@ -32,8 +32,8 @@ func (t *tcpStream) GetProtoIdentifier() *api.ProtoIdentifier {
 	return t.protoIdentifier
 }

-func (t *tcpStream) GetReqResMatcher() api.RequestResponseMatcher {
-	return t.reqResMatcher
+func (t *tcpStream) GetReqResMatchers() []api.RequestResponseMatcher {
+	return t.reqResMatchers
 }

 func (t *tcpStream) GetIsTapTarget() bool {
--- a/tap/extensions/kafka/go.mod
+++ b/tap/extensions/kafka/go.mod
@@ -22,5 +22,3 @@ require (
 )

 replace github.com/up9inc/mizu/tap/api v0.0.0 => ../../api
-
-replace github.com/up9inc/mizu/logger v0.0.0 => ../../../logger
--- a/tap/extensions/kafka/tcp_stream_mock_test.go
+++ b/tap/extensions/kafka/tcp_stream_mock_test.go
@@ -11,7 +11,7 @@ type tcpStream struct {
 	protoIdentifier *api.ProtoIdentifier
 	isTapTarget     bool
 	origin          api.Capture
-	reqResMatcher   api.RequestResponseMatcher
+	reqResMatchers  []api.RequestResponseMatcher
 	sync.Mutex
 }

@@ -32,8 +32,8 @@ func (t *tcpStream) GetProtoIdentifier() *api.ProtoIdentifier {
 	return t.protoIdentifier
 }

-func (t *tcpStream) GetReqResMatcher() api.RequestResponseMatcher {
-	return t.reqResMatcher
+func (t *tcpStream) GetReqResMatchers() []api.RequestResponseMatcher {
+	return t.reqResMatchers
 }

 func (t *tcpStream) GetIsTapTarget() bool {
--- a/tap/extensions/redis/go.mod
+++ b/tap/extensions/redis/go.mod
@@ -15,5 +15,3 @@ require (
 )

 replace github.com/up9inc/mizu/tap/api v0.0.0 => ../../api
-
-replace github.com/up9inc/mizu/logger v0.0.0 => ../../../logger
--- a/tap/extensions/redis/tcp_stream_mock_test.go
+++ b/tap/extensions/redis/tcp_stream_mock_test.go
@@ -11,7 +11,7 @@ type tcpStream struct {
 	protoIdentifier *api.ProtoIdentifier
 	isTapTarget     bool
 	origin          api.Capture
-	reqResMatcher   api.RequestResponseMatcher
+	reqResMatchers  []api.RequestResponseMatcher
 	sync.Mutex
 }

@@ -32,8 +32,8 @@ func (t *tcpStream) GetProtoIdentifier() *api.ProtoIdentifier {
 	return t.protoIdentifier
 }

-func (t *tcpStream) GetReqResMatcher() api.RequestResponseMatcher {
-	return t.reqResMatcher
+func (t *tcpStream) GetReqResMatchers() []api.RequestResponseMatcher {
+	return t.reqResMatchers
 }

 func (t *tcpStream) GetIsTapTarget() bool {
--- a/tap/go.mod
+++ b/tap/go.mod
@@ -18,6 +18,7 @@ require (
 	github.com/google/go-cmp v0.5.7 // indirect
 	github.com/google/gofuzz v1.2.0 // indirect
 	github.com/google/martian v2.1.0+incompatible // indirect
+	github.com/hashicorp/golang-lru v0.5.4 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
@@ -38,5 +39,3 @@ require (
 replace github.com/up9inc/mizu/logger v0.0.0 => ../logger

 replace github.com/up9inc/mizu/tap/api v0.0.0 => ./api
-
-replace github.com/up9inc/mizu/shared v0.0.0 => ../shared
--- a/tap/go.sum
+++ b/tap/go.sum
@@ -71,6 +71,8 @@ github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2c
 github.com/googleapis/gnostic v0.5.5/go.mod h1:7+EbHbldMins07ALC74bsA81Ovc97DwqyJO1AENw9kA=
 github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
+github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
--- a/tap/passive_tapper.go
+++ b/tap/passive_tapper.go
@@ -69,10 +69,12 @@ func StartPassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem,
 	extensions = extensionsRef
 	filteringOptions = options

+	streamsMap := NewTcpStreamMap()
+
 	if *tls {
 		for _, e := range extensions {
 			if e.Protocol.Name == "http" {
-				tlsTapperInstance = startTlsTapper(e, outputItems, options)
+				tlsTapperInstance = startTlsTapper(e, outputItems, options, streamsMap)
 				break
 			}
 		}
@@ -82,7 +84,7 @@ func StartPassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem,
 		diagnose.StartMemoryProfiler(os.Getenv(MemoryProfilingDumpPath), os.Getenv(MemoryProfilingTimeIntervalSeconds))
 	}

-	streamsMap, assembler := initializePassiveTapper(opts, outputItems)
+	assembler := initializePassiveTapper(opts, outputItems, streamsMap)
 	go startPassiveTapper(streamsMap, assembler)
 }

@@ -181,9 +183,7 @@ func initializePacketSources() error {
 	return err
 }

-func initializePassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem) (api.TcpStreamMap, *tcpAssembler) {
-	streamsMap := NewTcpStreamMap()
-
+func initializePassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelItem, streamsMap api.TcpStreamMap) *tcpAssembler {
 	diagnose.InitializeErrorsMap(*debug, *verbose, *quiet)
 	diagnose.InitializeTapperInternalStats()

@@ -195,7 +195,7 @@ func initializePassiveTapper(opts *TapOpts, outputItems chan *api.OutputChannelI

 	assembler := NewTcpAssembler(outputItems, streamsMap, opts)

-	return streamsMap, assembler
+	return assembler
 }

 func startPassiveTapper(streamsMap api.TcpStreamMap, assembler *tcpAssembler) {
@@ -232,7 +232,8 @@ func startPassiveTapper(streamsMap api.TcpStreamMap, assembler *tcpAssembler) {
 	logger.Log.Infof("AppStats: %v", diagnose.AppStats)
 }

-func startTlsTapper(extension *api.Extension, outputItems chan *api.OutputChannelItem, options *api.TrafficFilteringOptions) *tlstapper.TlsTapper {
+func startTlsTapper(extension *api.Extension, outputItems chan *api.OutputChannelItem,
+	options *api.TrafficFilteringOptions, streamsMap api.TcpStreamMap) *tlstapper.TlsTapper {
 	tls := tlstapper.TlsTapper{}
 	chunksBufferSize := os.Getpagesize() * 100
 	logBufferSize := os.Getpagesize()
@@ -262,7 +263,7 @@ func startTlsTapper(extension *api.Extension, outputItems chan *api.OutputChanne
 	}

 	go tls.PollForLogging()
-	go tls.Poll(emitter, options)
+	go tls.Poll(emitter, options, streamsMap)

 	return &tls
 }
--- a/tap/tcp_reassembly_stream.go
+++ b/tap/tcp_reassembly_stream.go
@@ -148,12 +148,12 @@ func (t *tcpReassemblyStream) ReassembledSG(sg reassembly.ScatterGather, ac reas
 			stream := t.tcpStream.(*tcpStream)
 			if dir == reassembly.TCPDirClientToServer {
 				for i := range stream.getClients() {
-					reader := stream.getClient(i).(*tcpReader)
+					reader := stream.getClient(i)
 					reader.sendMsgIfNotClosed(NewTcpReaderDataMsg(data, timestamp))
 				}
 			} else {
 				for i := range stream.getServers() {
-					reader := stream.getServer(i).(*tcpReader)
+					reader := stream.getServer(i)
 					reader.sendMsgIfNotClosed(NewTcpReaderDataMsg(data, timestamp))
 				}
 			}
--- a/tap/tcp_stream.go
+++ b/tap/tcp_stream.go
@@ -17,10 +17,10 @@ type tcpStream struct {
 	isClosed        bool
 	protoIdentifier *api.ProtoIdentifier
 	isTapTarget     bool
-	clients         []api.TcpReader
-	servers         []api.TcpReader
+	clients         []*tcpReader
+	servers         []*tcpReader
 	origin          api.Capture
-	reqResMatcher   api.RequestResponseMatcher
+	reqResMatchers  []api.RequestResponseMatcher
 	createdAt       time.Time
 	streamsMap      api.TcpStreamMap
 	sync.Mutex
@@ -57,38 +57,42 @@ func (t *tcpStream) close() {

 	for i := range t.clients {
 		reader := t.clients[i]
-		reader.(*tcpReader).close()
+		reader.close()
 	}
 	for i := range t.servers {
 		reader := t.servers[i]
-		reader.(*tcpReader).close()
+		reader.close()
 	}
 }

-func (t *tcpStream) addClient(reader api.TcpReader) {
+func (t *tcpStream) addClient(reader *tcpReader) {
 	t.clients = append(t.clients, reader)
 }

-func (t *tcpStream) addServer(reader api.TcpReader) {
+func (t *tcpStream) addServer(reader *tcpReader) {
 	t.servers = append(t.servers, reader)
 }

-func (t *tcpStream) getClients() []api.TcpReader {
+func (t *tcpStream) getClients() []*tcpReader {
 	return t.clients
 }

-func (t *tcpStream) getServers() []api.TcpReader {
+func (t *tcpStream) getServers() []*tcpReader {
 	return t.servers
 }

-func (t *tcpStream) getClient(index int) api.TcpReader {
+func (t *tcpStream) getClient(index int) *tcpReader {
 	return t.clients[index]
 }

-func (t *tcpStream) getServer(index int) api.TcpReader {
+func (t *tcpStream) getServer(index int) *tcpReader {
 	return t.servers[index]
 }

+func (t *tcpStream) addReqResMatcher(reqResMatcher api.RequestResponseMatcher) {
+	t.reqResMatchers = append(t.reqResMatchers, reqResMatcher)
+}
+
 func (t *tcpStream) SetProtocol(protocol *api.Protocol) {
 	t.Lock()
 	defer t.Unlock()
@@ -102,13 +106,13 @@ func (t *tcpStream) SetProtocol(protocol *api.Protocol) {
 	for i := range t.clients {
 		reader := t.clients[i]
 		if reader.GetExtension().Protocol != t.protoIdentifier.Protocol {
-			reader.(*tcpReader).close()
+			reader.close()
 		}
 	}
 	for i := range t.servers {
 		reader := t.servers[i]
 		if reader.GetExtension().Protocol != t.protoIdentifier.Protocol {
-			reader.(*tcpReader).close()
+			reader.close()
 		}
 	}

@@ -123,8 +127,8 @@ func (t *tcpStream) GetProtoIdentifier() *api.ProtoIdentifier {
 	return t.protoIdentifier
 }

-func (t *tcpStream) GetReqResMatcher() api.RequestResponseMatcher {
-	return t.reqResMatcher
+func (t *tcpStream) GetReqResMatchers() []api.RequestResponseMatcher {
+	return t.reqResMatchers
 }

 func (t *tcpStream) GetIsTapTarget() bool {
--- a/tap/tcp_stream_factory.go
+++ b/tap/tcp_stream_factory.go
@@ -64,6 +64,7 @@ func (factory *tcpStreamFactory) New(net, transport gopacket.Flow, tcpLayer *lay
 		stream.setId(factory.streamsMap.NextId())
 		for i, extension := range extensions {
 			reqResMatcher := extension.Dissector.NewResponseRequestMatcher()
+			stream.addReqResMatcher(reqResMatcher)
 			counterPair := &api.CounterPair{
 				Request:  0,
 				Response: 0,
@@ -114,8 +115,8 @@ func (factory *tcpStreamFactory) New(net, transport gopacket.Flow, tcpLayer *lay
 			factory.streamsMap.Store(stream.getId(), stream)

 			factory.wg.Add(2)
-			go stream.getClient(i).(*tcpReader).run(filteringOptions, &factory.wg)
-			go stream.getServer(i).(*tcpReader).run(filteringOptions, &factory.wg)
+			go stream.getClient(i).run(filteringOptions, &factory.wg)
+			go stream.getServer(i).run(filteringOptions, &factory.wg)
 		}
 	}
 	return reassemblyStream
--- a/tap/tcp_streams_map.go
+++ b/tap/tcp_streams_map.go
@@ -2,6 +2,7 @@ package tap

 import (
 	"runtime"
+	_debug "runtime/debug"
 	"sync"
 	"time"

@@ -47,8 +48,15 @@ func (streamMap *tcpStreamMap) CloseTimedoutTcpStreamChannels() {
 	for {
 		<-ticker.C

+		_debug.FreeOSMemory()
 		streamMap.streams.Range(func(key interface{}, value interface{}) bool {
-			stream := value.(*tcpStream)
+			// `*tlsStream` is not yet applicable to this routine.
+			// So, we cast into `(*tcpStream)` and ignore `*tlsStream`
+			stream, ok := value.(*tcpStream)
+			if !ok {
+				return true
+			}
+
 			if stream.protoIdentifier.Protocol == nil {
 				if !stream.isClosed && time.Now().After(stream.createdAt.Add(tcpStreamChannelTimeoutMs)) {
 					stream.close()
--- a/tap/tlstapper/bpf/openssl_uprobes.c
+++ b/tap/tlstapper/bpf/openssl_uprobes.c
@@ -48,14 +48,14 @@ static __always_inline int get_count_bytes(struct pt_regs *ctx, struct ssl_info*
 	return countBytes;
 }

-static __always_inline void add_address_to_chunk(struct pt_regs *ctx, struct tlsChunk* chunk, __u64 id, __u32 fd) {
+static __always_inline int add_address_to_chunk(struct pt_regs *ctx, struct tlsChunk* chunk, __u64 id, __u32 fd) {
 	__u32 pid = id >> 32;
 	__u64 key = (__u64) pid << 32 | fd;
 	
 	struct fd_info *fdinfo = bpf_map_lookup_elem(&file_descriptor_to_ipv4, &key);
 	
 	if (fdinfo == NULL) {
-		return;
+		return 0;
 	}
 	
 	int err = bpf_probe_read(chunk->address, sizeof(chunk->address), fdinfo->ipv4_addr);
@@ -63,7 +63,10 @@ static __always_inline void add_address_to_chunk(struct pt_regs *ctx, struct tls
 	
 	if (err != 0) {
 		log_error(ctx, LOG_ERROR_READING_FD_ADDRESS, id, err, 0l);
+		return 0;
 	}
+	
+	return 1;
 }

 static __always_inline void send_chunk_part(struct pt_regs *ctx, __u8* buffer, __u64 id, 
@@ -143,7 +146,12 @@ static __always_inline void output_ssl_chunk(struct pt_regs *ctx, struct ssl_inf
 	chunk->len = countBytes;
 	chunk->fd = info->fd;
 	
-	add_address_to_chunk(ctx, chunk, id, chunk->fd);
+	if (!add_address_to_chunk(ctx, chunk, id, chunk->fd)) {
+		// Without an address, we drop the chunk because there is not much to do with it in Go
+		//
+		return;
+	}
+	
 	send_chunk(ctx, info->buffer, id, chunk);
 }

--- a/tap/tlstapper/chunk.go
+++ b/tap/tlstapper/chunk.go
@@ -6,6 +6,7 @@ import (
 	"net"

 	"github.com/go-errors/errors"
+	"github.com/up9inc/mizu/tap/api"
 )

 const FLAGS_IS_CLIENT_BIT uint32 = (1 << 0)
@@ -73,3 +74,27 @@ func (c *tlsChunk) getRecordedData() []byte {
 func (c *tlsChunk) isRequest() bool {
 	return (c.isClient() && c.isWrite()) || (c.isServer() && c.isRead())
 }
+
+func (c *tlsChunk) getAddressPair() (addressPair, error) {
+	ip, port, err := c.getAddress()
+
+	if err != nil {
+		return addressPair{}, err
+	}
+
+	if c.isRequest() {
+		return addressPair{
+			srcIp:   api.UnknownIp,
+			srcPort: api.UnknownPort,
+			dstIp:   ip,
+			dstPort: port,
+		}, nil
+	} else {
+		return addressPair{
+			srcIp:   ip,
+			srcPort: port,
+			dstIp:   api.UnknownIp,
+			dstPort: api.UnknownPort,
+		}, nil
+	}
+}
--- a/tap/tlstapper/sockfd_info.go
+++ b/tap/tlstapper/sockfd_info.go
@@ -20,10 +20,17 @@ const (
 	INODE_FILED_INDEX       = 9
 )

+type addressPair struct {
+	srcIp   net.IP
+	srcPort uint16
+	dstIp   net.IP
+	dstPort uint16
+}
+
 // This file helps to extract Ip and Port out of a Socket file descriptor.
-// 
+//
 // The equivalent bash commands are:
-// 
+//
 //  > ls -l /proc/<pid>/fd/<fd>
 // 	    Output something like "socket:[1234]" for sockets - 1234 is the inode of the socket
 //  > cat /proc/<pid>/net/tcp | grep <inode>
@@ -31,18 +38,18 @@ const (
 //      The 1st and 2nd fields are the source and dest ip and ports in a Hex format
 //      0100007F:50 is 127.0.0.1:80

-func getAddressBySockfd(procfs string, pid uint32, fd uint32, src bool) (net.IP, uint16, error) {
+func getAddressBySockfd(procfs string, pid uint32, fd uint32) (addressPair, error) {
 	inode, err := getSocketInode(procfs, pid, fd)

 	if err != nil {
-		return nil, 0, err
+		return addressPair{}, err
 	}

 	tcppath := fmt.Sprintf("%s/%d/net/tcp", procfs, pid)
 	tcp, err := ioutil.ReadFile(tcppath)

 	if err != nil {
-		return nil, 0, errors.Wrap(err, 0)
+		return addressPair{}, errors.Wrap(err, 0)
 	}

 	for _, line := range strings.Split(string(tcp), "\n") {
@@ -53,15 +60,28 @@ func getAddressBySockfd(procfs string, pid uint32, fd uint32, src bool) (net.IP,
 		}

 		if inode == parts[INODE_FILED_INDEX] {
-			if src {
-				return parseHexAddress(parts[SRC_ADDRESS_FILED_INDEX])
-			} else {
-				return parseHexAddress(parts[DST_ADDRESS_FILED_INDEX])
+			srcIp, srcPort, srcErr := parseHexAddress(parts[SRC_ADDRESS_FILED_INDEX])
+
+			if srcErr != nil {
+				return addressPair{}, srcErr
 			}
+
+			dstIp, dstPort, dstErr := parseHexAddress(parts[DST_ADDRESS_FILED_INDEX])
+
+			if dstErr != nil {
+				return addressPair{}, dstErr
+			}
+
+			return addressPair{
+				srcIp:   srcIp,
+				srcPort: srcPort,
+				dstIp:   dstIp,
+				dstPort: dstPort,
+			}, nil
 		}
 	}

-	return nil, 0, errors.Errorf("address not found [pid: %d] [sockfd: %d] [inode: %s]", pid, fd, inode)
+	return addressPair{}, errors.Errorf("address not found [pid: %d] [sockfd: %d] [inode: %s]", pid, fd, inode)
 }

 func getSocketInode(procfs string, pid uint32, fd uint32) (string, error) {
--- a/tap/tlstapper/tls_poller.go
+++ b/tap/tlstapper/tls_poller.go
@@ -4,7 +4,6 @@ import (
 	"bufio"
 	"bytes"
 	"fmt"
-	"net"
 	"sync"
 	"time"

@@ -16,10 +15,14 @@ import (

 	"github.com/cilium/ebpf/perf"
 	"github.com/go-errors/errors"
+	"github.com/hashicorp/golang-lru/simplelru"
 	"github.com/up9inc/mizu/logger"
 	"github.com/up9inc/mizu/tap/api"
 )

+const fdCachedItemAvgSize = 40
+const fdCacheMaxItems = 500000 / fdCachedItemAvgSize
+
 type tlsPoller struct {
 	tls            *TlsTapper
 	readers        map[string]*tlsReader
@@ -29,10 +32,12 @@ type tlsPoller struct {
 	extension      *api.Extension
 	procfs         string
 	pidToNamespace sync.Map
+	fdCache        *simplelru.LRU // Actual typs is map[string]addressPair
+	evictedCounter int
 }

-func newTlsPoller(tls *TlsTapper, extension *api.Extension, procfs string) *tlsPoller {
-	return &tlsPoller{
+func newTlsPoller(tls *TlsTapper, extension *api.Extension, procfs string) (*tlsPoller, error) {
+	poller := &tlsPoller{
 		tls:           tls,
 		readers:       make(map[string]*tlsReader),
 		closedReaders: make(chan string, 100),
@@ -41,6 +46,15 @@ func newTlsPoller(tls *TlsTapper, extension *api.Extension, procfs string) *tlsP
 		chunksReader:  nil,
 		procfs:        procfs,
 	}
+
+	fdCache, err := simplelru.NewLRU(fdCacheMaxItems, poller.fdCacheEvictCallback)
+
+	if err != nil {
+		return nil, errors.Wrap(err, 0)
+	}
+
+	poller.fdCache = fdCache
+	return poller, nil
 }

 func (p *tlsPoller) init(bpfObjects *tlsTapperObjects, bufferSize int) error {
@@ -59,7 +73,7 @@ func (p *tlsPoller) close() error {
 	return p.chunksReader.Close()
 }

-func (p *tlsPoller) poll(emitter api.Emitter, options *api.TrafficFilteringOptions) {
+func (p *tlsPoller) poll(emitter api.Emitter, options *api.TrafficFilteringOptions, streamsMap api.TcpStreamMap) {
 	chunks := make(chan *tlsChunk)

 	go p.pollChunksPerfBuffer(chunks)
@@ -71,7 +85,7 @@ func (p *tlsPoller) poll(emitter api.Emitter, options *api.TrafficFilteringOptio
 				return
 			}

-			if err := p.handleTlsChunk(chunk, p.extension, emitter, options); err != nil {
+			if err := p.handleTlsChunk(chunk, p.extension, emitter, options, streamsMap); err != nil {
 				LogError(err)
 			}
 		case key := <-p.closedReaders:
@@ -115,36 +129,40 @@ func (p *tlsPoller) pollChunksPerfBuffer(chunks chan<- *tlsChunk) {
 	}
 }

-func (p *tlsPoller) handleTlsChunk(chunk *tlsChunk, extension *api.Extension,
-	emitter api.Emitter, options *api.TrafficFilteringOptions) error {
-	ip, port, err := chunk.getAddress()
+func (p *tlsPoller) handleTlsChunk(chunk *tlsChunk, extension *api.Extension, emitter api.Emitter,
+	options *api.TrafficFilteringOptions, streamsMap api.TcpStreamMap) error {
+	address, err := p.getSockfdAddressPair(chunk)

 	if err != nil {
-		return err
+		address, err = chunk.getAddressPair()
+
+		if err != nil {
+			return err
+		}
 	}

-	key := buildTlsKey(chunk, ip, port)
+	key := buildTlsKey(address)
 	reader, exists := p.readers[key]

 	if !exists {
-		reader = p.startNewTlsReader(chunk, ip, port, key, emitter, extension, options)
+		reader = p.startNewTlsReader(chunk, &address, key, emitter, extension, options, streamsMap)
 		p.readers[key] = reader
 	}

-	reader.captureTime = time.Now()
-	reader.chunks <- chunk
+	reader.newChunk(chunk)

 	if os.Getenv("MIZU_VERBOSE_TLS_TAPPER") == "true" {
-		p.logTls(chunk, ip, port)
+		p.logTls(chunk, key, reader)
 	}

 	return nil
 }

-func (p *tlsPoller) startNewTlsReader(chunk *tlsChunk, ip net.IP, port uint16, key string,
-	emitter api.Emitter, extension *api.Extension, options *api.TrafficFilteringOptions) *tlsReader {
+func (p *tlsPoller) startNewTlsReader(chunk *tlsChunk, address *addressPair, key string,
+	emitter api.Emitter, extension *api.Extension, options *api.TrafficFilteringOptions,
+	streamsMap api.TcpStreamMap) *tlsReader {

-	tcpid := p.buildTcpId(chunk, ip, port)
+	tcpid := p.buildTcpId(chunk, address)

 	doneHandler := func(r *tlsReader) {
 		p.closeReader(key, r)
@@ -173,6 +191,7 @@ func (p *tlsPoller) startNewTlsReader(chunk *tlsChunk, ip net.IP, port uint16, k
 		reader:          reader,
 		protoIdentifier: &api.ProtoIdentifier{},
 	}
+	streamsMap.Store(streamsMap.NextId(), stream)

 	reader.parent = stream

@@ -195,36 +214,52 @@ func (p *tlsPoller) closeReader(key string, r *tlsReader) {
 	p.closedReaders <- key
 }

-func buildTlsKey(chunk *tlsChunk, ip net.IP, port uint16) string {
-	return fmt.Sprintf("%v:%v-%v:%v", chunk.isClient(), chunk.isRead(), ip, port)
-}
+func (p *tlsPoller) getSockfdAddressPair(chunk *tlsChunk) (addressPair, error) {
+	address, err := getAddressBySockfd(p.procfs, chunk.Pid, chunk.Fd)
+	fdCacheKey := fmt.Sprintf("%d:%d", chunk.Pid, chunk.Fd)

-func (p *tlsPoller) buildTcpId(chunk *tlsChunk, ip net.IP, port uint16) api.TcpID {
-	myIp, myPort, err := getAddressBySockfd(p.procfs, chunk.Pid, chunk.Fd, chunk.isClient())
-
-	if err != nil {
-		// May happen if the socket already closed, very likely to happen for localhost
-		//
-		myIp = api.UnknownIp
-		myPort = api.UnknownPort
+	if err == nil {
+		if !chunk.isRequest() {
+			switchedAddress := addressPair{
+				srcIp:   address.dstIp,
+				srcPort: address.dstPort,
+				dstIp:   address.srcIp,
+				dstPort: address.srcPort,
+			}
+			p.fdCache.Add(fdCacheKey, switchedAddress)
+			return switchedAddress, nil
+		} else {
+			p.fdCache.Add(fdCacheKey, address)
+			return address, nil
+		}
 	}

-	if chunk.isRequest() {
-		return api.TcpID{
-			SrcIP:   myIp.String(),
-			DstIP:   ip.String(),
-			SrcPort: strconv.FormatUint(uint64(myPort), 10),
-			DstPort: strconv.FormatUint(uint64(port), 10),
-			Ident:   "",
-		}
-	} else {
-		return api.TcpID{
-			SrcIP:   ip.String(),
-			DstIP:   myIp.String(),
-			SrcPort: strconv.FormatUint(uint64(port), 10),
-			DstPort: strconv.FormatUint(uint64(myPort), 10),
-			Ident:   "",
-		}
+	fromCacheIfc, ok := p.fdCache.Get(fdCacheKey)
+
+	if !ok {
+		return addressPair{}, err
+	}
+
+	fromCache, ok := fromCacheIfc.(addressPair)
+
+	if !ok {
+		return address, errors.Errorf("Unable to cast %T to addressPair", fromCacheIfc)
+	}
+
+	return fromCache, nil
+}
+
+func buildTlsKey(address addressPair) string {
+	return fmt.Sprintf("%s:%d>%s:%d", address.srcIp, address.srcPort, address.dstIp, address.dstPort)
+}
+
+func (p *tlsPoller) buildTcpId(chunk *tlsChunk, address *addressPair) api.TcpID {
+	return api.TcpID{
+		SrcIP:   address.srcIp.String(),
+		DstIP:   address.dstIp.String(),
+		SrcPort: strconv.FormatUint(uint64(address.srcPort), 10),
+		DstPort: strconv.FormatUint(uint64(address.dstPort), 10),
+		Ident:   "",
 	}
 }

@@ -255,7 +290,7 @@ func (p *tlsPoller) clearPids() {
 	})
 }

-func (p *tlsPoller) logTls(chunk *tlsChunk, ip net.IP, port uint16) {
+func (p *tlsPoller) logTls(chunk *tlsChunk, key string, reader *tlsReader) {
 	var flagsStr string

 	if chunk.isClient() {
@@ -270,13 +305,18 @@ func (p *tlsPoller) logTls(chunk *tlsChunk, ip net.IP, port uint16) {
 		flagsStr += "W"
 	}

-	srcIp, srcPort, _ := getAddressBySockfd(p.procfs, chunk.Pid, chunk.Fd, true)
-	dstIp, dstPort, _ := getAddressBySockfd(p.procfs, chunk.Pid, chunk.Fd, false)
-
 	str := strings.ReplaceAll(strings.ReplaceAll(string(chunk.Data[0:chunk.Recorded]), "\n", " "), "\r", "")

-	logger.Log.Infof("PID: %v (tid: %v) (fd: %v) (client: %v) (addr: %v:%v) (fdaddr %v:%v>%v:%v) (recorded %v out of %v starting at %v) - %v - %v",
-		chunk.Pid, chunk.Tgid, chunk.Fd, flagsStr, ip, port,
-		srcIp, srcPort, dstIp, dstPort,
-		chunk.Recorded, chunk.Len, chunk.Start, str, hex.EncodeToString(chunk.Data[0:chunk.Recorded]))
+	logger.Log.Infof("[%-44s] %s #%-4d (fd: %d) (recorded %d/%d:%d) - %s - %s",
+		key, flagsStr, reader.seenChunks, chunk.Fd,
+		chunk.Recorded, chunk.Len, chunk.Start,
+		str, hex.EncodeToString(chunk.Data[0:chunk.Recorded]))
+}
+
+func (p *tlsPoller) fdCacheEvictCallback(key interface{}, value interface{}) {
+	p.evictedCounter = p.evictedCounter + 1
+
+	if p.evictedCounter%1000000 == 0 {
+		logger.Log.Infof("Tls fdCache evicted %d items", p.evictedCounter)
+	}
 }
--- a/tap/tlstapper/tls_reader.go
+++ b/tap/tlstapper/tls_reader.go
@@ -10,6 +10,7 @@ import (
 type tlsReader struct {
 	key           string
 	chunks        chan *tlsChunk
+	seenChunks    int
 	data          []byte
 	doneHandler   func(r *tlsReader)
 	progress      *api.ReadProgress
@@ -19,10 +20,16 @@ type tlsReader struct {
 	extension     *api.Extension
 	emitter       api.Emitter
 	counterPair   *api.CounterPair
-	parent        api.TcpStream
+	parent        *tlsStream
 	reqResMatcher api.RequestResponseMatcher
 }

+func (r *tlsReader) newChunk(chunk *tlsChunk) {
+	r.captureTime = time.Now()
+	r.seenChunks = r.seenChunks + 1
+	r.chunks <- chunk
+}
+
 func (r *tlsReader) Read(p []byte) (int, error) {
 	var chunk *tlsChunk

--- a/tap/tlstapper/tls_stream.go
+++ b/tap/tlstapper/tls_stream.go
@@ -19,8 +19,8 @@ func (t *tlsStream) SetProtocol(protocol *api.Protocol) {
 	t.protoIdentifier.Protocol = protocol
 }

-func (t *tlsStream) GetReqResMatcher() api.RequestResponseMatcher {
-	return t.reader.reqResMatcher
+func (t *tlsStream) GetReqResMatchers() []api.RequestResponseMatcher {
+	return []api.RequestResponseMatcher{t.reader.reqResMatcher}
 }

 func (t *tlsStream) GetIsTapTarget() bool {
--- a/tap/tlstapper/tls_tapper.go
+++ b/tap/tlstapper/tls_tapper.go
@@ -46,12 +46,18 @@ func (t *TlsTapper) Init(chunksBufferSize int, logBufferSize int, procfs string,
 		return err
 	}

-	t.poller = newTlsPoller(t, extension, procfs)
+	var err error
+	t.poller, err = newTlsPoller(t, extension, procfs)
+
+	if err != nil {
+		return err
+	}
+
 	return t.poller.init(&t.bpfObjects, chunksBufferSize)
 }

-func (t *TlsTapper) Poll(emitter api.Emitter, options *api.TrafficFilteringOptions) {
-	t.poller.poll(emitter, options)
+func (t *TlsTapper) Poll(emitter api.Emitter, options *api.TrafficFilteringOptions, streamsMap api.TcpStreamMap) {
+	t.poller.poll(emitter, options, streamsMap)
 }

 func (t *TlsTapper) PollForLogging() {
--- a/tap/tlstapper/tlstapper_bpfeb.o
+++ b/tap/tlstapper/tlstapper_bpfeb.o
--- a/tap/tlstapper/tlstapper_bpfel.o
+++ b/tap/tlstapper/tlstapper_bpfel.o
--- a/ui-common/src/components/ServiceMapModal/ServiceMapModal.tsx
+++ b/ui-common/src/components/ServiceMapModal/ServiceMapModal.tsx
@@ -49,6 +49,7 @@ const protocols = [
    { key: "HTTP", value: "HTTP", component: <LegentLabel color="#205cf5" name="HTTP" /> },
    { key: "HTTP/2", value: "HTTP/2", component: <LegentLabel color='#244c5a' name="HTTP/2" /> },
    { key: "gRPC", value: "gRPC", component: <LegentLabel color='#244c5a' name="gRPC" /> },
+    { key: "GQL", value: "GQL", component: <LegentLabel color='#e10098' name="GQL" /> },
    { key: "AMQP", value: "AMQP", component: <LegentLabel color='#ff6600' name="AMQP" /> },
    { key: "KAFKA", value: "KAFKA", component: <LegentLabel color='#000000' name="KAFKA" /> },
    { key: "REDIS", value: "REDIS", component: <LegentLabel color='#a41e11' name="REDIS" /> },]
--- a/ui/craco.config.js
+++ b/ui/craco.config.js
@@ -13,7 +13,7 @@ module.exports = {
                instanceOfMiniCssExtractPlugin.options.ignoreOrder = true;

            webpackConfig.resolve.alias['react']= path.resolve(__dirname, 'node_modules/react'); // solve 2  react instances
-
+            webpackConfig.resolve.alias['@material-ui/styles']= path.resolve("node_modules", "@material-ui/styles");

            return webpackConfig;
        }
Author	SHA1	Message	Date
David Levanon	57f8a8dca9	Feature/fix tls not listening (#1046 ) * avoid chunks with invalid address * tls tapper should distict between pids * prettfy tls verbose log and tls key * support tls from multi threads + duplicate calls to the same target * introduce fdCache and user address pair as tls key * remove unused comment * fix merge conflicts * use lru for fdcache * pr fixes - renaming * fix conflict issue	2022-05-02 21:33:26 +03:00
M. Mert Yıldıran	aead6cbc19	Bring back `debug.FreeOSMemory()` call to prevent `OOMKilled`(s) (#1063 ) * Revert "Revert "Bring back `debug.FreeOSMemory()` call"" This reverts commit `69538cb7e3`. * Revert "Revert "Revert "Remove `GOGC` environment variable from tapper""" This reverts commit `3d9f37ab4b`. * Revert "Revert "Revert "Revert "Remove `GOGC` environment variable from tapper"""" This reverts commit `58ff0607ff`. Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>	2022-05-02 14:13:04 +03:00
gadotroee	2f89690da6	Remove go-cache unused dependency (#1065 )	2022-05-02 13:24:28 +03:00
M. Mert Yıldıran	3e47abf208	Add from ARM64 to AMD64 cross-compilation support to the agent image (#910 ) * Add from ARM64 to AMD64 cross-compilation support to the agent image * Replace `RUN`s with `&&` Co-authored-by: gadotroee <55343099+gadotroee@users.noreply.github.com>	2022-05-02 11:16:05 +03:00
M. Mert Yıldıran	3cbccccb8b	Add GraphQL detection (#980 ) * Add GraphQL detection * Change the abbreviation to `GQL` * Fix the `TestMacros` test case * Try to fix the security issues reported by Snyk * Run `go mod tidy` in `agent` directory * Upgrade `golang.org/x/crypto` * Downgrade `golang.org/x/crypto` * Downgrade `golang.org/x/crypto` * Downgrade `golang.org/x/crypto` * Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2` * Run `go mod tidy` * Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns * Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6` * Don't upgrade `golang.org/x/net`	2022-05-02 11:06:01 +03:00
gadotroee	08ae2bf6d7	Move "es Lint" to different step and add go lint on logger module (#1064 )	2022-05-02 10:01:41 +03:00
M. Mert Yıldıran	684c51686f	Remove the obsolete `replace` statements that are artifacts of a merge conflict (#1061 )	2022-05-01 16:51:43 +03:00
M. Mert Yıldıran	1de50b0572	Fix the request-response matcher maps iteration in `clean()` method and share the streams map with the TLS tapper (#1059 ) * Fix `panic: interface conversion: api.RequestResponseMatcher is nil, not http.requestResponseMatcher` error Also fix the request-response matcher maps iteration in `clean()` method. Fix the mocks in the unit tests * Remove unnecessary fields from `tlsPoller` and implement `SetProtocol` method * Use concrete types in `tap` package * Share the streams map with the TLS tapper * Check interface conversion error	2022-05-01 16:16:22 +03:00
AmitUp9	0881dad17f	update craco config to resolve material ui instances (#1060 )	2022-05-01 16:05:53 +03:00