Implement Phases 1-3 of the performance optimization plan to address
issue #1793 - reduce CPU and memory consumption for system-constrained
environments.
Phase 1 - OPA Module Caching:
- Add compiledModules cache to OPAProcessor with thread-safe access
- Cache compiled OPA rules to eliminate redundant compilation
- Reuse compiled modules with double-checked locking pattern
- Expected CPU savings: 30-40%
Phase 2 - Map Pre-sizing:
- Add estimateClusterSize() to calculate resource count
- Pre-size AllResources, ResourcesResult, and related maps
- Reduce memory reallocations and GC pressure
- Expected memory savings: 10-20%
Phase 3 - Set-based Deduplication:
- Add thread-safe StringSet utility in core/pkg/utils
- Replace O(n) slices.Contains() with O(1) map operations
- Use StringSet for image scanning and related resources deduplication
- 100% test coverage for new utility
- Expected CPU savings: 5-10% for large clusters
Full optimization plan documented in optimization-plan.md
Related: #1793
Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>
* set cloud provider using nodes
* use scan metadata for scanning scope
* code cleanup
Signed-off-by: David Wertenteil <dwertent@armosec.io>
* handle error
Signed-off-by: David Wertenteil <dwertent@armosec.io>
---------
Signed-off-by: David Wertenteil <dwertent@armosec.io>
* Implementing container image name normalization built-in function for Rego
Signed-off-by: Ben <ben@armosec.io>
* updating go.mod t include docker/distribution
Signed-off-by: Ben <ben@armosec.io>
* fix test
Signed-off-by: Ben <ben@armosec.io>
---------
Signed-off-by: Ben <ben@armosec.io>
* support scanning scope
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update go mod
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update white list
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update go mod
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* scope empty return control should tested
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update rego scope for system test
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update test + mock
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* add comment
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update rego library
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update k8s-interface
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update opa utils - lots of file changes in this commit since armoapi-go bump up in opa-utils
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* move to temp k8s-interface - till PR in k8s-interface repo will approved
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update k8s-interface with released tag
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update go mod in httphandler
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* PR review corrections
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* change test name
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* scanning scope support for framework
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* test/mock adjustments after merge
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* add more informative log to the user
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* update go.mod and go.sum of the http handler
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* remove framework just scanning scope not matched to framework config scope
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* add system tests to workflow
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
* add system test to github workflow
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
---------
Signed-off-by: rcohencyberarmor <rcohen@armosec.io>
Signed-off-by: David Wertenteil <dwertent@armosec.io>
Co-authored-by: rcohencyberarmor <rcohen@armosec.io>
Co-authored-by: David Wertenteil <dwertent@armosec.io>
* code refactor
Signed-off-by: Amir Malka <amirm@armosec.io>
* use scaninfo object in resource handler
Signed-off-by: Amir Malka <amirm@armosec.io>
---------
Signed-off-by: Amir Malka <amirm@armosec.io>
* add cmd
Signed-off-by: Daniel Grunberger <danielgrunberger@armosec.io>
* support single workload scan
Signed-off-by: Amir Malka <amirm@armosec.io>
* fix conflict
Signed-off-by: Amir Malka <amirm@armosec.io>
* added unit tests
Signed-off-by: Amir Malka <amirm@armosec.io>
* added unit tests
Signed-off-by: Amir Malka <amirm@armosec.io>
* more refactoring
Signed-off-by: Amir Malka <amirm@armosec.io>
* add scanned workload reference to opasessionobj
Signed-off-by: Amir Malka <amirm@armosec.io>
* fix GetWorkloadParentKind
Signed-off-by: Amir Malka <amirm@armosec.io>
* remove namespace argument from pullSingleResource, using field selector instead
Signed-off-by: Amir Malka <amirm@armosec.io>
* removed designators (unused) field from PolicyIdentifier, and designators argument from GetResources function
Signed-off-by: Amir Malka <amirm@armosec.io>
* fix tests
Signed-off-by: Amir Malka <amirm@armosec.io>
* use ScanObject instead of workload identifier
Signed-off-by: Amir Malka <amirm@armosec.io>
* refactor logic after CR
Signed-off-by: Amir Malka <amirm@armosec.io>
---------
Signed-off-by: Daniel Grunberger <danielgrunberger@armosec.io>
Signed-off-by: Amir Malka <amirm@armosec.io>
Co-authored-by: Daniel Grunberger <danielgrunberger@armosec.io>
* control parallelism of opa rule processing by env var
Signed-off-by: Amir Malka <amirm@armosec.io>
* go 1.20
Signed-off-by: Amir Malka <amirm@armosec.io>
* update go.mod go.sum
Signed-off-by: Amir Malka <amirm@armosec.io>
---------
Signed-off-by: Amir Malka <amirm@armosec.io>
This parallelize the Process() portion of the OPA processor.
The main change is that called methods to evaluate a rule no longer
mutate the internal state of the opaprocessor and allocate maps (less
often, in larger chunks) that are merged at the end of the processing.
Signed-off-by: Frédéric BIDON <fredbi@yahoo.com>
Matthias Bertschy
Eng Zer JunandGitHub
Ben
David Wertenteil
YiscahLevySilas1
Amir MalkaandGitHub
Yuval LeibovichandGitHub
YiscahLevySilas1andGitHub
Ben HirschbergandGitHub
guoguangwu
Alessio Greggi
Frédéric BIDON