Compare commits

...

30 Commits

Author SHA1 Message Date
github-actions[bot]
fbef61d076 [Backport release-1.2] Fix: disable cochange for apprev when def changes (#3221)
* Fix: disable cochange for apprev when def changes

Signed-off-by: Somefive <yd219913@alibaba-inc.com>
(cherry picked from commit f67d1c7e08)

* Fix: add test

Signed-off-by: Somefive <yd219913@alibaba-inc.com>
(cherry picked from commit e3f95763ae)

Co-authored-by: Somefive <yd219913@alibaba-inc.com>
2022-02-10 11:42:04 +08:00
github-actions[bot]
52f9b7e691 [Backport release-1.2] Fix: upgrade package github.com/docker/cli for CVE-2021-41092 (#3218)
* Fix: upgrade package github.com/docker/cli for CVE-2021-41092

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit c0ce5b9931)

* Chore: change go version to 1.17

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 976d781dc2)

* Chore: change go mod

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 01724c9bf5)

* Fix: change install cue shell

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 3bab8d57fa)

Co-authored-by: barnettZQG <barnett.zqg@gmail.com>
2022-02-09 19:21:03 +08:00
github-actions[bot]
e721449c46 [Backport release-1.2] Feat: the golang version upgrade to 1.7 (#3213)
* Feat: the golang version upgrade to 1.7

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit f7e75a18a0)

* Feat: install expat lib

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit e641c39509)

Co-authored-by: barnettZQG <barnett.zqg@gmail.com>
2022-02-09 15:25:59 +08:00
github-actions[bot]
ab998ce3f4 Fix: fix flag conflict (#3212)
Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 440c9947c7)

Co-authored-by: barnettZQG <barnett.zqg@gmail.com>
2022-02-09 15:22:33 +08:00
github-actions[bot]
bcc978380f [Backport release-1.2] Feat: support install and uninstall vela core (#3209)
* Feat: support install and uninstall vela core

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 330b20cabd)

* Feat: support upgrade crd

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 61505a92d3)

* Feat: support set reuse args

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 9ae203f961)

* Feat: apply CRD before install or upgrade

Signed-off-by: barnettZQG <barnett.zqg@gmail.com>
(cherry picked from commit 0815ba6580)

Co-authored-by: barnettZQG <barnett.zqg@gmail.com>
2022-02-09 11:58:11 +08:00
github-actions[bot]
b9f9f7f3f9 Fix: support more Terraform variable types (#3207)
Support Any, set and some complicated variable types

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit c37ba34fc7)

Co-authored-by: Zheng Xi Zhou <zzxwill@gmail.com>
2022-02-08 18:50:58 +08:00
github-actions[bot]
18ceb467ed Feat: add componentName in context for traitDefinition (#3202)
Signed-off-by: Jianbo Sun <jianbo.sjb@alibaba-inc.com>
(cherry picked from commit 2e07453888)

Co-authored-by: Jianbo Sun <jianbo.sjb@alibaba-inc.com>
2022-02-08 16:27:19 +08:00
github-actions[bot]
06eb8f055d [Backport release-1.2] Feat: dispatch manifests in concurrent (#3201)
* Feat: dispatch manifests in concurrent

Signed-off-by: yangsoon <songyang.song@alibaba-inc.com>
(cherry picked from commit 774f108d19)

* Fix: merge workflow pkg convert to pkg util

Signed-off-by: yangsoon <songyang.song@alibaba-inc.com>
(cherry picked from commit 0a09a2fa8d)

Co-authored-by: yangsoon <songyang.song@alibaba-inc.com>
2022-02-08 16:26:54 +08:00
github-actions[bot]
9dec98fbba Fix: add unit test for getting Terraform Configuration from remote git (#3194)
Add another unit test when the configuration of Terraform locates
in a subpath of a git remote repository

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit bef07e9a08)

Co-authored-by: Zheng Xi Zhou <zzxwill@gmail.com>
2022-02-08 10:21:41 +08:00
github-actions[bot]
42e7f04267 Fix: error msg in webhook too complicated to read (#3170)
Signed-off-by: Somefive <yd219913@alibaba-inc.com>
(cherry picked from commit ddc272efa1)

Co-authored-by: Somefive <yd219913@alibaba-inc.com>
2022-01-28 16:19:59 +08:00
github-actions[bot]
188e453f8a Fix: Use systemDefinitionNamespace core Helm Chart (#3165)
Use value in Addon Registry CM and the Test Application

Signed-off-by: Oliver Otte <otte@gonicus.de>
(cherry picked from commit 8498c52aee)

Co-authored-by: Oliver Otte <otte@gonicus.de>
2022-01-25 19:17:56 +08:00
github-actions[bot]
c34cd657e8 Fix: krew install template of kubectl-vela.exe (#3164)
Signed-off-by: Jianbo Sun <jianbo.sjb@alibaba-inc.com>
(cherry picked from commit 533f7820a6)

Co-authored-by: Jianbo Sun <jianbo.sjb@alibaba-inc.com>
2022-01-25 18:51:59 +08:00
github-actions[bot]
29ecc5c0df Feat: support vela show for workflow step definition (#3161)
Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 23852b3d10)

Co-authored-by: FogDong <dongtianxin.tx@alibaba-inc.com>
2022-01-25 15:05:46 +08:00
github-actions[bot]
20c11f2b84 fix: vela addnon enable cannot support = (#3158)
Signed-off-by: 楚岳 <wangyike.wyk@alibaba-inc.com>
(cherry picked from commit ddefb8cb4e)

Co-authored-by: 楚岳 <wangyike.wyk@alibaba-inc.com>
2022-01-25 13:58:24 +08:00
github-actions[bot]
e9f7cf7c23 Fix: add context parameters into the error message (#3159)
Signed-off-by: zeed-w-beez <zeed.w.zhao@gmail.com>
(cherry picked from commit 170888063d)

Co-authored-by: zeed-w-beez <zeed.w.zhao@gmail.com>
2022-01-25 13:28:01 +08:00
github-actions[bot]
cebeff867a [Backport release-1.2] Feat: addon parameter support ui-shcema (#3155)
* Feat: addon parameter support ui-shcema

Signed-off-by: 楚岳 <wangyike.wyk@alibaba-inc.com>

fix ci

(cherry picked from commit 8e2bf9c68d)

* add more tests

Signed-off-by: 楚岳 <wangyike.wyk@alibaba-inc.com>
(cherry picked from commit 661d9f7a47)

Co-authored-by: 楚岳 <wangyike.wyk@alibaba-inc.com>
2022-01-25 10:47:07 +08:00
github-actions[bot]
7902a19aae [Backport release-1.2] Fix: retrieve Terraform variables from variables.tf (#3153)
* Fix: retrieve Terraform variables from variables.tf

If Terraform modules/resources are stored in remote git repos, get
variables from file `varialbes.tf`, or from `main.tf`.

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit b257825af6)

* set the required field per the variables' property

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit 4b151582c5)

* fix ut issue

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit b52145c8a4)

* fix ut issue

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit 410acc8ae2)

Co-authored-by: Zheng Xi Zhou <zzxwill@gmail.com>
2022-01-24 13:56:59 +08:00
github-actions[bot]
f98b8c7d8a Fix: add providerRef in generated ComponentDefinition (#3151)
If the provider is not provider, append the providerRef section in
the generated ComponentDefinition

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit 34ecc40aad)

Co-authored-by: Zheng Xi Zhou <zzxwill@gmail.com>
2022-01-24 11:00:41 +08:00
github-actions[bot]
ee8773e1cf [Backport release-1.2] Fix: handle workflow cache reconcile (#3148)
* Fix: handle workflow cache reconcile

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 12df87ac11)

* fix return and move backoff to memory

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit ee876f53c3)

* handle failed to patch case

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit eac4a1b370)

* add store in err case

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 32825c5c41)

* make reviewable

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 02b9c60922)

* fix ut

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit bff156cbe6)

* do cleanup in ut

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 463bd96e78)

Co-authored-by: FogDong <dongtianxin.tx@alibaba-inc.com>
2022-01-24 10:37:48 +08:00
github-actions[bot]
8cf2f20846 [Backport release-1.2] Fix: workflow skip executing all steps occasionally (#3147)
* fix asi

Signed-off-by: Jian.Li <lj176172@alibaba-inc.com>
(cherry picked from commit 2c26caedd3)

* fix lint

Signed-off-by: Jian.Li <lj176172@alibaba-inc.com>
(cherry picked from commit b63e892eb3)

* add trace tag

(cherry picked from commit 1a6d79642e)

* add args for this feature

Signed-off-by: Jian.Li <lj176172@alibaba-inc.com>
(cherry picked from commit b9181b9fc0)

* enable-asi-compatibility

Signed-off-by: Jian.Li <lj176172@alibaba-inc.com>
(cherry picked from commit 5d013db0bd)

Co-authored-by: Jian.Li <lj176172@alibaba-inc.com>
2022-01-23 10:13:59 +08:00
github-actions[bot]
2c6e8e7de7 [Backport release-1.2] Feat: extend gateway trait to set class in spec (#3146)
* Feat: extend gateway trait to set class in spec

`kubernetes.io/ingress.class` annotation is deprecated in favor of
`.spec.ingressClassName`. However, there is no way to set it through
the gateway trait for now.

This commit extends the gateway trait by adding `classInSpec` to
parameter. Forcing the use of `.spec.ingressClassName` makes sense, but
some ingress controllers (including old versions) may not recognize
this field. Therefore, set default value of `classInSpec` to `false`
for backward compatibility.

Signed-off-by: Sunghoon Kang <hoon@linecorp.com>
(cherry picked from commit ad8bd1c1c4)

* Chore: update classInSpec usage

Signed-off-by: Sunghoon Kang <hoon@linecorp.com>
(cherry picked from commit 721f879e3d)

Co-authored-by: Sunghoon Kang <hoon@linecorp.com>
2022-01-23 10:13:18 +08:00
github-actions[bot]
dbfd6a1d10 [Backport release-1.2] Fix: prioritize namespace flag for vela up (#3137)
* Fix: prioritize namespace flag for `vela up`

Currently, CLI applies application to `default` namespace when there is
no explicit namespace in application spec, even if the namespace flag
is set.

This commit fixes issue by overriding the namespace if the namespace
flag is set.

Signed-off-by: Sunghoon Kang <hoon@linecorp.com>
(cherry picked from commit 4cf07dcd97)

* Test: add test cases for overriding namespaces

Signed-off-by: Sunghoon Kang <hoon@linecorp.com>
(cherry picked from commit 5ba93541e4)

Co-authored-by: Sunghoon Kang <hoon@linecorp.com>
2022-01-21 11:27:11 +08:00
github-actions[bot]
ce8e652802 Fix: support generate Terraform ComponentDefinition from local HCL file (#3134)
Besides generating a Terraform ComponentDefinition from a remote git repo,
support generate one from local HCL file

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit bc06a5343e)

Co-authored-by: Zheng Xi Zhou <zzxwill@gmail.com>
2022-01-20 14:59:24 +08:00
github-actions[bot]
9968211163 add imagePullSecrets for helm templates to support private docker registry (#3129)
Signed-off-by: StevenLeiZhang <zhangleiic@163.com>
(cherry picked from commit cf9eb0ded9)

Co-authored-by: StevenLeiZhang <zhangleiic@163.com>
2022-01-19 13:00:45 +08:00
github-actions[bot]
c86776cca1 [Backport release-1.2] Feat: add port name in webservice (#3120)
* Feat: add port name in webservice

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 76789b0222)

* fix port name in container

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit b486a7c74d)

* generate port name if not specified

Signed-off-by: FogDong <dongtianxin.tx@alibaba-inc.com>
(cherry picked from commit 4eec002916)

Co-authored-by: FogDong <dongtianxin.tx@alibaba-inc.com>
2022-01-18 19:43:18 +08:00
github-actions[bot]
11a1b2fa72 Fix: add app samples for Terraform definition (#3119)
Added application samples for all Terraform typed ComponentDefinition,
and also localize the title for AWS and AZure

Signed-off-by: Zheng Xi Zhou <zzxwill@gmail.com>
(cherry picked from commit 670a327161)

Co-authored-by: Zheng Xi Zhou <zzxwill@gmail.com>
2022-01-18 18:10:16 +08:00
github-actions[bot]
cd036b87ae Feat: support wild match for env patch (#3116)
Signed-off-by: Somefive <yd219913@alibaba-inc.com>
(cherry picked from commit fe0d33c6d9)

Co-authored-by: Somefive <yd219913@alibaba-inc.com>
2022-01-18 18:07:05 +08:00
github-actions[bot]
7e447c6532 fix revision will change when add new trait with skiprevisionaffect to application (#3117)
Signed-off-by: chwetion <chwetion@foxmail.com>
(cherry picked from commit 86f6fef216)

Co-authored-by: chwetion <chwetion@foxmail.com>
2022-01-18 13:41:48 +08:00
github-actions[bot]
b836f86484 Fix: trait/comp command output without a new line (#3115)
Signed-off-by: qiaozp <chivalry.pp@gmail.com>
(cherry picked from commit 1ba3e1dd62)

Co-authored-by: qiaozp <chivalry.pp@gmail.com>
2022-01-18 13:05:23 +08:00
github-actions[bot]
d34372bf47 [Backport release-1.2] Feat: add JFrog webhook trigger (#3114)
* add jfrog webhook to update application image

Signed-off-by: chwetion <chwetion@foxmail.com>
(cherry picked from commit 962fce6870)

* edit jfrog default request header

Signed-off-by: chwetion <chwetion@foxmail.com>
(cherry picked from commit a977bfc9af)

Co-authored-by: chwetion <chwetion@foxmail.com>
2022-01-18 11:51:15 +08:00
133 changed files with 4017 additions and 565 deletions

View File

@@ -15,7 +15,7 @@ on:
env:
# Common versions
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
KIND_VERSION: 'v0.7.0'

View File

@@ -13,7 +13,7 @@ on:
env:
# Common versions
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
KIND_VERSION: 'v0.7.0'

View File

@@ -13,7 +13,7 @@ on:
env:
# Common versions
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
KIND_VERSION: 'v0.7.0'

View File

@@ -13,7 +13,7 @@ on:
env:
# Common versions
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
KIND_VERSION: 'v0.7.0'

View File

@@ -13,7 +13,7 @@ on:
env:
# Common versions
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
KIND_VERSION: 'v0.7.0'

View File

@@ -27,7 +27,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.16
go-version: 1.17
- name: Get release
id: get_release
uses: bruceadams/get-release@v1.2.2

View File

@@ -11,10 +11,10 @@ jobs:
sync-core-api:
runs-on: ubuntu-20.04
steps:
- name: Set up Go 1.16
- name: Set up Go 1.17
uses: actions/setup-go@v1
env:
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
with:
go-version: ${{ env.GO_VERSION }}

View File

@@ -13,7 +13,7 @@ on:
env:
# Common versions
GO_VERSION: '1.16'
GO_VERSION: '1.17'
GOLANGCI_VERSION: 'v1.38'
KIND_VERSION: 'v0.7.0'

View File

@@ -33,8 +33,8 @@ spec:
arch: amd64
{{addURIAndSha "https://github.com/oam-dev/kubevela/releases/download/{{ .TagName }}/kubectl-vela-{{ .TagName }}-windows-amd64.zip" .TagName }}
files:
- from: "*/kubectl-vela.exe"
to: "."
- from: "*/kubectl-vela"
to: "kubectl-vela.exe"
- from: "*/LICENSE"
to: "."
bin: "kubectl-vela.exe"

View File

@@ -1,6 +1,6 @@
ARG BASE_IMAGE="alpine:latest"
ARG BASE_IMAGE
# Build the manager binary
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.16-alpine as builder
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.17-alpine as builder
WORKDIR /workspace
# Copy the Go Modules manifests
@@ -34,9 +34,9 @@ RUN GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} \
# You can replace distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
# Overwrite `BASE_IMAGE` by passing `--build-arg=BASE_IMAGE=gcr.io/distroless/static:nonroot`
FROM ${BASE_IMAGE:-alpine:latest}
FROM ${BASE_IMAGE:-alpine:3.15}
# This is required by daemon connnecting with cri
RUN apk add --no-cache ca-certificates bash
RUN apk add --no-cache ca-certificates bash expat
WORKDIR /

View File

@@ -1,6 +1,6 @@
ARG BASE_IMAGE="alpine:latest"
ARG BASE_IMAGE
# Build the manager binary
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.16-alpine as builder
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.17-alpine as builder
ARG GOPROXY
ENV GOPROXY=${GOPROXY:-https://goproxy.cn}
WORKDIR /workspace
@@ -32,9 +32,9 @@ RUN GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} \
# Refer to https://github.com/GoogleContainerTools/distroless for more details
# Overwrite `BASE_IMAGE` by passing `--build-arg=BASE_IMAGE=gcr.io/distroless/static:nonroot`
FROM ${BASE_IMAGE:-alpine:latest}
FROM ${BASE_IMAGE:-alpine:3.15}
# This is required by daemon connnecting with cri
RUN apk add --no-cache ca-certificates bash
RUN apk add --no-cache ca-certificates bash expat
WORKDIR /

View File

@@ -1,5 +1,6 @@
ARG BASE_IMAGE
# Build the manager binary
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.16-alpine as builder
FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.17-alpine as builder
WORKDIR /workspace
# Copy the Go Modules manifests
@@ -24,8 +25,8 @@ ARG VERSION
ARG GITVERSION
RUN apk add gcc musl-dev libc-dev ;\
GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} \
go test -c -o manager-${TARGETARCH} -cover -covermode=atomic -coverpkg ./... .
GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} \
go test -c -o manager-${TARGETARCH} -cover -covermode=atomic -coverpkg ./... .
RUN GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} \
go build -a -ldflags "-s -w -X github.com/oam-dev/kubevela/version.VelaVersion=${VERSION:-undefined} -X github.com/oam-dev/kubevela/version.GitRevision=${GITVERSION:-undefined}" \
@@ -35,10 +36,10 @@ RUN GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} \
# You can replace distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
# Overwrite `BASE_IMAGE` by passing `--build-arg=BASE_IMAGE=gcr.io/distroless/static:nonroot`
ARG BASE_IMAGE
FROM ${BASE_IMAGE:-alpine:latest}
FROM ${BASE_IMAGE:-alpine:3.15}
# This is required by daemon connnecting with cri
RUN apk add --no-cache ca-certificates bash
RUN apk add --no-cache ca-certificates bash expat
WORKDIR /

View File

@@ -9,9 +9,12 @@ include makefiles/e2e.mk
all: build
# Run tests
test: vet lint staticcheck unit-test-core
test: vet lint staticcheck unit-test-core test-cli-gen
@$(OK) unit-tests pass
test-cli-gen:
mkdir -p ./bin/doc
go run ./hack/docgen/gen.go ./bin/doc
unit-test-core:
go test -coverprofile=coverage.txt $(shell go list ./pkg/... ./cmd/... ./apis/... | grep -v apiserver)
go test $(shell go list ./references/... | grep -v apiserver)

View File

@@ -213,6 +213,8 @@ const (
WorkflowStateFinished WorkflowState = "finished"
// WorkflowStateExecuting means workflow is still running or waiting some steps.
WorkflowStateExecuting WorkflowState = "executing"
// WorkflowStateSkipping means it will skip this reconcile and let next reconcile to handle it.
WorkflowStateSkipping WorkflowState = "skipping"
)
// ApplicationComponentStatus record the health status of App component

View File

@@ -1,3 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*

View File

@@ -1,3 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*

View File

@@ -1,3 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*

View File

@@ -1,3 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*

View File

@@ -17,7 +17,10 @@
package v1beta1
import (
"encoding/json"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
"github.com/oam-dev/kubevela/apis/core.oam.dev/common"
@@ -138,3 +141,36 @@ func (app *Application) GetComponent(workloadType string) *common.ApplicationCom
}
return nil
}
// Unstructured convert application to unstructured.Unstructured.
func (app *Application) Unstructured() (*unstructured.Unstructured, error) {
var obj = &unstructured.Unstructured{}
app.SetGroupVersionKind(ApplicationKindVersionKind)
bt, err := json.Marshal(app)
if err != nil {
return nil, err
}
if err := obj.UnmarshalJSON(bt); err != nil {
return nil, err
}
if app.Status.Services == nil {
if err := unstructured.SetNestedSlice(obj.Object, []interface{}{}, "status", "services"); err != nil {
return nil, err
}
}
if app.Status.AppliedResources == nil {
if err := unstructured.SetNestedSlice(obj.Object, []interface{}{}, "status", "appliedResources"); err != nil {
return nil, err
}
}
if wfStatus := app.Status.Workflow; wfStatus != nil && wfStatus.Steps == nil {
if err := unstructured.SetNestedSlice(obj.Object, []interface{}{}, "status", "workflow", "steps"); err != nil {
return nil, err
}
}
return obj, nil
}

View File

@@ -1,3 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*

View File

@@ -1,3 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*

View File

@@ -2,7 +2,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: vela-addon-registry
namespace: vela-system
namespace: {{.Values.systemDefinitionNamespace}}
data:
registries: '{
"KubeVela":{
@@ -13,4 +13,4 @@ data:
"path": ""
}
}
}'
}'

View File

@@ -22,6 +22,10 @@ spec:
app: {{ template "kubevela.name" . }}-admission-create
{{- include "kubevela.labels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: create
image: {{ .Values.imageRegistry }}{{ .Values.admissionWebhooks.patch.image.repository }}:{{ .Values.admissionWebhooks.patch.image.tag }}

View File

@@ -22,6 +22,10 @@ spec:
app: {{ template "kubevela.name" . }}-admission-patch
{{- include "kubevela.labels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: patch
image: {{ .Values.imageRegistry }}{{ .Values.admissionWebhooks.patch.image.repository }}:{{ .Values.admissionWebhooks.patch.image.tag }}

View File

@@ -198,6 +198,10 @@ spec:
app: {{ template "kubevela.fullname" . }}-cluster-gateway-tls-secret-create
{{- include "kubevela.labels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: create
image: {{ .Values.imageRegistry }}{{ .Values.admissionWebhooks.patch.image.repository }}:{{ .Values.admissionWebhooks.patch.image.tag }}
@@ -241,6 +245,10 @@ spec:
app: {{ template "kubevela.fullname" . }}-cluster-gateway-tls-secret-patch
{{- include "kubevela.labels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: patch
image: {{ .Values.imageRegistry }}{{ .Values.multicluster.clusterGateway.image.repository }}:{{ .Values.multicluster.clusterGateway.image.tag }}

View File

@@ -32,21 +32,30 @@ spec:
kind: "Ingress"
metadata: {
name: context.name
annotations: "kubernetes.io/ingress.class": parameter.class
annotations: {
if !parameter.classInSpec {
"kubernetes.io/ingress.class": parameter.class
}
}
}
spec: {
if parameter.classInSpec {
ingressClassName: parameter.class
}
rules: [{
host: parameter.domain
http: paths: [
for k, v in parameter.http {
path: k
pathType: "ImplementationSpecific"
backend: service: {
name: context.name
port: number: v
}
},
]
}]
}
spec: rules: [{
host: parameter.domain
http: paths: [
for k, v in parameter.http {
path: k
pathType: "ImplementationSpecific"
backend: service: {
name: context.name
port: number: v
}
},
]
}]
}
parameter: {
// +usage=Specify the domain you want to expose
@@ -57,6 +66,9 @@ spec:
// +usage=Specify the class of ingress to use
class: *"nginx" | string
// +usage=Set ingress class in '.spec.ingressClassName' instead of 'kubernetes.io/ingress.class' annotation.
classInSpec: *false | bool
}
status:
customStatus: |-

View File

@@ -11,6 +11,10 @@ spec:
schematic:
cue:
template: |
import (
"strconv"
)
mountsArray: {
pvc: *[
for v in parameter.volumeMounts.pvc {
@@ -152,6 +156,12 @@ spec:
{
containerPort: v.port
protocol: v.protocol
if v.name != _|_ {
name: v.name
}
if v.name == _|_ {
name: "port-" + strconv.FormatInt(v.port, 10)
}
}}]
}
@@ -262,6 +272,12 @@ spec:
for v in parameter.ports if v.expose == true {
port: v.port
targetPort: v.port
if v.name != _|_ {
name: v.name
}
if v.name == _|_ {
name: "port-" + strconv.FormatInt(v.port, 10)
}
},
]
outputs: {
@@ -304,6 +320,8 @@ spec:
ports?: [...{
// +usage=Number of port to expose on the pod's IP address
port: int
// +usage=Name of the port
name?: string
// +usage=Protocol for port. Must be UDP, TCP, or SCTP
protocol: *"TCP" | "UDP" | "SCTP"
// +usage=Specify if the port should be exposed

View File

@@ -5,6 +5,7 @@ metadata:
helm.sh/hook: test-success
helm.sh/hook-delete-policy: hook-succeeded
name: helm-test-vela-app
namespace: {{.Values.systemDefinitionNamespace}}
spec:
components:
- name: helm-test-express-server
@@ -23,6 +24,7 @@ apiVersion: v1
kind: Pod
metadata:
name: "{{ .Release.Name }}-application-test"
namespace: {{.Values.systemDefinitionNamespace}}
annotations:
"helm.sh/hook": test
helm.sh/hook-delete-policy: hook-succeeded
@@ -42,16 +44,16 @@ spec:
echo "Waiting application is ready..."
echo "waiting for application being Ready"
kubectl -n vela-system wait --for=condition=Ready applications.core.oam.dev helm-test-vela-app --timeout=3m
kubectl -n {{.Values.systemDefinitionNamespace}} wait --for=condition=Ready applications.core.oam.dev helm-test-vela-app --timeout=3m
echo "application is Ready"
# wait for deploy being created
echo "waiting for deployment being available"
kubectl -n vela-system wait --for=condition=available deployments helm-test-express-server --timeout 3m
kubectl -n {{.Values.systemDefinitionNamespace}} wait --for=condition=available deployments helm-test-express-server --timeout 3m
echo "deployment being available"
# wait for ingress being created
while ! [ `kubectl -n vela-system get ing helm-test-express-server | grep -v NAME | wc -l` = 1 ]; do
while ! [ `kubectl -n {{.Values.systemDefinitionNamespace}} get ing helm-test-express-server | grep -v NAME | wc -l` = 1 ]; do
echo "waiting for ingress being created"
sleep 1
done
@@ -59,4 +61,4 @@ spec:
echo "Application and its components are created"
restartPolicy: Never
restartPolicy: Never

View File

@@ -117,7 +117,7 @@ multicluster:
image:
repository: oamdev/cluster-gateway
tag: v1.1.7
pullPolicy: Always
pullPolicy: IfNotPresent
resources:
limits:
cpu: 100m

View File

@@ -32,21 +32,30 @@ spec:
kind: "Ingress"
metadata: {
name: context.name
annotations: "kubernetes.io/ingress.class": parameter.class
annotations: {
if !parameter.classInSpec {
"kubernetes.io/ingress.class": parameter.class
}
}
}
spec: {
if parameter.classInSpec {
ingressClassName: parameter.class
}
rules: [{
host: parameter.domain
http: paths: [
for k, v in parameter.http {
path: k
pathType: "ImplementationSpecific"
backend: service: {
name: context.name
port: number: v
}
},
]
}]
}
spec: rules: [{
host: parameter.domain
http: paths: [
for k, v in parameter.http {
path: k
pathType: "ImplementationSpecific"
backend: service: {
name: context.name
port: number: v
}
},
]
}]
}
parameter: {
// +usage=Specify the domain you want to expose
@@ -57,6 +66,9 @@ spec:
// +usage=Specify the class of ingress to use
class: *"nginx" | string
// +usage=Set ingress class in '.spec.ingressClassName' instead of 'kubernetes.io/ingress.class' annotation.
classInSpec: *false | bool
}
status:
customStatus: |-

View File

@@ -11,6 +11,10 @@ spec:
schematic:
cue:
template: |
import (
"strconv"
)
mountsArray: {
pvc: *[
for v in parameter.volumeMounts.pvc {
@@ -152,6 +156,12 @@ spec:
{
containerPort: v.port
protocol: v.protocol
if v.name != _|_ {
name: v.name
}
if v.name == _|_ {
name: "port-" + strconv.FormatInt(v.port, 10)
}
}}]
}
@@ -262,6 +272,12 @@ spec:
for v in parameter.ports if v.expose == true {
port: v.port
targetPort: v.port
if v.name != _|_ {
name: v.name
}
if v.name == _|_ {
name: "port-" + strconv.FormatInt(v.port, 10)
}
},
]
outputs: {
@@ -304,6 +320,8 @@ spec:
ports?: [...{
// +usage=Number of port to expose on the pod's IP address
port: int
// +usage=Name of the port
name?: string
// +usage=Protocol for port. Must be UDP, TCP, or SCTP
protocol: *"TCP" | "UDP" | "SCTP"
// +usage=Specify if the port should be exposed

View File

@@ -46,6 +46,7 @@ import (
"github.com/oam-dev/kubevela/pkg/multicluster"
"github.com/oam-dev/kubevela/pkg/oam"
"github.com/oam-dev/kubevela/pkg/oam/discoverymapper"
"github.com/oam-dev/kubevela/pkg/resourcekeeper"
"github.com/oam-dev/kubevela/pkg/utils/common"
"github.com/oam-dev/kubevela/pkg/utils/system"
oamwebhook "github.com/oam-dev/kubevela/pkg/webhook/core.oam.dev"
@@ -129,7 +130,9 @@ func main() {
flag.DurationVar(&retryPeriod, "leader-election-retry-period", 2*time.Second,
"The duration the LeaderElector clients should wait between tries of actions")
flag.BoolVar(&enableClusterGateway, "enable-cluster-gateway", false, "Enable cluster-gateway to use multicluster, disabled by default.")
flag.BoolVar(&controllerArgs.EnableCompatibility, "enable-asi-compatibility", false, "enable compatibility for asi")
standardcontroller.AddOptimizeFlags()
flag.IntVar(&resourcekeeper.MaxDispatchConcurrent, "max-dispatch-concurrent", 10, "Set the max dispatch concurrent number, default is 10")
flag.Parse()
// setup logging

View File

@@ -4,7 +4,7 @@ This guide helps you get started developing KubeVela.
## Prerequisites
1. Golang version 1.16+
1. Golang version 1.17+
2. Kubernetes version v1.18+ with `~/.kube/config` configured.
3. ginkgo 1.14.0+ (just for [E2E test](./developer-guide.md#e2e-test))
4. golangci-lint 1.38.0+, it will install automatically if you run `make`, you can [install it manually](https://golangci-lint.run/usage/install/#local-installation) if the installation is too slow.
@@ -15,6 +15,7 @@ This guide helps you get started developing KubeVela.
<summary>Install Kubebuilder manually</summary>
linux:
```
wget https://storage.googleapis.com/kubebuilder-tools/kubebuilder-tools-1.21.2-linux-amd64.tar.gz
tar -zxvf kubebuilder-tools-1.21.2-linux-amd64.tar.gz
@@ -23,6 +24,7 @@ sudo mv kubebuilder/bin/* /usr/local/kubebuilder/bin
```
macOS:
```
wget https://storage.googleapis.com/kubebuilder-tools/kubebuilder-tools-1.21.2-darwin-amd64.tar.gz
tar -zxvf kubebuilder-tools-1.21.2-darwin-amd64.tar.gz
@@ -30,14 +32,15 @@ mkdir -p /usr/local/kubebuilder/bin
sudo mv kubebuilder/bin/* /usr/local/kubebuilder/bin
```
For other OS or system architecture, please refer to https://storage.googleapis.com/kubebuilder-tools/
For other OS or system architecture, please refer to https://storage.googleapis.com/kubebuilder-tools/
</details>
You may also be interested with KubeVela's [design](https://github.com/oam-dev/kubevela/tree/master/design/vela-core) before diving into its code.
## Build
* Clone this project
- Clone this project
```shell script
git clone git@github.com:oam-dev/kubevela.git
@@ -50,7 +53,7 @@ KubeVela includes two parts, `vela core` and `vela cli`.
For local development, we probably need to build both of them.
* Build Vela CLI
- Build Vela CLI
```shell script
make
@@ -58,7 +61,7 @@ make
After the vela cli built successfully, `make` command will create `vela` binary to `bin/` under the project.
* Configure `vela` binary to System PATH
- Configure `vela` binary to System PATH
```shell script
export PATH=$PATH:/your/path/to/project/kubevela/bin
@@ -66,13 +69,13 @@ export PATH=$PATH:/your/path/to/project/kubevela/bin
Then you can use `vela` command directly.
* Build Vela Core
- Build Vela Core
```shell script
make manager
```
* Run Vela Core
- Run Vela Core
Firstly make sure your cluster has CRDs, below is the command that can help install all CRDs.
@@ -82,11 +85,13 @@ make core-install
To ensure you have created vela-system namespace and install definitions of necessary module.
you can run the command:
```shell script
make def-install
```
And then run locally:
```shell script
make core-run
```
@@ -182,8 +187,7 @@ mv ~/.kube/config.save ~/.kube/config
make e2e-apiserver-test
```
## Next steps
* Read our [code conventions](coding-conventions.md)
* Learn how to [Create a pull request](create-pull-request.md)
- Read our [code conventions](coding-conventions.md)
- Learn how to [Create a pull request](create-pull-request.md)

View File

@@ -14,8 +14,12 @@ spec:
ports:
- port: 8000
- port: 8001
name: exposeport1
protocol: UDP
expose: true
expose: true
- port: 8002
protocol: UDP
expose: true
volumeMounts:
pvc:
- name: my-mount

View File

@@ -0,0 +1,74 @@
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
name: example-app
namespace: default
spec:
components:
- name: podinfo
type: webservice
properties:
image: stefanprodan/podinfo
traits:
- type: scaler
properties:
replicas: 1
- name: hello-world
type: webservice
properties:
image: crccheck/hello-world
traits:
- type: scaler
properties:
replicas: 1
- name: nginx
type: worker
properties:
image: nginx
traits:
- type: scaler
properties:
replicas: 1
policies:
- name: example-multi-env-policy
type: env-binding
properties:
envs:
- name: test
placement:
clusterSelector:
name: local
namespaceSelector:
name: test
patch:
components:
- name: podinfo # patch to component named podinfo, no type check
traits:
- type: scaler
properties:
replicas: 2
- name: staging
placement:
clusterSelector:
name: remote
patch:
components: # patch to all webservice components
- type: webservice
traits:
- type: scaler
properties:
replicas: 3
- name: prod
placement:
clusterSelector:
name: remote
namespaceSelector:
name: prod
patch:
components: # patch to all components
- traits:
- type: scaler
properties:
replicas: 3

186
go.mod
View File

@@ -1,6 +1,6 @@
module github.com/oam-dev/kubevela
go 1.16
go 1.17
require (
cuelang.org/go v0.2.2
@@ -90,7 +90,191 @@ require (
sigs.k8s.io/yaml v1.2.0
)
require (
cloud.google.com/go v0.81.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest v0.11.18 // indirect
github.com/Azure/go-autorest/autorest/adal v0.9.13 // indirect
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/BurntSushi/toml v0.3.1 // indirect
github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver v1.5.0 // indirect
github.com/Masterminds/semver/v3 v3.1.1 // indirect
github.com/Masterminds/sprig v2.22.0+incompatible // indirect
github.com/Masterminds/sprig/v3 v3.2.2 // indirect
github.com/Masterminds/squirrel v1.5.0 // indirect
github.com/Microsoft/go-winio v0.4.16 // indirect
github.com/Microsoft/hcsshim v0.8.14 // indirect
github.com/PuerkitoBio/purell v1.1.1 // indirect
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/agext/levenshtein v1.2.2 // indirect
github.com/alessio/shellescape v1.2.2 // indirect
github.com/alibabacloud-go/debug v0.0.0-20190504072949-9472017b5c68 // indirect
github.com/alibabacloud-go/endpoint-util v1.1.0 // indirect
github.com/alibabacloud-go/openapi-util v0.0.7 // indirect
github.com/alibabacloud-go/tea-utils v1.3.9 // indirect
github.com/aliyun/credentials-go v1.1.2 // indirect
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535 // indirect
github.com/aws/aws-sdk-go v1.36.30 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
github.com/cespare/xxhash/v2 v2.1.1 // indirect
github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5 // indirect
github.com/cockroachdb/apd/v2 v2.0.1 // indirect
github.com/containerd/cgroups v0.0.0-20200531161412-0dbf7f05ba59 // indirect
github.com/containerd/continuity v0.0.0-20201208142359-180525291bb7 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.0 // indirect
github.com/creack/pty v1.1.11 // indirect
github.com/cyphar/filepath-securejoin v0.2.2 // indirect
github.com/deislabs/oras v0.11.1 // indirect
github.com/docker/cli v20.10.5+incompatible // indirect
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/docker v17.12.0-ce-rc1.0.20200618181300-9dc6525e6118+incompatible // indirect
github.com/docker/docker-credential-helpers v0.6.3 // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-metrics v0.0.0-20180209012529-399ea8c73916 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/emicklei/go-restful v2.9.5+incompatible // indirect
github.com/emicklei/proto v1.6.15 // indirect
github.com/emirpasic/gods v1.12.0 // indirect
github.com/evanphx/json-patch/v5 v5.1.0 // indirect
github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d // indirect
github.com/fatih/camelcase v1.0.0 // indirect
github.com/form3tech-oss/jwt-go v3.2.3+incompatible // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-errors/errors v1.0.1 // indirect
github.com/go-logr/zapr v0.4.0 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
github.com/go-openapi/jsonreference v0.19.5 // indirect
github.com/go-openapi/swag v0.19.14 // indirect
github.com/go-playground/locales v0.14.0 // indirect
github.com/go-playground/universal-translator v0.18.0 // indirect
github.com/go-stack/stack v1.8.0 // indirect
github.com/gobuffalo/flect v0.2.3 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.1 // indirect
github.com/google/btree v1.0.1 // indirect
github.com/google/go-querystring v1.0.0 // indirect
github.com/google/gofuzz v1.1.0 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/googleapis/gnostic v0.5.5 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/jmoiron/sqlx v1.3.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.11 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 // indirect
github.com/klauspost/compress v1.11.0 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/kr/pty v1.1.8 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 // indirect
github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect
github.com/leodido/go-urn v1.2.1 // indirect
github.com/lib/pq v1.10.0 // indirect
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
github.com/mailru/easyjson v0.7.6 // indirect
github.com/mattn/go-colorable v0.1.8 // indirect
github.com/mattn/go-isatty v0.0.12 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/go-wordwrap v1.0.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/spdystream v0.2.0 // indirect
github.com/moby/term v0.0.0-20210610120745-9d4ed1856297 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/mpvl/unique v0.0.0-20150818121801-cbe035fff7de // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/pelletier/go-toml v1.9.3 // indirect
github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/common v0.26.0 // indirect
github.com/prometheus/procfs v0.6.0 // indirect
github.com/rogpeppe/go-internal v1.8.0 // indirect
github.com/rubenv/sql-migrate v0.0.0-20200616145509-8d140a17f351 // indirect
github.com/russross/blackfriday v1.5.2 // indirect
github.com/russross/blackfriday/v2 v2.0.1 // indirect
github.com/sergi/go-diff v1.1.0 // indirect
github.com/shopspring/decimal v1.2.0 // indirect
github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
github.com/spf13/afero v1.6.0 // indirect
github.com/spf13/cast v1.3.1 // indirect
github.com/src-d/gcfg v1.4.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
github.com/tjfoc/gmsm v1.3.2 // indirect
github.com/xanzy/ssh-agent v0.3.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.0.2 // indirect
github.com/xdg-go/stringprep v1.0.2 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca // indirect
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
github.com/zclconf/go-cty v1.8.0 // indirect
go.opencensus.io v0.23.0 // indirect
go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 // indirect
go.uber.org/atomic v1.7.0 // indirect
go.uber.org/multierr v1.6.0 // indirect
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97 // indirect
golang.org/x/mod v0.4.2 // indirect
golang.org/x/net v0.0.0-20211029224645-99673261e6eb // indirect
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b // indirect
golang.org/x/text v0.3.6 // indirect
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c // indirect
google.golang.org/grpc v1.38.0 // indirect
google.golang.org/protobuf v1.26.0 // indirect
gopkg.in/gorp.v1 v1.7.2 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.62.0 // indirect
gopkg.in/src-d/go-billy.v4 v4.3.2 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
istio.io/api v0.0.0-20210128181506-0c4b8e54850f // indirect
istio.io/gogo-genproto v0.0.0-20190930162913-45029607206a // indirect
k8s.io/apiserver v0.22.1 // indirect
k8s.io/component-base v0.22.1 // indirect
sigs.k8s.io/apiserver-network-proxy v0.0.24 // indirect
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.24 // indirect
sigs.k8s.io/apiserver-runtime v1.0.3-0.20210913073608-0663f60bfee2 // indirect
sigs.k8s.io/kustomize/api v0.8.5 // indirect
sigs.k8s.io/kustomize/kyaml v0.10.15 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.1.2 // indirect
)
replace (
github.com/docker/cli => github.com/docker/cli v20.10.9+incompatible
github.com/docker/docker => github.com/moby/moby v17.12.0-ce-rc1.0.20200618181300-9dc6525e6118+incompatible
github.com/wercker/stern => github.com/oam-dev/stern v1.13.2
sigs.k8s.io/apiserver-network-proxy/konnectivity-client => sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.24

4
go.sum
View File

@@ -406,8 +406,8 @@ github.com/dgryski/go-bitstream v0.0.0-20180413035011-3522498ce2c8/go.mod h1:VMa
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/dgryski/go-sip13 v0.0.0-20190329191031-25c5027a8c7b/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/dnaeon/go-vcr v1.0.1/go.mod h1:aBB1+wY4s93YsC3HHjMBMrwTj2R9FHDzUr9KyGc8n1E=
github.com/docker/cli v20.10.5+incompatible h1:bjflayQbWg+xOkF2WPEAOi4Y7zWhR7ptoPhV/VqLVDE=
github.com/docker/cli v20.10.5+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/cli v20.10.9+incompatible h1:OJ7YkwQA+k2Oi51lmCojpjiygKpi76P7bg91b2eJxYU=
github.com/docker/cli v20.10.9+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/distribution v0.0.0-20191216044856-a8371794149d/go.mod h1:0+TTO4EOBfRPhZXAeF1Vu+W3hHZ8eLp8PgKVZlcvtFY=
github.com/docker/distribution v2.7.1+incompatible h1:a5mlkVzth6W5A4fOsS3D2EO5BUmsJpcB+cRlLU7cSug=
github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=

View File

@@ -37,7 +37,7 @@ goimports:
ifeq (, $(shell which goimports))
@{ \
set -e ;\
GO111MODULE=off go get -u golang.org/x/tools/cmd/goimports ;\
go install golang.org/x/tools/cmd/goimports@latest ;\
}
GOIMPORTS=$(GOBIN)/goimports
else
@@ -49,7 +49,7 @@ installcue:
ifeq (, $(shell which cue))
@{ \
set -e ;\
GO111MODULE=off go get -u cuelang.org/go/cmd/cue ;\
go install cuelang.org/go/cmd/cue@latest ;\
}
CUE=$(GOBIN)/cue
else

View File

@@ -324,6 +324,8 @@ const (
PayloadTypeACR = "acr"
// PayloadTypeHarbor is the payload type harbor
PayloadTypeHarbor = "harbor"
// PayloadTypeJFrog is the payload type jfrog
PayloadTypeJFrog = "jfrog"
// ComponentTypeWebservice is the component type webservice
ComponentTypeWebservice = "webservice"
@@ -336,6 +338,10 @@ const (
const (
// HarborEventTypePushArtifact is the event type PUSH_ARTIFACT
HarborEventTypePushArtifact = "PUSH_ARTIFACT"
// JFrogEventTypePush is push event type of jfrog webhook
JFrogEventTypePush = "pushed"
// JFrogDomainDocker is webhook domain of jfrog docker
JFrogDomainDocker = "docker"
)
// TableName return custom table name

View File

@@ -464,6 +464,24 @@ type DockerHubRepository struct {
Status string `json:"status"`
}
// HandleApplicationTriggerJFrogRequest application trigger JFrog webhook request
type HandleApplicationTriggerJFrogRequest struct {
Domain string `json:"domain"`
EventType string `json:"event_type"`
Data JFrogWebhookData `json:"data"`
}
// JFrogWebhookData is the data of JFrog webhook request
type JFrogWebhookData struct {
URL string
ImageName string `json:"image_name"`
Name string `json:"name"`
Path string `json:"path"`
RepoKey string `json:"repo_key"`
Digest string `json:"sha256"`
Tag string `json:"tag"`
}
// EnvBinding application env binding
type EnvBinding struct {
Name string `json:"name" validate:"checkname"`

View File

@@ -18,6 +18,7 @@ package usecase
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
@@ -25,6 +26,8 @@ import (
"sync"
"time"
k8stypes "k8s.io/apimachinery/pkg/types"
v1 "k8s.io/api/core/v1"
errors2 "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
@@ -39,6 +42,7 @@ import (
"github.com/oam-dev/kubevela/pkg/apiserver/clients"
"github.com/oam-dev/kubevela/pkg/apiserver/log"
apis "github.com/oam-dev/kubevela/pkg/apiserver/rest/apis/v1"
"github.com/oam-dev/kubevela/pkg/apiserver/rest/utils"
"github.com/oam-dev/kubevela/pkg/apiserver/rest/utils/bcode"
"github.com/oam-dev/kubevela/pkg/oam"
"github.com/oam-dev/kubevela/pkg/utils/apply"
@@ -155,7 +159,9 @@ func (u *defaultAddonHandler) GetAddon(ctx context.Context, name string, registr
if addon == nil {
return nil, bcode.ErrAddonNotExist
}
addon.UISchema = renderDefaultUISchema(addon.APISchema)
addon.UISchema = renderAddonCustomUISchema(ctx, u.kubeClient, name, renderDefaultUISchema(addon.APISchema))
a, err := AddonImpl2AddonRes(addon)
if err != nil {
return nil, err
@@ -455,6 +461,29 @@ func convertAppStateToAddonPhase(state common2.ApplicationPhase) apis.AddonPhase
}
}
func renderAddonCustomUISchema(ctx context.Context, cli client.Client, addonName string, defaultSchema []*utils.UIParameter) []*utils.UIParameter {
var cm v1.ConfigMap
if err := cli.Get(ctx, k8stypes.NamespacedName{
Namespace: types.DefaultKubeVelaNS,
Name: fmt.Sprintf("addon-uischema-%s", addonName),
}, &cm); err != nil {
if !errors2.IsNotFound(err) {
log.Logger.Errorf("find uischema configmap from cluster failure %s", err.Error())
}
return defaultSchema
}
data, ok := cm.Data[types.UISchema]
if !ok {
return defaultSchema
}
schema := []*utils.UIParameter{}
if err := json.Unmarshal([]byte(data), &schema); err != nil {
log.Logger.Errorf("unmarshal ui schema failure %s", err.Error())
return defaultSchema
}
return patchSchema(defaultSchema, schema)
}
// ConvertAddonRegistryModel2AddonRegistryMeta will convert from model to AddonRegistry
func ConvertAddonRegistryModel2AddonRegistryMeta(r pkgaddon.Registry) apis.AddonRegistry {
return apis.AddonRegistry{

View File

@@ -0,0 +1,106 @@
/*
Copyright 2021 The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package usecase
import (
"context"
"fmt"
"io/ioutil"
"github.com/oam-dev/kubevela/pkg/oam/util"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/yaml"
"github.com/oam-dev/kubevela/apis/types"
"github.com/oam-dev/kubevela/pkg/apiserver/rest/utils"
)
var _ = Describe("addon usecase test", func() {
var ctx context.Context
BeforeEach(func() {
ctx = context.Background()
Expect(k8sClient.Create(ctx, &v1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: types.DefaultKubeVelaNS}})).Should(SatisfyAny(BeNil(), util.AlreadyExistMatcher{}))
})
It("Test render customize ui-schema", func() {
schemaData, err := ioutil.ReadFile("testdata/addon-uischema-test.yaml")
addonName := "test"
Expect(err).Should(BeNil())
jsonData, err := yaml.YAMLToJSON(schemaData)
Expect(err).Should(BeNil())
cm := v1.ConfigMap{
TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"},
ObjectMeta: metav1.ObjectMeta{Namespace: types.DefaultKubeVelaNS, Name: fmt.Sprintf("addon-uischema-%s", addonName)},
Data: map[string]string{
types.UISchema: string(jsonData),
}}
Expect(k8sClient.Create(ctx, &cm)).Should(BeNil())
defaultSchema := []*utils.UIParameter{
{
JSONKey: "version",
Sort: 3,
},
{
JSONKey: "domain",
Sort: 8,
},
}
res := renderAddonCustomUISchema(ctx, k8sClient, addonName, defaultSchema)
Expect(len(res)).Should(BeEquivalentTo(2))
for _, re := range res {
if re.JSONKey == "version" {
Expect(re.Validate.DefaultValue.(string)).Should(BeEquivalentTo("1.2.0-rc1"))
Expect(re.Sort).Should(BeEquivalentTo(1))
}
if re.JSONKey == "domain" {
Expect(re.Sort).Should(BeEquivalentTo(9))
}
}
})
It("Test render without ui-schema", func() {
addonName := "test-without-schema"
defaultSchema := []*utils.UIParameter{
{
JSONKey: "version",
Sort: 3,
},
{
JSONKey: "domain",
Sort: 8,
},
}
res := renderAddonCustomUISchema(ctx, k8sClient, addonName, defaultSchema)
Expect(len(res)).Should(BeEquivalentTo(2))
for _, re := range res {
if re.JSONKey == "version" {
Expect(re.Validate).Should(BeNil())
Expect(re.Sort).Should(BeEquivalentTo(3))
}
if re.JSONKey == "domain" {
Expect(re.Sort).Should(BeEquivalentTo(8))
}
}
})
})

View File

@@ -0,0 +1,18 @@
- jsonKey: version
validate:
defaultValue: 1.2.0-rc1
sort: 1
- jsonKey: dbType
label: DBType
validate:
defaultValue: kubeapi
sort: 3
- jsonKey: dbURL
label: DBURL
sort: 5
- jsonKey: database
sort: 7
validate:
defaultValue: kubevela
- jsonKey: domain
sort: 9

View File

@@ -62,6 +62,7 @@ func registerHandlers() {
new(acrHandlerImpl).install()
new(dockerHubHandlerImpl).install()
new(harborHandlerImpl).install()
new(jfrogHandlerImpl).install()
}
type webhookHandler interface {
@@ -160,6 +161,11 @@ func (c *webhookUsecaseImpl) HandleApplicationWebhook(ctx context.Context, token
if err != nil {
return nil, err
}
case model.PayloadTypeJFrog:
handler, err = c.newJFrogHandler(req)
if err != nil {
return nil, err
}
default:
return nil, bcode.ErrInvalidWebhookPayloadType
}
@@ -423,3 +429,74 @@ func (c *harborHandlerImpl) handle(ctx context.Context, webhookTrigger *model.Ap
},
})
}
type jfrogHandlerImpl struct {
req apisv1.HandleApplicationTriggerJFrogRequest
w *webhookUsecaseImpl
}
func (c *webhookUsecaseImpl) newJFrogHandler(req *restful.Request) (webhookHandler, error) {
var jfrogReq apisv1.HandleApplicationTriggerJFrogRequest
if err := req.ReadEntity(&jfrogReq); err != nil {
return nil, bcode.ErrInvalidWebhookPayloadBody
}
if jfrogReq.Domain != model.JFrogDomainDocker || jfrogReq.EventType != model.JFrogEventTypePush {
return nil, bcode.ErrInvalidWebhookPayloadBody
}
// jfrog should use request header to give URL, it is not exist in request body
jfrogReq.Data.URL = req.HeaderParameter("X-JFrogURL")
return &jfrogHandlerImpl{
req: jfrogReq,
w: c,
}, nil
}
func (j *jfrogHandlerImpl) handle(ctx context.Context, webhookTrigger *model.ApplicationTrigger, app *model.Application) (interface{}, error) {
jfrogReq := j.req
comp := &model.ApplicationComponent{
AppPrimaryKey: webhookTrigger.AppPrimaryKey,
}
comps, err := j.w.ds.List(ctx, comp, &datastore.ListOptions{})
if err != nil {
return nil, err
}
if len(comps) == 0 {
return nil, bcode.ErrApplicationComponetNotExist
}
// use the first component as the target component
component := comps[0].(*model.ApplicationComponent)
image := fmt.Sprintf("%s/%s:%s", jfrogReq.Data.RepoKey, jfrogReq.Data.ImageName, jfrogReq.Data.Tag)
if jfrogReq.Data.URL != "" {
image = fmt.Sprintf("%s/%s", jfrogReq.Data.URL, image)
}
if err := j.w.patchComponentProperties(ctx, component, &runtime.RawExtension{
Raw: []byte(fmt.Sprintf(`{"image": "%s"}`, image)),
}); err != nil {
return nil, err
}
return j.w.applicationUsecase.Deploy(ctx, app, apisv1.ApplicationDeployRequest{
WorkflowName: webhookTrigger.WorkflowName,
Note: "triggered by webhook jfrog",
TriggerType: apisv1.TriggerTypeWebhook,
Force: true,
ImageInfo: &model.ImageInfo{
Type: model.PayloadTypeHarbor,
Resource: &model.ImageResource{
Digest: jfrogReq.Data.Digest,
Tag: jfrogReq.Data.Tag,
URL: image,
},
Repository: &model.ImageRepository{
Name: jfrogReq.Data.ImageName,
Namespace: jfrogReq.Data.RepoKey,
FullName: fmt.Sprintf("%s/%s", jfrogReq.Data.RepoKey, jfrogReq.Data.ImageName),
},
},
})
}
func (j *jfrogHandlerImpl) install() {
WebhookHandlers = append(WebhookHandlers, model.PayloadTypeJFrog)
}

View File

@@ -256,5 +256,45 @@ var _ = Describe("Test application usecase function", func() {
comp, err = appUsecase.GetApplicationComponent(context.TODO(), appModel, "component-name-webhook")
Expect(err).Should(BeNil())
Expect((*comp.Properties)["image"]).Should(Equal("docker.io/test-namespace/test-repo:test-tag"))
By("Test HandleApplicationWebhook function with jfrog payload without header of X-JFrogURL")
jfrogTrigger, err := appUsecase.CreateApplicationTrigger(context.TODO(), appModel, apisv1.CreateApplicationTriggerRequest{
Name: "test-jfrog",
PayloadType: "jfrog",
Type: "webhook",
ComponentName: "component-name-webhook",
})
Expect(err).Should(BeNil())
jfrogBody := apisv1.HandleApplicationTriggerJFrogRequest{
Domain: "docker",
EventType: "pushed",
Data: apisv1.JFrogWebhookData{
ImageName: "test-image",
RepoKey: "test-repo",
Digest: "test-digest",
Tag: "test-tag",
},
}
body, err = json.Marshal(jfrogBody)
Expect(err).Should(BeNil())
httpreq, err = http.NewRequest("post", "/", bytes.NewBuffer(body))
httpreq.Header.Add(restful.HEADER_ContentType, "application/json")
Expect(err).Should(BeNil())
_, err = webhookUsecase.HandleApplicationWebhook(context.TODO(), jfrogTrigger.Token, restful.NewRequest(httpreq))
Expect(err).Should(BeNil())
comp, err = appUsecase.GetApplicationComponent(context.TODO(), appModel, "component-name-webhook")
Expect(err).Should(BeNil())
Expect((*comp.Properties)["image"]).Should(Equal("test-repo/test-image:test-tag"))
By("Test HandleApplicationWebhook function with jfrog payload with header of X-JFrogURL")
httpreq, err = http.NewRequest("post", "/", bytes.NewBuffer(body))
Expect(err).Should(BeNil())
httpreq.Header.Add(restful.HEADER_ContentType, "application/json")
httpreq.Header.Add("X-JFrogURL", "test-addr")
_, err = webhookUsecase.HandleApplicationWebhook(context.TODO(), jfrogTrigger.Token, restful.NewRequest(httpreq))
Expect(err).Should(BeNil())
comp, err = appUsecase.GetApplicationComponent(context.TODO(), appModel, "component-name-webhook")
Expect(err).Should(BeNil())
Expect((*comp.Properties)["image"]).Should(Equal("test-addr/test-repo/test-image:test-tag"))
})
})

View File

@@ -504,6 +504,7 @@ func generateComponentFromTerraformModule(wl *Workload, appName, ns string) (*ty
func baseGenerateComponent(pCtx process.Context, wl *Workload, appName, ns string) (*types.ComponentManifest, error) {
var err error
pCtx.PushData(model.ContextComponentType, wl.Type)
for _, tr := range wl.Traits {
if err := tr.EvalContext(pCtx); err != nil {
return nil, errors.Wrapf(err, "evaluate template trait=%s app=%s", tr.Name, wl.Name)

View File

@@ -25,6 +25,7 @@ import (
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"cuelang.org/go/cue"
"github.com/crossplane/crossplane-runtime/pkg/test"
"github.com/google/go-cmp/cmp"
terraformtypes "github.com/oam-dev/terraform-controller/api/types/crossplane-runtime"
@@ -1322,3 +1323,47 @@ spec:
}
}
func TestBaseGenerateComponent(t *testing.T) {
var appName = "test-app"
var ns = "test-ns"
var traitName = "mytrait"
var wlName = "my-wl-1"
pContext := NewBasicContext(appName, wlName, "rev-1", ns, nil)
base := `
apiVersion: "apps/v1"
kind: "Deployment"
spec: {
template: {
spec: containers: [{
image: "nginx"
}]
}
}
`
var r cue.Runtime
inst, err := r.Compile("-", base)
assert.NilError(t, err)
bs, _ := model.NewBase(inst.Value())
err = pContext.SetBase(bs)
assert.NilError(t, err)
tr := &Trait{
Name: traitName,
engine: definition.NewTraitAbstractEngine(traitName, nil),
Template: `outputs:mytrait:{
if context.componentType == "stateless" {
kind: "Deployment"
}
if context.componentType == "stateful" {
kind: "StatefulSet"
}
name: context.name
envSourceContainerName: context.name
}`,
}
wl := &Workload{Type: "stateful", Traits: []*Trait{tr}}
cm, err := baseGenerateComponent(pContext, wl, appName, ns)
assert.NilError(t, err)
assert.Equal(t, cm.Traits[0].Object["kind"], "StatefulSet")
assert.Equal(t, cm.Traits[0].Object["name"], wlName)
}

View File

@@ -35,7 +35,7 @@ func (p *Parser) ValidateCUESchematicAppfile(a *Appfile) error {
}
pCtx, err := newValidationProcessContext(wl, a.Name, a.AppRevisionName, a.Namespace)
if err != nil {
return errors.WithMessage(err, "cannot create validationg process context")
return errors.WithMessagef(err, "cannot create the validation process context of app=%s in namespace=%s", a.Name, a.Namespace)
}
for _, tr := range wl.Traits {
if tr.CapabilityCategory != types.CUECategory {

View File

@@ -21,7 +21,6 @@ import (
"strings"
"sync"
"github.com/oam-dev/kubevela/pkg/oam"
"github.com/oam-dev/kubevela/pkg/resourcetracker"
"github.com/pkg/errors"
@@ -87,15 +86,13 @@ func DefaultNewControllerClient(cache cache.Cache, config *rest.Config, options
AddFunc: func(obj interface{}) {
lock.Lock()
rtCount++
metrics.ResourceTrackerNumberGauge.WithLabelValues(
metrics.ExtractMetricValuesFromObjectLabel(obj, oam.LabelAppName, oam.LabelAppNamespace)...).Set(float64(rtCount))
metrics.ResourceTrackerNumberGauge.WithLabelValues("application").Set(float64(rtCount))
lock.Unlock()
},
DeleteFunc: func(obj interface{}) {
lock.Lock()
rtCount--
metrics.ResourceTrackerNumberGauge.WithLabelValues(
metrics.ExtractMetricValuesFromObjectLabel(obj, oam.LabelAppName, oam.LabelAppNamespace)...).Set(float64(rtCount))
metrics.ResourceTrackerNumberGauge.WithLabelValues("application").Set(float64(rtCount))
lock.Unlock()
},
})

View File

@@ -80,4 +80,7 @@ type Args struct {
// OAMSpecVer is the oam spec version controller want to setup
OAMSpecVer string
// EnableCompatibility indicates that will change some functions of controller to adapt to multiple platforms, such as asi.
EnableCompatibility bool
}

View File

@@ -81,12 +81,17 @@ var (
// Reconciler reconciles an Application object
type Reconciler struct {
client.Client
dm discoverymapper.DiscoveryMapper
pd *packages.PackageDiscover
Scheme *runtime.Scheme
Recorder event.Recorder
dm discoverymapper.DiscoveryMapper
pd *packages.PackageDiscover
Scheme *runtime.Scheme
Recorder event.Recorder
options
}
type options struct {
appRevisionLimit int
concurrentReconciles int
disableStatusUpdate bool
}
// +kubebuilder:rbac:groups=core.oam.dev,resources=applications,verbs=get;list;watch;create;update;patch;delete
@@ -121,6 +126,8 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu
if annotations := app.GetAnnotations(); annotations == nil || annotations[oam.AnnotationKubeVelaVersion] == "" {
metav1.SetMetaDataAnnotation(&app.ObjectMeta, oam.AnnotationKubeVelaVersion, version.VelaVersion)
}
logCtx.AddTag("publish_version", app.GetAnnotations()[oam.AnnotationKubeVelaVersion])
appParser := appfile.NewApplicationParser(r.Client, r.dm, r.pd)
handler, err := NewAppHandler(logCtx, r, app, appParser)
if err != nil {
@@ -238,6 +245,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu
if status := app.Status.Workflow; status != nil && status.Terminated {
return r.result(nil).ret()
}
case common.WorkflowStateSkipping:
logCtx.Info("Skip this reconcile")
return ctrl.Result{}, nil
}
var phase = common.ApplicationRunning
@@ -286,7 +296,7 @@ func (r *Reconciler) gcResourceTrackers(logCtx monitorContext.Context, handler *
return r.endWithNegativeCondition(logCtx, handler.app, condition.ReconcileError(err), phase)
}
if !finished {
logCtx.Info("GarbageCollecting resourcetrackers")
logCtx.Info("GarbageCollecting resourcetrackers unfinished")
cond := condition.Deleting()
if len(waiting) > 0 {
cond.Message = fmt.Sprintf("Waiting for %s to delete. (At least %d resources are deleting.)", waiting[0].DisplayName(), len(waiting))
@@ -382,13 +392,31 @@ func (r *Reconciler) endWithNegativeCondition(ctx context.Context, app *v1beta1.
func (r *Reconciler) patchStatus(ctx context.Context, app *v1beta1.Application, phase common.ApplicationPhase) error {
app.Status.Phase = phase
updateObservedGeneration(app)
return r.Status().Patch(ctx, app, client.Merge)
if err := r.Status().Patch(ctx, app, client.Merge); err != nil {
// set to -1 to re-run workflow if status is failed to patch
workflow.StepStatusCache.Store(fmt.Sprintf("%s-%s", app.Name, app.Namespace), -1)
return err
}
return nil
}
func (r *Reconciler) updateStatus(ctx context.Context, app *v1beta1.Application, phase common.ApplicationPhase) error {
app.Status.Phase = phase
updateObservedGeneration(app)
return r.Status().Update(ctx, app)
if !r.disableStatusUpdate {
return r.Status().Update(ctx, app)
}
obj, err := app.Unstructured()
if err != nil {
return err
}
if err := r.Status().Update(ctx, obj); err != nil {
// set to -1 to re-run workflow if status is failed to update
workflow.StepStatusCache.Store(fmt.Sprintf("%s-%s", app.Name, app.Namespace), -1)
return err
}
return nil
}
func (r *Reconciler) doWorkflowFinish(app *v1beta1.Application, wf workflow.Workflow) error {
@@ -501,13 +529,12 @@ func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error {
// Setup adds a controller that reconciles AppRollout.
func Setup(mgr ctrl.Manager, args core.Args) error {
reconciler := Reconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Recorder: event.NewAPIRecorder(mgr.GetEventRecorderFor("Application")),
dm: args.DiscoveryMapper,
pd: args.PackageDiscover,
appRevisionLimit: args.AppRevisionLimit,
concurrentReconciles: args.ConcurrentReconciles,
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Recorder: event.NewAPIRecorder(mgr.GetEventRecorderFor("Application")),
dm: args.DiscoveryMapper,
pd: args.PackageDiscover,
options: parseOptions(args),
}
return reconciler.SetupWithManager(mgr)
}
@@ -557,3 +584,11 @@ func timeReconcile(app *v1beta1.Application) func() {
metrics.ApplicationReconcileTimeHistogram.WithLabelValues(beginPhase, string(app.Status.Phase)).Observe(v)
}
}
func parseOptions(args core.Args) options {
return options{
disableStatusUpdate: args.EnableCompatibility,
appRevisionLimit: args.AppRevisionLimit,
concurrentReconciles: args.ConcurrentReconciles,
}
}

View File

@@ -2275,7 +2275,51 @@ var _ = Describe("Test Application Controller", func() {
Namespace: app.Namespace,
}, checkWeb)).Should(BeNil())
Expect(*(checkWeb.Spec.Replicas)).Should(BeEquivalentTo(int32(0)))
})
It("app apply resource in parallel", func() {
wfDef := &v1beta1.WorkflowStepDefinition{}
wfDefJson, _ := yaml.YAMLToJSON([]byte(applyInParallelWorkflowDefinitionYaml))
Expect(json.Unmarshal(wfDefJson, wfDef)).Should(BeNil())
Expect(k8sClient.Create(ctx, wfDef.DeepCopy())).Should(SatisfyAny(BeNil(), &util.AlreadyExistMatcher{}))
ns := &corev1.Namespace{
ObjectMeta: metav1.ObjectMeta{
Name: "vela-test-apply-in-parallel",
},
}
app := appwithNoTrait.DeepCopy()
app.Name = "vela-test-app"
app.SetNamespace(ns.Name)
app.Spec.Workflow = &v1beta1.Workflow{
Steps: []v1beta1.WorkflowStep{{
Name: "apply-in-parallel",
Type: "apply-test",
Properties: &runtime.RawExtension{Raw: []byte(`{"parallelism": 20}`)},
}},
}
Expect(k8sClient.Create(ctx, ns)).Should(BeNil())
Expect(k8sClient.Create(ctx, app)).Should(BeNil())
appKey := client.ObjectKey{
Name: app.Name,
Namespace: app.Namespace,
}
_, err := testutil.ReconcileOnceAfterFinalizer(reconciler, reconcile.Request{NamespacedName: appKey})
Expect(err).Should(BeNil())
deployList := new(v1.DeploymentList)
Expect(k8sClient.List(ctx, deployList, client.InNamespace(app.Namespace))).Should(BeNil())
Expect(len(deployList.Items)).Should(Equal(20))
checkApp := new(v1beta1.Application)
Expect(k8sClient.Get(ctx, appKey, checkApp)).Should(Succeed())
rt := new(v1beta1.ResourceTracker)
expectRTName := fmt.Sprintf("%s-%s", checkApp.Status.LatestRevision.Name, checkApp.GetNamespace())
Eventually(func() error {
return k8sClient.Get(ctx, client.ObjectKey{Name: expectRTName}, rt)
}, 10*time.Second, 500*time.Millisecond).Should(Succeed())
Expect(len(rt.Spec.ManagedResources)).Should(Equal(20))
})
})
@@ -3184,6 +3228,46 @@ spec:
}
}
parameter: objects: [...{}]
`
applyInParallelWorkflowDefinitionYaml = `
apiVersion: core.oam.dev/v1beta1
kind: WorkflowStepDefinition
metadata:
name: apply-test
namespace: vela-system
spec:
schematic:
cue:
template: |
import (
"vela/op"
"list"
)
components: op.#LoadInOrder & {}
targetComponent: components.value[0]
resources: op.#RenderComponent & {
value: targetComponent
}
workload: resources.output
arr: list.Range(0, parameter.parallelism, 1)
patchWorkloads: op.#Steps & {
for idx in arr {
"\(idx)": op.#PatchK8sObject & {
value: workload
patch: {
// +patchStrategy=retainKeys
metadata: name: "\(targetComponent.name)-\(idx)"
}
}
}
}
workloads: [ for patchResult in patchWorkloads {patchResult.result}]
apply: op.#ApplyInParallel & {
value: workloads
}
parameter: parallelism: int
`
)

View File

@@ -311,7 +311,7 @@ func ComputeAppRevisionHash(appRevision *v1beta1.ApplicationRevision) (string, e
}
appRevisionHash.ComponentDefinitionHash[key] = hash
}
for key, td := range appRevision.Spec.TraitDefinitions {
for key, td := range filterSkipAffectAppRevTraitDefinitions(appRevision.Spec.TraitDefinitions) {
hash, err := utils.ComputeSpecHash(&td.Spec)
if err != nil {
return "", err
@@ -374,9 +374,19 @@ func (h *AppHandler) currentAppRevIsNew(ctx context.Context) (bool, bool, error)
return true, true, nil
}
isLatestRev := deepEqualAppInRevision(h.latestAppRev, h.currentAppRev)
if metav1.HasAnnotation(h.app.ObjectMeta, oam.AnnotationAutoUpdate) {
isLatestRev = h.app.Status.LatestRevision.RevisionHash == h.currentRevHash && DeepEqualRevision(h.latestAppRev, h.currentAppRev)
}
// diff the latest revision first
if h.app.Status.LatestRevision.RevisionHash == h.currentRevHash && DeepEqualRevision(h.latestAppRev, h.currentAppRev) {
if isLatestRev {
appSpec := h.currentAppRev.Spec.Application.Spec
traitDef := h.currentAppRev.Spec.TraitDefinitions
h.currentAppRev = h.latestAppRev.DeepCopy()
h.currentRevHash = h.app.Status.LatestRevision.RevisionHash
h.currentAppRev.Spec.Application.Spec = appSpec
h.currentAppRev.Spec.TraitDefinitions = traitDef
return false, false, nil
}
@@ -409,7 +419,9 @@ func DeepEqualRevision(old, new *v1beta1.ApplicationRevision) bool {
if len(old.Spec.WorkloadDefinitions) != len(new.Spec.WorkloadDefinitions) {
return false
}
if len(old.Spec.TraitDefinitions) != len(new.Spec.TraitDefinitions) {
oldTraitDefinitions := filterSkipAffectAppRevTraitDefinitions(old.Spec.TraitDefinitions)
newTraitDefinitions := filterSkipAffectAppRevTraitDefinitions(new.Spec.TraitDefinitions)
if len(oldTraitDefinitions) != len(newTraitDefinitions) {
return false
}
if len(old.Spec.ComponentDefinitions) != len(new.Spec.ComponentDefinitions) {
@@ -428,8 +440,8 @@ func DeepEqualRevision(old, new *v1beta1.ApplicationRevision) bool {
return false
}
}
for key, td := range new.Spec.TraitDefinitions {
if !apiequality.Semantic.DeepEqual(old.Spec.TraitDefinitions[key].Spec, td.Spec) {
for key, td := range newTraitDefinitions {
if !apiequality.Semantic.DeepEqual(oldTraitDefinitions[key].Spec, td.Spec) {
return false
}
}
@@ -438,6 +450,10 @@ func DeepEqualRevision(old, new *v1beta1.ApplicationRevision) bool {
return false
}
}
return deepEqualAppInRevision(old, new)
}
func deepEqualAppInRevision(old, new *v1beta1.ApplicationRevision) bool {
return apiequality.Semantic.DeepEqual(filterSkipAffectAppRevTrait(old.Spec.Application.Spec, old.Spec.TraitDefinitions),
filterSkipAffectAppRevTrait(new.Spec.Application.Spec, new.Spec.TraitDefinitions))
}
@@ -820,6 +836,17 @@ func filterSkipAffectAppRevTrait(appSpec v1beta1.ApplicationSpec, tds map[string
return *res
}
// before computing hash or deepEqual, filterSkipAffectAppRevTraitDefinitions filter can ignore `SkipAffectRevision` trait definition from appRev
func filterSkipAffectAppRevTraitDefinitions(tds map[string]v1beta1.TraitDefinition) map[string]v1beta1.TraitDefinition {
res := make(map[string]v1beta1.TraitDefinition)
for key, td := range tds {
if !td.Spec.SkipRevisionAffect {
res[key] = td
}
}
return res
}
type historiesByRevision []v1beta1.ApplicationRevision
func (h historiesByRevision) Len() int { return len(h) }
@@ -837,7 +864,7 @@ func cleanUpWorkflowComponentRevision(ctx context.Context, h *AppHandler) error
}
// collect component revision in use
compRevisionInUse := map[string]map[string]struct{}{}
for _, resource := range h.app.Status.AppliedResources {
for i, resource := range h.app.Status.AppliedResources {
compName := resource.Name
ns := resource.Namespace
r := &unstructured.Unstructured{}
@@ -846,7 +873,7 @@ func cleanUpWorkflowComponentRevision(ctx context.Context, h *AppHandler) error
err := h.r.Get(_ctx, ktypes.NamespacedName{Name: compName, Namespace: ns}, r)
notFound := apierrors.IsNotFound(err)
if err != nil && !notFound {
return err
return errors.WithMessagef(err, "get applied resource index=%d", i)
}
if compRevisionInUse[compName] == nil {
compRevisionInUse[compName] = map[string]struct{}{}

View File

@@ -151,7 +151,6 @@ var _ = Describe("test generate revision ", func() {
appRevision1.Spec.ComponentDefinitions[cd.Name] = cd
appRevision1.Spec.WorkloadDefinitions[wd.Name] = wd
appRevision1.Spec.TraitDefinitions[td.Name] = td
appRevision1.Spec.TraitDefinitions[rolloutTd.Name] = rolloutTd
appRevision1.Spec.ScopeDefinitions[sd.Name] = sd
appRevision2 = *appRevision1.DeepCopy()
@@ -168,6 +167,10 @@ var _ = Describe("test generate revision ", func() {
Expect(k8sClient.Delete(context.TODO(), &ns)).Should(Succeed())
})
verifyDeepEqualRevision := func() {
Expect(DeepEqualRevision(&appRevision1, &appRevision2)).Should(BeTrue())
}
verifyEqual := func() {
appHash1, err := ComputeAppRevisionHash(&appRevision1)
Expect(err).Should(Succeed())
@@ -228,7 +231,7 @@ var _ = Describe("test generate revision ", func() {
verifyNotEqual()
})
It("Test appliction contain a SkipAppRevision tait will have same hash", func() {
It("Test appliction contain a SkipAppRevision tait will have same hash and revision will equal", func() {
rolloutTrait := common.ApplicationTrait{
Type: "rollout",
Properties: &runtime.RawExtension{
@@ -236,7 +239,32 @@ var _ = Describe("test generate revision ", func() {
},
}
appRevision2.Spec.Application.Spec.Components[0].Traits = append(appRevision2.Spec.Application.Spec.Components[0].Traits, rolloutTrait)
// appRevision will have no traitDefinition of rollout
appRevision2.Spec.TraitDefinitions[rolloutTd.Name] = rolloutTd
verifyEqual()
verifyDeepEqualRevision()
})
It("Test application revision compare", func() {
By("Apply the application")
appParser := appfile.NewApplicationParser(reconciler.Client, reconciler.dm, reconciler.pd)
ctx = util.SetNamespaceInCtx(ctx, app.Namespace)
generatedAppfile, err := appParser.GenerateAppFile(ctx, &app)
Expect(err).Should(Succeed())
comps, err = generatedAppfile.GenerateComponentManifests()
Expect(err).Should(Succeed())
Expect(handler.PrepareCurrentAppRevision(ctx, generatedAppfile)).Should(Succeed())
Expect(handler.FinalizeAndApplyAppRevision(ctx)).Should(Succeed())
prevHash := generatedAppfile.AppRevisionHash
handler.app.Status.LatestRevision = &common.Revision{Name: generatedAppfile.AppRevisionName, Revision: 1, RevisionHash: generatedAppfile.AppRevisionHash}
generatedAppfile.Workloads[0].FullTemplate.ComponentDefinition = nil
Expect(handler.PrepareCurrentAppRevision(ctx, generatedAppfile)).Should(Succeed())
nonChangeHash := generatedAppfile.AppRevisionHash
handler.app.Annotations = map[string]string{oam.AnnotationAutoUpdate: "true"}
Expect(handler.PrepareCurrentAppRevision(ctx, generatedAppfile)).Should(Succeed())
changedHash := generatedAppfile.AppRevisionHash
Expect(nonChangeHash).Should(Equal(prevHash))
Expect(changedHash).ShouldNot(Equal(prevHash))
})
It("Test apply success for none rollout case", func() {

View File

@@ -137,13 +137,14 @@ var _ = BeforeSuite(func(done Done) {
appParser = appfile.NewApplicationParser(k8sClient, dm, pd)
reconciler = &Reconciler{
Client: k8sClient,
Scheme: testScheme,
dm: dm,
pd: pd,
Recorder: event.NewAPIRecorder(recorder),
appRevisionLimit: appRevisionLimit,
Client: k8sClient,
Scheme: testScheme,
dm: dm,
pd: pd,
Recorder: event.NewAPIRecorder(recorder),
}
reconciler.appRevisionLimit = appRevisionLimit
// setup the controller manager since we need the component handler to run in the background
mgr, err = ctrl.NewManager(cfg, ctrl.Options{
Scheme: testScheme,

View File

@@ -61,11 +61,13 @@ const (
TerraformVariableMap string = "map"
TerraformVariableObject string = "object"
TerraformVariableNull string = ""
TerraformVariableAny string = "any"
TerraformListTypePrefix string = "list("
TerraformTupleTypePrefix string = "tuple("
TerraformMapTypePrefix string = "map("
TerraformObjectTypePrefix string = "object("
TerraformSetTypePrefix string = "set("
typeTraitDefinition = "trait"
typeComponentDefinition = "component"
@@ -157,17 +159,38 @@ func GetOpenAPISchemaFromTerraformComponentDefinition(configuration string) ([]b
schema = openapi3.NewArraySchema()
case TerraformVariableMap, TerraformVariableObject:
schema = openapi3.NewObjectSchema()
case TerraformVariableAny:
switch v.Default.(type) {
case []interface{}:
schema = openapi3.NewArraySchema()
case map[string]interface{}:
schema = openapi3.NewObjectSchema()
}
case TerraformVariableNull:
return nil, fmt.Errorf("null type variable is NOT supported, please specify a type for the variable: %s", v.Name)
switch v.Default.(type) {
case nil, string:
schema = openapi3.NewStringSchema()
case []interface{}:
schema = openapi3.NewArraySchema()
case map[string]interface{}:
schema = openapi3.NewObjectSchema()
case int, float64:
schema = openapi3.NewFloat64Schema()
default:
return nil, fmt.Errorf("null type variable is NOT supported, please specify a type for the variable: %s", v.Name)
}
}
// To identify unusual list type
if schema == nil {
switch {
case strings.HasPrefix(v.Type, TerraformListTypePrefix) || strings.HasPrefix(v.Type, TerraformTupleTypePrefix):
case strings.HasPrefix(v.Type, TerraformListTypePrefix) || strings.HasPrefix(v.Type, TerraformTupleTypePrefix) ||
strings.HasPrefix(v.Type, TerraformSetTypePrefix):
schema = openapi3.NewArraySchema()
case strings.HasPrefix(v.Type, TerraformMapTypePrefix) || strings.HasPrefix(v.Type, TerraformObjectTypePrefix):
schema = openapi3.NewObjectSchema()
default:
return nil, fmt.Errorf("the type `%s` of variable %s is NOT supported", v.Type, v.Name)
}
}
schema.Title = k
@@ -205,12 +228,17 @@ func GetTerraformConfigurationFromRemote(name, remoteURL, remotePath string) (st
return "", err
}
tfPath := filepath.Join(tmpPath, remotePath, "main.tf")
tfPath := filepath.Join(tmpPath, remotePath, "variables.tf")
if _, err := os.Stat(tfPath); err != nil {
tfPath = filepath.Join(tmpPath, remotePath, "main.tf")
if _, err := os.Stat(tfPath); err != nil {
return "", errors.Wrap(err, "failed to find main.tf or variables.tf in Terraform configurations of the remote repository")
}
}
conf, err := ioutil.ReadFile(filepath.Clean(tfPath))
if err != nil {
return "", err
return "", errors.Wrap(err, "failed to read Terraform configuration")
}
if err := os.RemoveAll(tmpPath); err != nil {
return "", err
}

View File

@@ -20,6 +20,7 @@ package utils
import (
"context"
"errors"
"fmt"
"io/ioutil"
"os"
"path/filepath"
@@ -326,8 +327,38 @@ variable "name" {
default = "abc"
}`,
want: want{
subStr: "",
err: errors.New("null type variable is NOT supported, please specify a type for the variable: name"),
subStr: "abc",
err: nil,
},
},
"null type variable, while default value is a slice": {
configuration: `
variable "name" {
default = [123]
}`,
want: want{
subStr: "123",
err: nil,
},
},
"null type variable, while default value is a map": {
configuration: `
variable "name" {
default = {a = 1}
}`,
want: want{
subStr: "a",
err: nil,
},
},
"null type variable, while default value is number": {
configuration: `
variable "name" {
default = 123
}`,
want: want{
subStr: "123",
err: nil,
},
},
"complicated list variable": {
@@ -354,6 +385,38 @@ variable "bbb" {
config = string
})
default = []
}`,
want: want{
subStr: "bbb",
err: nil,
},
},
"not supported complicated variable": {
configuration: `
variable "bbb" {
type = xxxxx(string)
}`,
want: want{
subStr: "",
err: fmt.Errorf("the type `%s` of variable %s is NOT supported", "xxxxx(string)", "bbb"),
},
},
"any type, slice default": {
configuration: `
variable "bbb" {
type = any
default = []
}`,
want: want{
subStr: "bbb",
err: nil,
},
},
"any type, map default": {
configuration: `
variable "bbb" {
type = any
default = {}
}`,
want: want{
subStr: "bbb",
@@ -382,20 +445,28 @@ func TestGetTerraformConfigurationFromRemote(t *testing.T) {
// panic: permission denied
type want struct {
config string
err error
errMsg string
}
type args struct {
name string
url string
path string
data []byte
variableFile string
// mockWorkingPath will create `/tmp/terraform`
mockWorkingPath bool
}
cases := map[string]struct {
name string
url string
path string
data []byte
args args
want want
}{
"valid": {
name: "valid",
url: "https://github.com/kubevela-contrib/terraform-modules.git",
path: "",
data: []byte(`
args: args{
name: "valid",
url: "https://github.com/kubevela-contrib/terraform-modules.git",
path: "",
data: []byte(`
variable "aaa" {
type = list(object({
type = string
@@ -404,6 +475,8 @@ variable "aaa" {
}))
default = []
}`),
variableFile: "main.tf",
},
want: want{
config: `
variable "aaa" {
@@ -414,28 +487,84 @@ variable "aaa" {
}))
default = []
}`,
err: nil,
},
},
"configuration is remote with path": {
args: args{
name: "aws-subnet",
url: "https://github.com/kubevela-contrib/terraform-modules.git",
path: "aws/subnet",
data: []byte(`
variable "aaa" {
type = list(object({
type = string
sourceArn = string
config = string
}))
default = []
}`),
variableFile: "variables.tf",
},
want: want{
config: `
variable "aaa" {
type = list(object({
type = string
sourceArn = string
config = string
}))
default = []
}`,
},
},
"working path exists": {
args: args{
variableFile: "main.tf",
mockWorkingPath: true,
},
want: want{
errMsg: "failed to remove the directory",
},
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
if tc.args.mockWorkingPath {
err := os.MkdirAll("./tmp/terraform", 0755)
assert.NilError(t, err)
defer os.RemoveAll("./tmp/terraform")
patch1 := ApplyFunc(os.Remove, func(_ string) error {
return errors.New("failed")
})
defer patch1.Reset()
patch2 := ApplyFunc(os.Open, func(_ string) (*os.File, error) {
return nil, errors.New("failed")
})
defer patch2.Reset()
}
patch := ApplyFunc(git.PlainCloneContext, func(ctx context.Context, path string, isBare bool, o *git.CloneOptions) (*git.Repository, error) {
tmpPath := filepath.Join("./tmp/terraform", tc.name)
var tmpPath string
if tc.args.path != "" {
tmpPath = filepath.Join("./tmp/terraform", tc.args.name, tc.args.path)
} else {
tmpPath = filepath.Join("./tmp/terraform", tc.args.name)
}
err := os.MkdirAll(tmpPath, os.ModePerm)
assert.NilError(t, err)
err = ioutil.WriteFile(filepath.Clean(filepath.Join(tmpPath, "main.tf")), tc.data, 0644)
err = ioutil.WriteFile(filepath.Clean(filepath.Join(tmpPath, tc.args.variableFile)), tc.args.data, 0644)
assert.NilError(t, err)
return nil, nil
})
defer patch.Reset()
conf, err := GetTerraformConfigurationFromRemote(tc.name, tc.url, tc.path)
if diff := cmp.Diff(tc.want.err, err, test.EquateErrors()); diff != "" {
t.Errorf("\n%s\nGetTerraformConfigurationFromRemote(...): -want error, +got error:\n%s", name, diff)
}
if tc.want.err == nil {
conf, err := GetTerraformConfigurationFromRemote(tc.args.name, tc.args.url, tc.args.path)
if tc.want.errMsg != "" {
if err != nil && !strings.Contains(err.Error(), tc.want.errMsg) {
t.Errorf("\n%s\nGetTerraformConfigurationFromRemote(...): -want error %v, +got error:%s", name, err, tc.want.errMsg)
}
} else {
assert.Equal(t, tc.want.config, conf)
}
})

View File

@@ -22,17 +22,34 @@ import (
"strings"
"cuelang.org/go/cue"
"cuelang.org/go/cue/build"
"github.com/oam-dev/kubevela/apis/types"
"github.com/oam-dev/kubevela/pkg/cue/model"
"github.com/oam-dev/kubevela/pkg/cue/packages"
)
// GetParameters get parameter from cue template
func GetParameters(templateStr string) ([]types.Parameter, error) {
r := cue.Runtime{}
template, err := r.Compile("", templateStr+BaseTemplate)
if err != nil {
return nil, err
func GetParameters(templateStr string, pd *packages.PackageDiscover) ([]types.Parameter, error) {
var template *cue.Instance
var err error
if pd != nil {
bi := build.NewContext().NewInstance("", nil)
err := bi.AddFile("-", templateStr+BaseTemplate)
if err != nil {
return nil, err
}
template, err = pd.ImportPackagesAndBuildInstance(bi)
if err != nil {
return nil, err
}
} else {
r := cue.Runtime{}
template, err = r.Compile("", templateStr+BaseTemplate)
if err != nil {
return nil, err
}
}
tempStruct, err := template.Value().Struct()
if err != nil {

View File

@@ -28,7 +28,7 @@ import (
func TestGetParameter(t *testing.T) {
data, _ := os.ReadFile("testdata/workloads/metrics.cue")
params, err := GetParameters(string(data))
params, err := GetParameters(string(data), nil)
assert.NoError(t, err)
assert.Equal(t, params, []types.Parameter{
{Name: "format", Required: false, Default: "prometheus", Usage: "format of the metrics, " +
@@ -38,7 +38,7 @@ func TestGetParameter(t *testing.T) {
{Name: "selector", Required: false, Usage: "the label selector for the pods, default is the workload labels", Type: cue.StructKind},
})
data, _ = os.ReadFile("testdata/workloads/deployment.cue")
params, err = GetParameters(string(data))
params, err = GetParameters(string(data), nil)
assert.NoError(t, err)
assert.Equal(t, []types.Parameter{
{Name: "name", Required: true, Default: "", Type: cue.StringKind},
@@ -50,7 +50,7 @@ func TestGetParameter(t *testing.T) {
params)
data, _ = os.ReadFile("testdata/workloads/test-param.cue")
params, err = GetParameters(string(data))
params, err = GetParameters(string(data), nil)
assert.NoError(t, err)
assert.Equal(t, []types.Parameter{
{Name: "name", Required: true, Default: "", Type: cue.StringKind},
@@ -61,13 +61,13 @@ func TestGetParameter(t *testing.T) {
{Name: "fval", Default: 64.3, Type: cue.FloatKind},
{Name: "nval", Default: float64(0), Required: true, Type: cue.NumberKind}}, params)
data, _ = os.ReadFile("testdata/workloads/empty.cue")
params, err = GetParameters(string(data))
params, err = GetParameters(string(data), nil)
assert.NoError(t, err)
var exp []types.Parameter
assert.Equal(t, exp, params)
data, _ = os.ReadFile("testdata/workloads/webservice.cue") // test cue parameter with "// +ignore" annotation
params, err = GetParameters(string(data)) // Only test for func RetrieveComments
params, err = GetParameters(string(data), nil) // Only test for func RetrieveComments
assert.NoError(t, err)
var flag bool
for _, para := range params {

View File

@@ -41,6 +41,8 @@ const (
ContextCompRevisionName = "revision"
// ContextComponents is the components of app
ContextComponents = "components"
// ContextComponentType is the component type of current trait binding with
ContextComponentType = "componentType"
// ComponentRevisionPlaceHolder is the component revision name placeHolder, this field will be replace with real value
// after component be created
ComponentRevisionPlaceHolder = "KUBEVELA_COMPONENT_REVISION_PLACEHOLDER"

View File

@@ -107,18 +107,17 @@ func (pd *PackageDiscover) ImportBuiltinPackagesFor(bi *build.Instance) {
// ImportPackagesAndBuildInstance Combine import built-in packages and build cue template together to avoid data race
func (pd *PackageDiscover) ImportPackagesAndBuildInstance(bi *build.Instance) (inst *cue.Instance, err error) {
var r cue.Runtime
if pd == nil {
return r.Build(bi)
}
pd.ImportBuiltinPackagesFor(bi)
if err := stdlib.AddImportsFor(bi, ""); err != nil {
return nil, err
}
var r cue.Runtime
pd.mutex.Lock()
defer pd.mutex.Unlock()
cueInst, err := r.Build(bi)
if err != nil {
return nil, err
}
return cueInst, err
return r.Build(bi)
}
// ListPackageKinds list packages and their kinds

View File

@@ -18,7 +18,6 @@ package metrics
import (
"github.com/prometheus/client_golang/prometheus"
"sigs.k8s.io/controller-runtime/pkg/client"
)
var (
@@ -116,17 +115,5 @@ var (
ResourceTrackerNumberGauge = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "resourcetracker_number",
Help: "resourceTracker number.",
}, []string{"application", "namespace"})
}, []string{"controller"})
)
// ExtractMetricValuesFromObjectLabel extract metric values from k8s object's labels
func ExtractMetricValuesFromObjectLabel(obj interface{}, labelKeys ...string) (values []string) {
if resource, ok := obj.(client.Object); ok {
for _, labelKey := range labelKeys {
values = append(values, resource.GetLabels()[labelKey])
}
} else {
values = make([]string, len(labelKeys))
}
return
}

View File

@@ -165,6 +165,9 @@ const (
// AnnotationPublishVersion is annotation that record the application workflow version.
AnnotationPublishVersion = "app.oam.dev/publishVersion"
// AnnotationAutoUpdate is annotation that let application auto update when it finds definition changes
AnnotationAutoUpdate = "app.oam.dev/autoUpdate"
// AnnotationWorkflowName specifies the workflow name for execution.
AnnotationWorkflowName = "app.oam.dev/workflowName"

View File

@@ -146,8 +146,17 @@ func PatchApplication(base *v1beta1.Application, patch *v1alpha1.EnvPatch, selec
var errs errors2.ErrorList
var err error
for _, comp := range patch.Components {
if baseComp, exists := compMaps[comp.Name]; exists {
if baseComp.Type != comp.Type {
if comp.Name == "" {
for compName, baseComp := range compMaps {
if comp.Type == "" || comp.Type == baseComp.Type {
compMaps[compName], err = MergeComponent(baseComp, comp.DeepCopy())
if err != nil {
errs = append(errs, errors.Wrapf(err, "failed to merge component %s", compName))
}
}
}
} else if baseComp, exists := compMaps[comp.Name]; exists {
if baseComp.Type != comp.Type && comp.Type != "" {
compMaps[comp.Name] = comp.ToApplicationComponent()
} else {
compMaps[comp.Name], err = MergeComponent(baseComp, comp.DeepCopy())

View File

@@ -294,6 +294,240 @@ func Test_EnvBindApp_GenerateConfiguredApplication(t *testing.T) {
},
},
},
"patch-all-comp": {
baseApp: &v1beta1.Application{
Spec: v1beta1.ApplicationSpec{
Components: []common.ApplicationComponent{{
Name: "express-server",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a.example.com",
}),
}, {
Type: "ingress-1-20",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a-1-20.example.com",
}),
}},
}, {
Name: "express-worker",
Type: "worker",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
}},
},
},
envName: "prod",
envPatch: v1alpha1.EnvPatch{
Components: []v1alpha1.EnvComponentPatch{{
Name: "",
Type: "",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []v1alpha1.EnvTraitPatch{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}},
}},
},
expectedApp: &v1beta1.Application{
Spec: v1beta1.ApplicationSpec{
Components: []common.ApplicationComponent{{
Name: "express-server",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}, {
Type: "ingress-1-20",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a-1-20.example.com",
}),
}},
}, {
Name: "express-worker",
Type: "worker",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}},
}},
},
},
},
"patch-type-comp": {
baseApp: &v1beta1.Application{
Spec: v1beta1.ApplicationSpec{
Components: []common.ApplicationComponent{{
Name: "express-server",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a.example.com",
}),
}, {
Type: "ingress-1-20",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a-1-20.example.com",
}),
}},
}, {
Name: "express-worker",
Type: "worker",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
}},
},
},
envName: "prod",
envPatch: v1alpha1.EnvPatch{
Components: []v1alpha1.EnvComponentPatch{{
Name: "",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []v1alpha1.EnvTraitPatch{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}},
}},
},
expectedApp: &v1beta1.Application{
Spec: v1beta1.ApplicationSpec{
Components: []common.ApplicationComponent{{
Name: "express-server",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}, {
Type: "ingress-1-20",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a-1-20.example.com",
}),
}},
}, {
Name: "express-worker",
Type: "worker",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
}},
},
},
},
"patch-without-type-specified": {
baseApp: &v1beta1.Application{
Spec: v1beta1.ApplicationSpec{
Components: []common.ApplicationComponent{{
Name: "express-server",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a.example.com",
}),
}, {
Type: "ingress-1-20",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a-1-20.example.com",
}),
}},
}, {
Name: "express-worker",
Type: "worker",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
}},
},
},
envName: "prod",
envPatch: v1alpha1.EnvPatch{
Components: []v1alpha1.EnvComponentPatch{{
Name: "express-worker",
Type: "",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []v1alpha1.EnvTraitPatch{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}},
}},
},
expectedApp: &v1beta1.Application{
Spec: v1beta1.ApplicationSpec{
Components: []common.ApplicationComponent{{
Name: "express-server",
Type: "webservice",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "crccheck/hello-world",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a.example.com",
}),
}, {
Type: "ingress-1-20",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "a-1-20.example.com",
}),
}},
}, {
Name: "express-worker",
Type: "worker",
Properties: util.Object2RawExtension(map[string]interface{}{
"image": "busybox",
}),
Traits: []common.ApplicationTrait{{
Type: "ingress",
Properties: util.Object2RawExtension(map[string]interface{}{
"domain": "b.example.com",
}),
}},
}},
},
},
},
}
for name, tc := range testCases {

View File

@@ -39,7 +39,7 @@ func (h *resourceKeeper) DispatchComponentRevision(ctx context.Context, cr *v1.C
obj.SetName(cr.Name)
obj.SetNamespace(cr.Namespace)
obj.SetLabels(cr.Labels)
if err = resourcetracker.RecordManifestInResourceTracker(multicluster.ContextInLocalCluster(ctx), h.Client, rt, obj, true); err != nil {
if err = resourcetracker.RecordManifestsInResourceTracker(multicluster.ContextInLocalCluster(ctx), h.Client, rt, []*unstructured.Unstructured{obj}, true); err != nil {
return errors.Wrapf(err, "failed to record componentrevision %s/%s/%s", oam.GetCluster(cr), cr.Namespace, cr.Name)
}
if err = h.Client.Create(multicluster.ContextWithClusterName(ctx, oam.GetCluster(cr)), cr); err != nil {

View File

@@ -18,17 +18,21 @@ package resourcekeeper
import (
"context"
"sync"
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
kerrors "k8s.io/apimachinery/pkg/util/errors"
"github.com/oam-dev/kubevela/apis/core.oam.dev/v1beta1"
"github.com/oam-dev/kubevela/pkg/multicluster"
"github.com/oam-dev/kubevela/pkg/oam"
"github.com/oam-dev/kubevela/pkg/resourcetracker"
"github.com/oam-dev/kubevela/pkg/utils/apply"
)
// MaxDispatchConcurrent is the max dispatch concurrent number
var MaxDispatchConcurrent = 10
// DispatchOption option for dispatch
type DispatchOption interface {
ApplyToDispatchConfig(*dispatchConfig)
@@ -52,6 +56,21 @@ func (h *resourceKeeper) Dispatch(ctx context.Context, manifests []*unstructured
if h.applyOncePolicy != nil && h.applyOncePolicy.Enable {
options = append(options, MetaOnlyOption{})
}
// 1. record manifests in resourcetracker
if err = h.record(ctx, manifests, options...); err != nil {
return err
}
// 2. apply manifests
if err = h.dispatch(ctx, manifests); err != nil {
return err
}
return nil
}
func (h *resourceKeeper) record(ctx context.Context, manifests []*unstructured.Unstructured, options ...DispatchOption) error {
var rootManifests []*unstructured.Unstructured
var versionManifests []*unstructured.Unstructured
for _, manifest := range manifests {
if manifest != nil {
_options := options
@@ -61,34 +80,61 @@ func (h *resourceKeeper) Dispatch(ctx context.Context, manifests []*unstructured
}
}
cfg := newDispatchConfig(_options...)
if err = h.dispatch(ctx, manifest, cfg); err != nil {
return err
if !cfg.skipRT {
if cfg.useRoot {
rootManifests = append(rootManifests, manifest)
} else {
versionManifests = append(versionManifests, manifest)
}
}
}
}
cfg := newDispatchConfig(options...)
if len(rootManifests) != 0 {
rt, err := h.getRootRT(ctx)
if err != nil {
return errors.Wrapf(err, "failed to get resourcetracker")
}
if err = resourcetracker.RecordManifestsInResourceTracker(multicluster.ContextInLocalCluster(ctx), h.Client, rt, rootManifests, cfg.metaOnly); err != nil {
return errors.Wrapf(err, "failed to record resources in resourcetracker %s", rt.Name)
}
}
rt, err := h.getCurrentRT(ctx)
if err != nil {
return errors.Wrapf(err, "failed to get resourcetracker")
}
if err = resourcetracker.RecordManifestsInResourceTracker(multicluster.ContextInLocalCluster(ctx), h.Client, rt, versionManifests, cfg.metaOnly); err != nil {
return errors.Wrapf(err, "failed to record resources in resourcetracker %s", rt.Name)
}
return nil
}
func (h *resourceKeeper) dispatch(ctx context.Context, manifest *unstructured.Unstructured, cfg *dispatchConfig) (err error) {
// 1. record manifests in resourcetracker
if !cfg.skipRT {
var rt *v1beta1.ResourceTracker
if cfg.useRoot {
rt, err = h.getRootRT(ctx)
} else {
rt, err = h.getCurrentRT(ctx)
}
if err != nil {
return errors.Wrapf(err, "failed to get resourcetracker")
}
if err = resourcetracker.RecordManifestInResourceTracker(multicluster.ContextInLocalCluster(ctx), h.Client, rt, manifest, cfg.metaOnly); err != nil {
return errors.Wrapf(err, "failed to record resources in resourcetracker %s", rt.Name)
}
}
// 2. apply manifests
func (h *resourceKeeper) dispatch(ctx context.Context, manifests []*unstructured.Unstructured) error {
var errs []error
var l sync.Mutex
var wg sync.WaitGroup
ch := make(chan struct{}, MaxDispatchConcurrent)
applyOpts := []apply.ApplyOption{apply.MustBeControlledByApp(h.app), apply.NotUpdateRenderHashEqual()}
if err := h.applicator.Apply(multicluster.ContextWithClusterName(ctx, oam.GetCluster(manifest)), manifest, applyOpts...); err != nil {
return errors.Wrapf(err, "cannot apply manifest, name: %s apiVersion: %s kind: %s", manifest.GetName(), manifest.GetAPIVersion(), manifest.GetKind())
for i := 0; i < len(manifests); i++ {
ch <- struct{}{}
wg.Add(1)
go func(index int) {
defer wg.Done()
manifest := manifests[index]
applyCtx := multicluster.ContextWithClusterName(ctx, oam.GetCluster(manifest))
err := h.applicator.Apply(applyCtx, manifest, applyOpts...)
if err != nil {
l.Lock()
errs = append(errs, err)
l.Unlock()
}
<-ch
}(i)
}
return nil
wg.Wait()
return kerrors.NewAggregate(errs)
}

View File

@@ -101,9 +101,9 @@ func TestResourceKeeperGarbageCollect(t *testing.T) {
obj.SetName(cr.GetName())
obj.SetNamespace(cr.GetNamespace())
obj.SetLabels(cr.GetLabels())
r.NoError(resourcetracker.RecordManifestInResourceTracker(ctx, cli, crRT, obj, true))
r.NoError(resourcetracker.RecordManifestsInResourceTracker(ctx, cli, crRT, []*unstructured.Unstructured{obj}, true))
}
r.NoError(resourcetracker.RecordManifestInResourceTracker(ctx, cli, _rt, cmMaps[i], true))
r.NoError(resourcetracker.RecordManifestsInResourceTracker(ctx, cli, _rt, []*unstructured.Unstructured{cmMaps[i]}, true))
}
checkCount := func(cmCount, rtCount int, crCount int) {

View File

@@ -142,12 +142,15 @@ func ListApplicationResourceTrackers(ctx context.Context, cli client.Client, app
return rootRT, currentRT, historyRTs, crRT, nil
}
// RecordManifestInResourceTracker records resources in ResourceTracker
func RecordManifestInResourceTracker(ctx context.Context, cli client.Client, rt *v1beta1.ResourceTracker, manifest *unstructured.Unstructured, metaOnly bool) error {
if updated := rt.AddManagedResource(manifest, metaOnly); !updated {
return nil
// RecordManifestsInResourceTracker records resources in ResourceTracker
func RecordManifestsInResourceTracker(ctx context.Context, cli client.Client, rt *v1beta1.ResourceTracker, manifests []*unstructured.Unstructured, metaOnly bool) error {
if len(manifests) != 0 {
for _, manifest := range manifests {
rt.AddManagedResource(manifest, metaOnly)
}
return cli.Update(ctx, rt)
}
return cli.Update(ctx, rt)
return nil
}
// DeletedManifestInResourceTracker marks resources as deleted in resourcetracker, if remove is true, resources will be removed from resourcetracker

View File

@@ -97,7 +97,7 @@ func TestRecordAndDeleteManifestsInResourceTracker(t *testing.T) {
obj := &unstructured.Unstructured{}
obj.SetName(fmt.Sprintf("workload-%d", i))
objs = append(objs, obj)
r.NoError(RecordManifestInResourceTracker(context.Background(), cli, rt, obj, rand.Int()%2 == 0))
r.NoError(RecordManifestsInResourceTracker(context.Background(), cli, rt, []*unstructured.Unstructured{obj}, rand.Int()%2 == 0))
}
rand.Shuffle(len(objs), func(i, j int) { objs[i], objs[j] = objs[j], objs[i] })
for i := 0; i < n; i++ {

View File

@@ -17,6 +17,8 @@ import (
#Apply: kube.#Apply
#ApplyInParallel: kube.#ApplyInParallel
#Read: kube.#Read
#List: kube.#List
@@ -156,7 +158,7 @@ import (
#HTTPDelete: http.#Do & {method: "DELETE"}
#ConvertString: convert.#String
#ConvertString: util.#String
#DateToTimestamp: time.#DateToTimestamp
@@ -168,6 +170,8 @@ import (
#LoadInOrder: oam.#LoadComponetsInOrder
#PatchK8sObject: util.#PatchK8sObject
#Steps: {
#do: "steps"
...

View File

@@ -1,8 +0,0 @@
#String: {
#do: "string"
#provider: "convert"
bt: bytes
str?: string
...
}

View File

@@ -6,6 +6,14 @@
...
}
#ApplyInParallel: {
#do: "apply-in-parallel"
#provider: "kube"
cluster: *"" | string
value: [...{...}]
...
}
#Read: {
#do: "read"
#provider: "kube"

View File

@@ -15,8 +15,8 @@
}
#Component: {
name: string
type: string
name?: string
type?: string
properties?: {...}
traits?: [...{
type: string

17
pkg/stdlib/pkgs/util.cue Normal file
View File

@@ -0,0 +1,17 @@
#PatchK8sObject: {
#do: "patch-k8s-object"
#provider: "util"
value: {...}
patch: {...}
result: {...}
...
}
#String: {
#do: "string"
#provider: "util"
bt: bytes
str?: string
...
}

View File

@@ -30,6 +30,7 @@ import (
"cuelang.org/go/cue"
"cuelang.org/go/cue/ast"
"cuelang.org/go/cue/build"
"cuelang.org/go/cue/format"
"cuelang.org/go/encoding/openapi"
"github.com/AlecAivazis/survey/v2"
@@ -62,6 +63,7 @@ import (
"github.com/oam-dev/kubevela/apis/types"
velacue "github.com/oam-dev/kubevela/pkg/cue"
"github.com/oam-dev/kubevela/pkg/cue/model"
"github.com/oam-dev/kubevela/pkg/cue/packages"
"github.com/oam-dev/kubevela/pkg/oam"
)
@@ -146,11 +148,26 @@ func HTTPGet(ctx context.Context, url string) ([]byte, error) {
}
// GetCUEParameterValue converts definitions to cue format
func GetCUEParameterValue(cueStr string) (cue.Value, error) {
r := cue.Runtime{}
template, err := r.Compile("", cueStr+velacue.BaseTemplate)
if err != nil {
return cue.Value{}, err
func GetCUEParameterValue(cueStr string, pd *packages.PackageDiscover) (cue.Value, error) {
var template *cue.Instance
var err error
if pd != nil {
bi := build.NewContext().NewInstance("", nil)
err := bi.AddFile("-", cueStr+velacue.BaseTemplate)
if err != nil {
return cue.Value{}, err
}
template, err = pd.ImportPackagesAndBuildInstance(bi)
if err != nil {
return cue.Value{}, err
}
} else {
r := cue.Runtime{}
template, err = r.Compile("", cueStr+velacue.BaseTemplate)
if err != nil {
return cue.Value{}, err
}
}
tempStruct, err := template.Value().Struct()
if err != nil {

View File

@@ -127,7 +127,7 @@ output: {
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
_, err := GetCUEParameterValue(tc.cueStr)
_, err := GetCUEParameterValue(tc.cueStr, nil)
if tc.want.err != nil {
if diff := cmp.Diff(tc.want.err, err, test.EquateErrors()); diff != "" {
t.Errorf("\n%s\nGenOpenAPIFromFile(...): -want error, +got error:\n%s", tc.reason, diff)
@@ -162,7 +162,7 @@ name
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
_, err := GetCUEParameterValue(tc.cueStr)
_, err := GetCUEParameterValue(tc.cueStr, nil)
if diff := cmp.Diff(tc.want.errMsg, err.Error(), test.EquateConditions()); diff != "" {
t.Errorf("\n%s\nGenOpenAPIFromFile(...): -want error, +got error:\n%s", tc.reason, diff)
}

View File

@@ -0,0 +1,252 @@
/*
Copyright 2021 The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package helm
import (
"bytes"
"context"
"fmt"
"io"
"io/ioutil"
"net/url"
"path"
"path/filepath"
"strings"
"time"
"github.com/pkg/errors"
"helm.sh/helm/v3/pkg/action"
"helm.sh/helm/v3/pkg/chart"
"helm.sh/helm/v3/pkg/chart/loader"
"helm.sh/helm/v3/pkg/kube"
"helm.sh/helm/v3/pkg/release"
"helm.sh/helm/v3/pkg/repo"
"helm.sh/helm/v3/pkg/storage"
"helm.sh/helm/v3/pkg/storage/driver"
appsv1 "k8s.io/api/apps/v1"
crdv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
kyaml "k8s.io/apimachinery/pkg/util/yaml"
"k8s.io/client-go/rest"
k8scmdutil "k8s.io/kubectl/pkg/cmd/util"
"sigs.k8s.io/yaml"
"github.com/oam-dev/kubevela/pkg/utils"
"github.com/oam-dev/kubevela/pkg/utils/common"
cmdutil "github.com/oam-dev/kubevela/pkg/utils/util"
)
// Helper provides helper functions for common Helm operations
type Helper struct {
}
// NewHelper creates a Helper
func NewHelper() *Helper {
return &Helper{}
}
// LoadCharts load helm chart from local or remote
func (h *Helper) LoadCharts(chartRepoURL string) (*chart.Chart, error) {
var err error
var chart *chart.Chart
if utils.IsValidURL(chartRepoURL) {
chartBytes, err := common.HTTPGet(context.Background(), chartRepoURL)
if err != nil {
return nil, errors.New("error retrieving Helm Chart at " + chartRepoURL + ": " + err.Error())
}
ch, err := loader.LoadArchive(bytes.NewReader(chartBytes))
if err != nil {
return nil, errors.New("error retrieving Helm Chart at " + chartRepoURL + ": " + err.Error())
}
return ch, err
}
chart, err = loader.Load(chartRepoURL)
if err != nil {
return nil, err
}
return chart, nil
}
// UpgradeChartOptions options for upgrade chart
type UpgradeChartOptions struct {
Config *rest.Config
Detail bool
Logging cmdutil.IOStreams
Wait bool
ReuseValues bool
}
// UpgradeChart install or upgrade helm chart
func (h *Helper) UpgradeChart(ch *chart.Chart, releaseName, namespace string, values map[string]interface{}, config UpgradeChartOptions) (*release.Release, error) {
if ch == nil || len(ch.Templates) == 0 {
return nil, fmt.Errorf("empty chart provided for %s", releaseName)
}
config.Logging.Infof("Start upgrading Helm Chart %s in namespace %s\n", releaseName, namespace)
cfg, err := newActionConfig(config.Config, namespace, config.Detail, config.Logging)
if err != nil {
return nil, err
}
histClient := action.NewHistory(cfg)
var newRelease *release.Release
timeoutInMinutes := 18
releases, err := histClient.Run(releaseName)
if err != nil {
if errors.Is(err, driver.ErrReleaseNotFound) {
// fresh install
install := action.NewInstall(cfg)
install.Namespace = namespace
install.ReleaseName = releaseName
install.Wait = config.Wait
install.Timeout = time.Duration(timeoutInMinutes) * time.Minute
newRelease, err = install.Run(ch, values)
} else {
return nil, fmt.Errorf("could not retrieve history of releases associated to %s: %w", releaseName, err)
}
} else {
config.Logging.Infof("Found existing installation, overwriting...")
// check if the previous installation is still pending (e.g., waiting to complete)
for _, r := range releases {
if r.Info.Status == release.StatusPendingInstall || r.Info.Status == release.StatusPendingUpgrade ||
r.Info.Status == release.StatusPendingRollback {
return nil, fmt.Errorf("previous installation (e.g., using vela install or helm upgrade) is still in progress. Please try again in %d minutes", timeoutInMinutes)
}
}
// overwrite existing installation
install := action.NewUpgrade(cfg)
install.Namespace = namespace
install.Wait = config.Wait
install.Timeout = time.Duration(timeoutInMinutes) * time.Minute
install.ReuseValues = config.ReuseValues
newRelease, err = install.Run(releaseName, ch, values)
}
// check if install/upgrade worked
if err != nil {
return nil, fmt.Errorf("error when installing/upgrading Helm Chart %s in namespace %s: %w",
releaseName, namespace, err)
}
if newRelease == nil {
return nil, fmt.Errorf("failed to install release %s", releaseName)
}
return newRelease, nil
}
// UninstallRelease uninstalls the provided release
func (h *Helper) UninstallRelease(releaseName, namespace string, config *rest.Config, showDetail bool, logging cmdutil.IOStreams) error {
cfg, err := newActionConfig(config, namespace, showDetail, logging)
if err != nil {
return err
}
iCli := action.NewUninstall(cfg)
_, err = iCli.Run(releaseName)
if err != nil {
return fmt.Errorf("error when uninstalling Helm release %s in namespace %s: %w",
releaseName, namespace, err)
}
return nil
}
// ListVersions list available versions from repo
func (h *Helper) ListVersions(repoURL string, chartName string) (repo.ChartVersions, error) {
var body []byte
if utils.IsValidURL(repoURL) {
parsedURL, err := url.Parse(repoURL)
if err != nil {
return nil, err
}
parsedURL.RawPath = path.Join(parsedURL.RawPath, "index.yaml")
parsedURL.Path = path.Join(parsedURL.Path, "index.yaml")
indexURL := parsedURL.String()
body, err = common.HTTPGet(context.Background(), indexURL)
if err != nil {
return nil, fmt.Errorf("download index file from %s failure %w", repoURL, err)
}
} else {
var err error
body, err = ioutil.ReadFile(path.Join(filepath.Clean(repoURL), "index.yaml"))
if err != nil {
return nil, fmt.Errorf("read index file from %s failure %w", repoURL, err)
}
}
i := &repo.IndexFile{}
if err := yaml.UnmarshalStrict(body, i); err != nil {
return nil, fmt.Errorf("parse index file from %s failure %w", repoURL, err)
}
return i.Entries[chartName], nil
}
// GetDeploymentsFromManifest get deployment from helm manifest
func GetDeploymentsFromManifest(helmManifest string) []*appsv1.Deployment {
deployments := []*appsv1.Deployment{}
dec := kyaml.NewYAMLToJSONDecoder(strings.NewReader(helmManifest))
for {
var deployment appsv1.Deployment
err := dec.Decode(&deployment)
if errors.Is(err, io.EOF) {
break
}
if err != nil {
continue
}
if strings.EqualFold(deployment.Kind, "deployment") {
deployments = append(deployments, &deployment)
}
}
return deployments
}
// GetCRDFromChart get crd from helm chart
func GetCRDFromChart(chart *chart.Chart) []*crdv1.CustomResourceDefinition {
crds := []*crdv1.CustomResourceDefinition{}
for _, crdFile := range chart.CRDs() {
var crd crdv1.CustomResourceDefinition
err := kyaml.Unmarshal(crdFile.Data, &crd)
if err != nil {
continue
}
crds = append(crds, &crd)
}
return crds
}
func newActionConfig(config *rest.Config, namespace string, showDetail bool, logging cmdutil.IOStreams) (*action.Configuration, error) {
restClientGetter := cmdutil.NewRestConfigGetterByConfig(config, namespace)
log := func(format string, a ...interface{}) {
if showDetail {
logging.Infof(format+"\n", a...)
}
}
kubeClient := &kube.Client{
Factory: k8scmdutil.NewFactory(restClientGetter),
Log: log,
}
client, err := kubeClient.Factory.KubernetesClientSet()
if err != nil {
return nil, err
}
s := driver.NewSecrets(client.CoreV1().Secrets(namespace))
s.Log = log
return &action.Configuration{
RESTClientGetter: restClientGetter,
Releases: storage.Init(s),
KubeClient: kubeClient,
Log: log,
}, nil
}

View File

@@ -0,0 +1,68 @@
/*
Copyright 2021 The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package helm
import (
"os"
"github.com/google/go-cmp/cmp"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"github.com/oam-dev/kubevela/pkg/utils/util"
)
var _ = Describe("Test helm helper", func() {
It("Test LoadCharts ", func() {
helper := NewHelper()
chart, err := helper.LoadCharts("./testdata/autoscalertrait-0.1.0.tgz")
Expect(err).Should(BeNil())
Expect(chart).ShouldNot(BeNil())
Expect(chart.Metadata).ShouldNot(BeNil())
Expect(cmp.Diff(chart.Metadata.Version, "0.1.0")).Should(BeEmpty())
})
It("Test UpgradeChart", func() {
helper := NewHelper()
chart, err := helper.LoadCharts("./testdata/autoscalertrait-0.1.0.tgz")
Expect(err).Should(BeNil())
release, err := helper.UpgradeChart(chart, "autoscalertrait", "default", nil, UpgradeChartOptions{
Config: cfg,
Detail: false,
Logging: util.IOStreams{Out: os.Stdout, ErrOut: os.Stderr},
Wait: false,
})
crds := GetCRDFromChart(release.Chart)
Expect(cmp.Diff(len(crds), 1)).Should(BeEmpty())
Expect(err).Should(BeNil())
})
It("Test UninstallRelease", func() {
helper := NewHelper()
err := helper.UninstallRelease("autoscalertrait", "default", cfg, false, util.IOStreams{Out: os.Stdout, ErrOut: os.Stderr})
Expect(err).Should(BeNil())
})
It("Test ListVersions ", func() {
helper := NewHelper()
versions, err := helper.ListVersions("./testdata", "autoscalertrait")
Expect(err).Should(BeNil())
Expect(cmp.Diff(len(versions), 2)).Should(BeEmpty())
})
})

View File

@@ -0,0 +1,63 @@
/*
Copyright 2021 The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package helm
import (
"math/rand"
"testing"
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"k8s.io/client-go/rest"
"k8s.io/utils/pointer"
"sigs.k8s.io/controller-runtime/pkg/envtest"
)
var cfg *rest.Config
var testEnv *envtest.Environment
var _ = BeforeSuite(func(done Done) {
rand.Seed(time.Now().UnixNano())
By("bootstrapping test environment")
testEnv = &envtest.Environment{
ControlPlaneStartTimeout: time.Minute * 3,
ControlPlaneStopTimeout: time.Minute,
UseExistingCluster: pointer.BoolPtr(false),
}
By("start kube test env")
var err error
cfg, err = testEnv.Start()
Expect(err).ShouldNot(HaveOccurred())
Expect(cfg).ToNot(BeNil())
close(done)
}, 240)
var _ = AfterSuite(func() {
if testEnv != nil {
err := testEnv.Stop()
Expect(err).Should(BeNil())
}
})
func TestHelm(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Helm Suite")
}

Binary file not shown.

24
pkg/utils/helm/testdata/index.yaml vendored Normal file
View File

@@ -0,0 +1,24 @@
apiVersion: v1
entries:
autoscalertrait:
- apiVersion: v2
appVersion: 1.16.0
created: "2021-03-13T23:42:30.9837659+09:00"
description: A Helm chart for kubevela autoscalertrait controller
digest: 33c4b9eeabf6c26e6aa8c4edaaf0e4a5d2a2f1fd857ffe32b6d3be97d6d971f0
name: autoscalertrait
type: application
urls:
- https://charts.kubevela.net/example/autoscalertrait-0.1.0.tgz
version: 0.1.0
- apiVersion: v2
appVersion: 1.16.0
created: "2021-03-13T23:42:30.9837659+09:00"
description: A Helm chart for kubevela autoscalertrait controller
digest: 33c4b9eeabf6c26e6aa8c4edaaf0e4a5d2a2f1fd857ffe32b6d3be97d6d971f0
name: autoscalertrait
type: application
urls:
- https://charts.kubevela.net/example/autoscalertrait-0.1.0.tgz
version: 0.2.0
generated: "2021-03-13T23:42:30.9832644+09:00"

View File

@@ -18,6 +18,7 @@ package utils
import (
"fmt"
"net/url"
"regexp"
)
@@ -56,3 +57,16 @@ func ParseAPIServerEndpoint(server string) (string, error) {
}
return fmt.Sprintf("%s://%s:%s", scheme, host, port), nil
}
// IsValidURL checks whether the given string is a valid URL or not
func IsValidURL(strURL string) bool {
_, err := url.ParseRequestURI(strURL)
if err != nil {
return false
}
u, err := url.Parse(strURL)
if err != nil || u.Scheme == "" || u.Host == "" {
return false
}
return true
}

View File

@@ -48,6 +48,14 @@ func NewRestConfigGetter(namespace string) genericclioptions.RESTClientGetter {
}
}
// NewRestConfigGetterByConfig new rest config getter
func NewRestConfigGetterByConfig(config *rest.Config, namespace string) genericclioptions.RESTClientGetter {
return &restConfigGetter{
config: config,
namespace: namespace,
}
}
type restConfigGetter struct {
config *rest.Config
namespace string

View File

@@ -85,11 +85,24 @@ func (c ViewContext) GetMutableValue(paths ...string) string {
func (c ViewContext) SetMutableValue(data string, paths ...string) {
}
// IncreaseMutableCountValue increase mutable count in workflow context.
func (c ViewContext) IncreaseMutableCountValue(paths ...string) int {
// IncreaseCountValueInMemory increase count in workflow context memory store.
func (c ViewContext) IncreaseCountValueInMemory(paths ...string) int {
return 0
}
// SetValueInMemory set data in workflow context memory store.
func (c ViewContext) SetValueInMemory(data interface{}, paths ...string) {
}
// GetValueInMemory get data in workflow context memory store.
func (c ViewContext) GetValueInMemory(paths ...string) (interface{}, bool) {
return "", true
}
// DeleteValueInMemory delete data in workflow context memory store.
func (c ViewContext) DeleteValueInMemory(paths ...string) {
}
// DeleteMutableValue delete mutable data in workflow context.
func (c ViewContext) DeleteMutableValue(paths ...string) {
}

View File

@@ -18,9 +18,11 @@ package application
import (
"context"
"fmt"
"net/http"
admissionv1 "k8s.io/api/admission/v1"
"k8s.io/apimachinery/pkg/util/validation/field"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/runtime/inject"
@@ -67,6 +69,23 @@ func (h *ValidatingHandler) InjectDecoder(d *admission.Decoder) error {
return nil
}
func simplifyError(err error) error {
switch e := err.(type) { // nolint
case *field.Error:
return fmt.Errorf("field \"%s\": %s error encountered, %s. ", e.Field, e.Type, e.Detail)
default:
return err
}
}
func mergeErrors(errs field.ErrorList) error {
s := ""
for _, err := range errs {
s += fmt.Sprintf("field \"%s\": %s error encountered, %s. ", err.Field, err.Type, err.Detail)
}
return fmt.Errorf(s)
}
// Handle validate Application Spec here
func (h *ValidatingHandler) Handle(ctx context.Context, req admission.Request) admission.Response {
app := &v1beta1.Application{}
@@ -77,16 +96,16 @@ func (h *ValidatingHandler) Handle(ctx context.Context, req admission.Request) a
switch req.Operation {
case admissionv1.Create:
if allErrs := h.ValidateCreate(ctx, app); len(allErrs) > 0 {
return admission.Errored(http.StatusUnprocessableEntity, allErrs.ToAggregate())
return admission.Errored(http.StatusUnprocessableEntity, mergeErrors(allErrs))
}
case admissionv1.Update:
oldApp := &v1beta1.Application{}
if err := h.Decoder.DecodeRaw(req.AdmissionRequest.OldObject, oldApp); err != nil {
return admission.Errored(http.StatusBadRequest, err)
return admission.Errored(http.StatusBadRequest, simplifyError(err))
}
if app.ObjectMeta.DeletionTimestamp.IsZero() {
if allErrs := h.ValidateUpdate(ctx, app, oldApp); len(allErrs) > 0 {
return admission.Errored(http.StatusUnprocessableEntity, allErrs.ToAggregate())
return admission.Errored(http.StatusUnprocessableEntity, mergeErrors(allErrs))
}
}
default:

View File

@@ -20,8 +20,8 @@ import (
"context"
"encoding/json"
"fmt"
"strconv"
"strings"
"sync"
"time"
"cuelang.org/go/cue"
@@ -48,13 +48,18 @@ const (
AnnotationStartTimestamp = "vela.io/startTime"
)
var (
workflowMemoryCache sync.Map
)
// WorkflowContext is workflow context.
type WorkflowContext struct {
cli client.Client
store *corev1.ConfigMap
components map[string]*ComponentManifest
vars *value.Value
modified bool
cli client.Client
store *corev1.ConfigMap
memoryStore *sync.Map
components map[string]*ComponentManifest
vars *value.Value
modified bool
}
// GetComponent Get ComponentManifest from workflow context.
@@ -105,7 +110,7 @@ func (wf *WorkflowContext) SetVar(v *value.Value, paths ...string) error {
return nil
}
// GetStore get configmap of workflow context.
// GetStore get store of workflow context.
func (wf *WorkflowContext) GetStore() *corev1.ConfigMap {
return wf.store
}
@@ -121,23 +126,6 @@ func (wf *WorkflowContext) SetMutableValue(data string, paths ...string) {
wf.modified = true
}
// IncreaseMutableCountValue increase mutable count in workflow context.
func (wf *WorkflowContext) IncreaseMutableCountValue(paths ...string) int {
c := wf.GetMutableValue(paths...)
if c == "" {
wf.SetMutableValue("0", paths...)
return 0
}
count, err := strconv.Atoi(c)
if err != nil {
wf.SetMutableValue("0", paths...)
return 0
}
count++
wf.SetMutableValue(strconv.Itoa(count), paths...)
return count
}
// DeleteMutableValue delete mutable data in workflow context.
func (wf *WorkflowContext) DeleteMutableValue(paths ...string) {
key := strings.Join(paths, ".")
@@ -147,6 +135,39 @@ func (wf *WorkflowContext) DeleteMutableValue(paths ...string) {
}
}
// IncreaseCountValueInMemory increase count in workflow context memory store.
func (wf *WorkflowContext) IncreaseCountValueInMemory(paths ...string) int {
key := strings.Join(paths, ".")
c, ok := wf.memoryStore.Load(key)
if !ok {
wf.memoryStore.Store(key, 0)
return 0
}
count, ok := c.(int)
if !ok {
wf.memoryStore.Store(key, 0)
return 0
}
count++
wf.memoryStore.Store(key, count)
return count
}
// SetValueInMemory set data in workflow context memory store.
func (wf *WorkflowContext) SetValueInMemory(data interface{}, paths ...string) {
wf.memoryStore.Store(strings.Join(paths, "."), data)
}
// GetValueInMemory get data in workflow context memory store.
func (wf *WorkflowContext) GetValueInMemory(paths ...string) (interface{}, bool) {
return wf.memoryStore.Load(strings.Join(paths, "."))
}
// DeleteValueInMemory delete data in workflow context memory store.
func (wf *WorkflowContext) DeleteValueInMemory(paths ...string) {
wf.memoryStore.Delete(strings.Join(paths, "."))
}
// MakeParameter make 'value' with interface{}
func (wf *WorkflowContext) MakeParameter(parameter interface{}) (*value.Value, error) {
var s = "{}"
@@ -317,6 +338,11 @@ func NewContext(cli client.Client, ns, app string, appUID types.UID) (Context, e
return wfCtx, wfCtx.Commit()
}
// CleanupMemoryStore cleans up memory store.
func CleanupMemoryStore(app, ns string) {
workflowMemoryCache.Delete(fmt.Sprintf("%s-%s", app, ns))
}
func newContext(cli client.Client, ns, app string, appUID types.UID) (*WorkflowContext, error) {
var (
ctx = context.Background()
@@ -347,11 +373,13 @@ func newContext(cli client.Client, ns, app string, appUID types.UID) (*WorkflowC
store.Annotations = map[string]string{
AnnotationStartTimestamp: time.Now().String(),
}
memCache := getMemoryStore(fmt.Sprintf("%s-%s", app, ns))
wfCtx := &WorkflowContext{
cli: cli,
store: &store,
components: map[string]*ComponentManifest{},
modified: true,
cli: cli,
store: &store,
memoryStore: memCache,
components: map[string]*ComponentManifest{},
modified: true,
}
var err error
wfCtx.vars, err = value.NewValue("", nil, "")
@@ -359,6 +387,20 @@ func newContext(cli client.Client, ns, app string, appUID types.UID) (*WorkflowC
return wfCtx, err
}
func getMemoryStore(key string) *sync.Map {
memCache := &sync.Map{}
mc, ok := workflowMemoryCache.Load(key)
if !ok {
workflowMemoryCache.Store(key, memCache)
} else {
memCache, ok = mc.(*sync.Map)
if !ok {
workflowMemoryCache.Store(key, memCache)
}
}
return memCache
}
// LoadContext load workflow context from store.
func LoadContext(cli client.Client, ns, app string) (Context, error) {
var store corev1.ConfigMap
@@ -372,9 +414,11 @@ func LoadContext(cli client.Client, ns, app string) (Context, error) {
}, &store); err != nil {
return nil, err
}
memCache := getMemoryStore(fmt.Sprintf("%s-%s", app, ns))
ctx := &WorkflowContext{
cli: cli,
store: &store,
cli: cli,
store: &store,
memoryStore: memCache,
}
if err := ctx.LoadFromConfigMap(store); err != nil {
return nil, err

View File

@@ -275,14 +275,33 @@ func TestMutableValue(t *testing.T) {
wfCtx.DeleteMutableValue("test", "key")
v = wfCtx.GetMutableValue("test", "key")
r.Equal(v, "")
}
wfCtx.SetMutableValue("value", "test", "key")
count := wfCtx.IncreaseMutableCountValue("test", "key")
func TestMemoryValue(t *testing.T) {
cli := newCliForTest(t, nil)
r := require.New(t)
wfCtx, err := NewContext(cli, "default", "app-v1", "testuid")
r.NoError(err)
err = wfCtx.Commit()
r.NoError(err)
wfCtx.SetValueInMemory("value", "test", "key")
v, ok := wfCtx.GetValueInMemory("test", "key")
r.Equal(ok, true)
r.Equal(v.(string), "value")
wfCtx.DeleteValueInMemory("test", "key")
_, ok = wfCtx.GetValueInMemory("test", "key")
r.Equal(ok, false)
wfCtx.SetValueInMemory("value", "test", "key")
count := wfCtx.IncreaseCountValueInMemory("test", "key")
r.Equal(count, 0)
count = wfCtx.IncreaseMutableCountValue("notfound", "key")
count = wfCtx.IncreaseCountValueInMemory("notfound", "key")
r.Equal(count, 0)
wfCtx.SetMutableValue("10", "number", "key")
count = wfCtx.IncreaseMutableCountValue("number", "key")
wfCtx.SetValueInMemory(10, "number", "key")
count = wfCtx.IncreaseCountValueInMemory("number", "key")
r.Equal(count, 11)
}

View File

@@ -32,8 +32,11 @@ type Context interface {
GetStore() *corev1.ConfigMap
GetMutableValue(path ...string) string
SetMutableValue(data string, path ...string)
IncreaseMutableCountValue(paths ...string) int
DeleteMutableValue(paths ...string)
IncreaseCountValueInMemory(paths ...string) int
SetValueInMemory(data interface{}, paths ...string)
GetValueInMemory(paths ...string) (interface{}, bool)
DeleteValueInMemory(paths ...string)
Commit() error
MakeParameter(parameter interface{}) (*value.Value, error)
StoreRef() *corev1.ObjectReference

View File

@@ -33,9 +33,6 @@ type Workflow interface {
// Trace record workflow state in controllerRevision.
Trace() error
// CleanupCountersInContext cleans up the temporary counters in workflow context.
CleanupCountersInContext(ctx monitorContext.Context)
// GetBackoffWaitTime returns the wait time for next retry.
GetBackoffWaitTime() time.Duration
}

View File

@@ -1,53 +0,0 @@
/*
Copyright 2021 The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package convert
import (
"github.com/oam-dev/kubevela/pkg/cue/model/value"
wfContext "github.com/oam-dev/kubevela/pkg/workflow/context"
"github.com/oam-dev/kubevela/pkg/workflow/providers"
"github.com/oam-dev/kubevela/pkg/workflow/types"
)
const (
// ProviderName is provider name for install.
ProviderName = "convert"
)
type provider struct {
}
// String convert byte to string
func (h *provider) String(ctx wfContext.Context, v *value.Value, act types.Action) error {
b, err := v.LookupValue("bt")
if err != nil {
return err
}
s, err := b.CueValue().Bytes()
if err != nil {
return err
}
return v.FillObject(string(s), "str")
}
// Install register handlers to provider discover.
func Install(p providers.Providers) {
prd := &provider{}
p.Register(ProviderName, map[string]providers.Handler{
"string": prd.String,
})
}

View File

@@ -1,73 +0,0 @@
/*
Copyright 2021 The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package convert
import (
"errors"
"testing"
"github.com/stretchr/testify/require"
"github.com/oam-dev/kubevela/pkg/cue/model/value"
"github.com/oam-dev/kubevela/pkg/workflow/providers"
)
func TestConvertString(t *testing.T) {
testCases := map[string]struct {
from string
expected string
expectedErr error
}{
"success": {
from: `bt: 'test'`,
expected: "test",
},
"fail": {
from: `bt: 123`,
expectedErr: errors.New("bt: cannot use value 123 (type int) as string|bytes"),
},
}
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
r := require.New(t)
v, err := value.NewValue(tc.from, nil, "")
r.NoError(err)
prd := &provider{}
err = prd.String(nil, v, nil)
if tc.expectedErr != nil {
r.Equal(tc.expectedErr.Error(), err.Error())
return
}
r.NoError(err)
expected, err := v.LookupValue("str")
r.NoError(err)
ret, err := expected.CueValue().String()
r.NoError(err)
r.Equal(ret, tc.expected)
})
}
}
func TestInstall(t *testing.T) {
p := providers.NewProviders()
Install(p)
h, ok := p.GetHandler("convert", "string")
r := require.New(t)
r.Equal(ok, true)
r.Equal(h != nil, true)
}

View File

@@ -91,6 +91,40 @@ func (h *provider) Apply(ctx wfContext.Context, v *value.Value, act types.Action
return v.FillObject(workload.Object, "value")
}
// ApplyInParallel create or update CRs in parallel.
func (h *provider) ApplyInParallel(ctx wfContext.Context, v *value.Value, act types.Action) error {
val, err := v.LookupValue("value")
if err != nil {
return err
}
iter, err := val.CueValue().List()
if err != nil {
return err
}
workloadNum := 0
for iter.Next() {
workloadNum++
}
var workloads = make([]*unstructured.Unstructured, workloadNum)
if err = val.UnmarshalTo(&workloads); err != nil {
return err
}
for i := range workloads {
if workloads[i].GetNamespace() == "" {
workloads[i].SetNamespace("default")
}
}
cluster, err := v.GetString("cluster")
if err != nil {
return err
}
deployCtx := multicluster.ContextWithClusterName(context.Background(), cluster)
if err = h.apply(deployCtx, cluster, common.WorkflowResourceCreator, workloads...); err != nil {
return v.FillObject(err, "err")
}
return nil
}
// Read get CR from cluster.
func (h *provider) Read(ctx wfContext.Context, v *value.Value, act types.Action) error {
val, err := v.LookupValue("value")
@@ -188,9 +222,10 @@ func Install(p providers.Providers, cli client.Client, apply Dispatcher, deleter
cli: cli,
}
p.Register(ProviderName, map[string]providers.Handler{
"apply": prd.Apply,
"read": prd.Read,
"list": prd.List,
"delete": prd.Delete,
"apply": prd.Apply,
"apply-in-parallel": prd.ApplyInParallel,
"read": prd.Read,
"list": prd.List,
"delete": prd.Delete,
})
}

View File

@@ -158,6 +158,7 @@ cluster: ""
err = result.FillObject(expected.Object)
Expect(err).ToNot(HaveOccurred())
})
It("patch & apply", func() {
p := &provider{
apply: func(ctx context.Context, _ string, _ common.ResourceCreatorRole, manifests ...*unstructured.Unstructured) error {
@@ -409,7 +410,6 @@ val: {
err = p.Apply(ctx, v, nil)
Expect(err).To(HaveOccurred())
})
})
func newWorkflowContextForTest() (wfContext.Context, error) {

View File

@@ -20,13 +20,12 @@ import (
"encoding/json"
"strings"
"github.com/oam-dev/kubevela/pkg/cue/model/sets"
"github.com/pkg/errors"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"github.com/oam-dev/kubevela/apis/core.oam.dev/common"
"github.com/oam-dev/kubevela/apis/core.oam.dev/v1beta1"
"github.com/oam-dev/kubevela/pkg/cue/model/sets"
"github.com/oam-dev/kubevela/pkg/cue/model/value"
"github.com/oam-dev/kubevela/pkg/oam"
wfContext "github.com/oam-dev/kubevela/pkg/workflow/context"

View File

@@ -0,0 +1,82 @@
/*
Copyright 2022. The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"github.com/oam-dev/kubevela/pkg/cue/model"
"github.com/oam-dev/kubevela/pkg/cue/model/value"
wfContext "github.com/oam-dev/kubevela/pkg/workflow/context"
"github.com/oam-dev/kubevela/pkg/workflow/providers"
"github.com/oam-dev/kubevela/pkg/workflow/types"
)
const (
// ProviderName is provider name for install.
ProviderName = "util"
)
type provider struct{}
func (p *provider) PatchK8sObject(ctx wfContext.Context, v *value.Value, act types.Action) error {
val, err := v.LookupValue("value")
if err != nil {
return err
}
pv, err := v.LookupValue("patch")
if err != nil {
return err
}
base, err := model.NewBase(val.CueValue())
if err != nil {
return err
}
patcher, err := model.NewOther(pv.CueValue())
if err != nil {
return err
}
if err = base.Unify(patcher); err != nil {
return v.FillObject(err, "err")
}
workload, err := base.Unstructured()
if err != nil {
return v.FillObject(err, "err")
}
return v.FillObject(workload.Object, "result")
}
// String convert byte to string
func (p *provider) String(ctx wfContext.Context, v *value.Value, act types.Action) error {
b, err := v.LookupValue("bt")
if err != nil {
return err
}
s, err := b.CueValue().Bytes()
if err != nil {
return err
}
return v.FillObject(string(s), "str")
}
// Install register handlers to provider discover.
func Install(p providers.Providers) {
prd := &provider{}
p.Register(ProviderName, map[string]providers.Handler{
"patch-k8s-object": prd.PatchK8sObject,
"string": prd.String,
})
}

View File

@@ -0,0 +1,202 @@
/*
Copyright 2022. The KubeVela Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/oam-dev/kubevela/pkg/cue/model/value"
"github.com/oam-dev/kubevela/pkg/workflow/providers"
)
func TestPatchK8sObject(t *testing.T) {
testcases := map[string]struct {
value string
expectedErr error
patchResult string
}{
"test patch k8s object": {
value: `
value: {
apiVersion: "apps/v1"
kind: "Deployment"
spec: template: metadata: {
labels: {
"oam.dev/name": "test"
}
}
}
patch: {
spec: template: metadata: {
labels: {
"test-label": "true"
}
}
}
`,
expectedErr: nil,
patchResult: `
apiVersion: "apps/v1"
kind: "Deployment"
spec: template: metadata: {
labels: {
"oam.dev/name": "test"
"test-label": "true"
}
}
`,
},
"test patch k8s object with patchKey": {
value: `
value: {
apiVersion: "apps/v1"
kind: "Deployment"
spec: template: spec: {
containers: [{
name: "test"
}]
}
}
patch: {
spec: template: spec: {
// +patchKey=name
containers: [{
name: "test"
env: [{
name: "test-env"
value: "test-value"
}]
}]
}
}
`,
expectedErr: nil,
patchResult: `
apiVersion: "apps/v1"
kind: "Deployment"
spec: template: spec: {
containers: [{
name: "test"
env: [{
name: "test-env"
value: "test-value"
}]
}]
}
`,
},
"test patch k8s object with patchStrategy": {
value: `
value: {
apiVersion: "apps/v1"
kind: "Deployment"
spec: template: metadata: {
name: "test-name"
}
}
patch: {
// +patchStrategy=retainKeys
spec: template: metadata: {
name: "test-patchStrategy"
}
}
`,
expectedErr: nil,
patchResult: `
apiVersion: "apps/v1"
kind: "Deployment"
spec: template: metadata: {
name: "test-patchStrategy"
}
`,
},
}
for name, tc := range testcases {
t.Run(name, func(t *testing.T) {
r := require.New(t)
v, err := value.NewValue(tc.value, nil, "")
r.NoError(err)
prd := &provider{}
err = prd.PatchK8sObject(nil, v, nil)
if tc.expectedErr != nil {
r.Equal(tc.expectedErr.Error(), err.Error())
return
}
r.NoError(err)
result, err := v.LookupValue("result")
r.NoError(err)
var patchResult map[string]interface{}
r.NoError(result.UnmarshalTo(&patchResult))
var expectResult map[string]interface{}
resultValue, err := value.NewValue(tc.patchResult, nil, "")
r.NoError(err)
r.NoError(resultValue.UnmarshalTo(&expectResult))
assert.Equal(t, expectResult, patchResult)
})
}
}
func TestConvertString(t *testing.T) {
testCases := map[string]struct {
from string
expected string
expectedErr error
}{
"success": {
from: `bt: 'test'`,
expected: "test",
},
"fail": {
from: `bt: 123`,
expectedErr: errors.New("bt: cannot use value 123 (type int) as string|bytes"),
},
}
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
r := require.New(t)
v, err := value.NewValue(tc.from, nil, "")
r.NoError(err)
prd := &provider{}
err = prd.String(nil, v, nil)
if tc.expectedErr != nil {
r.Equal(tc.expectedErr.Error(), err.Error())
return
}
r.NoError(err)
expected, err := v.LookupValue("str")
r.NoError(err)
ret, err := expected.CueValue().String()
r.NoError(err)
r.Equal(ret, tc.expected)
})
}
}
func TestInstall(t *testing.T) {
p := providers.NewProviders()
Install(p)
h, ok := p.GetHandler("util", "string")
r := require.New(t)
r.Equal(ok, true)
r.Equal(h != nil, true)
}

View File

@@ -286,7 +286,7 @@ func (exec *executor) err(ctx wfContext.Context, err error, reason string) {
}
func (exec *executor) checkErrorTimes(ctx wfContext.Context) {
times := ctx.IncreaseMutableCountValue(wfTypes.ContextPrefixFailedTimes, exec.wfStatus.ID)
times := ctx.IncreaseCountValueInMemory(wfTypes.ContextPrefixFailedTimes, exec.wfStatus.ID)
if times >= MaxErrorTimes {
exec.wait = false
exec.failedAfterRetries = true

View File

@@ -243,6 +243,7 @@ close({
r.Equal(operation.Waiting, true)
r.Equal(status.Phase, common.WorkflowStepPhaseFailed)
case "failed-after-retries":
wfContext.CleanupMemoryStore("app-v1", "default")
newCtx := newWorkflowContextForTest(t)
for i := 0; i < MaxErrorTimes; i++ {
status, operation, err = run.Run(newCtx, &types.TaskRunOptions{})

View File

@@ -30,11 +30,11 @@ import (
"github.com/oam-dev/kubevela/pkg/velaql/providers/query"
wfContext "github.com/oam-dev/kubevela/pkg/workflow/context"
"github.com/oam-dev/kubevela/pkg/workflow/providers"
"github.com/oam-dev/kubevela/pkg/workflow/providers/convert"
"github.com/oam-dev/kubevela/pkg/workflow/providers/email"
"github.com/oam-dev/kubevela/pkg/workflow/providers/http"
"github.com/oam-dev/kubevela/pkg/workflow/providers/kube"
"github.com/oam-dev/kubevela/pkg/workflow/providers/time"
"github.com/oam-dev/kubevela/pkg/workflow/providers/util"
"github.com/oam-dev/kubevela/pkg/workflow/providers/workspace"
"github.com/oam-dev/kubevela/pkg/workflow/tasks/custom"
"github.com/oam-dev/kubevela/pkg/workflow/tasks/template"
@@ -77,8 +77,9 @@ func suspend(step v1beta1.WorkflowStep, opt *types.GeneratorOptions) (types.Task
func NewTaskDiscover(providerHandlers providers.Providers, pd *packages.PackageDiscover, cli client.Client, dm discoverymapper.DiscoveryMapper) types.TaskDiscover {
// install builtin provider
workspace.Install(providerHandlers)
convert.Install(providerHandlers)
email.Install(providerHandlers)
util.Install(providerHandlers)
templateLoader := template.NewWorkflowStepTemplateLoader(cli, dm)
return &taskDiscover{
builtins: map[string]types.TaskGenerator{
@@ -123,7 +124,6 @@ func NewViewTaskDiscover(pd *packages.PackageDiscover, cli client.Client, cfg *r
time.Install(handlerProviders)
kube.Install(handlerProviders, cli, apply, delete)
http.Install(handlerProviders, cli, viewNs)
convert.Install(handlerProviders)
email.Install(handlerProviders)
templateLoader := template.NewViewTemplateLoader(cli, viewNs)

Some files were not shown because too many files have changed in this diff Show More