github/awesome-kubernetes
1
0
Fork 0
mirror of https://github.com/nubenetes/awesome-kubernetes.git synced 2026-03-05 11:10:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a50bcdf40f0e11aec00ae7decdcd24552e2286c4
awesome-kubernetes/scripts/README.md
Inaki a128c1a6f5 Update README.md
2019-07-06 11:14:07 +02:00

363 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Scripts Inventory
* **cleanup.Jenkinsfile**: Jenkinsfile with Declarative Pipeline Multiline sh that cleanups old builds. All the Stages are now visually monitored. It is triggered every saturday night and ends with jenkins restart. These Multi-line bash commands make easier to read Jenkins Projects.
* **daily_restart.Jenkinsfile**: A script that automatically triggers a daily restart of Jenkins due to performance issues (Jenkins is a Java application). Jenkins with Declarative Pipeline multiline sh that restarts Jenkins every night except on Saturday nights (when cleanup.Jenkinsfile is triggered).
* **confluence6-docker-build.Jenkinsfile**: Declarative Jenkinsfile for building and uploading a docker image to Openshift-DEV, Dockerhub and Openshift-PROD (Stages are disabled via Conditional Build Steps). Tip: A Docker Plugin for Jenkins can easily replace this Jenkinsfile. [This script can be found here](https://github.com/inafevwork/confluence6-atlassian).
## Configuration requirements
This cleanup.Jenkinsfile script is supposed to be run in jenkins master nodes, where all the legacy build artifacts are achived ($JENKINS_HOME/jobs/YourProjectName/builds).
### Jenkins-CLI Authentication
Authentication is preferably with an -auth option, which takes a **username:apitoken** argument. Get your API token from **[jenkins_url]/me/configure**
The corresponding q-number's password can be used instead of the mentioned **"apitoken"**. The former one is LDAP based, while **apitoken is unique to each Jenkins Master node.**
Username and password credentials (or username and apitoken) need to be saved in each jenkins master server in **$JENKINS_HOME/.jenkins-cli** with the following format: **"username:password"** or **"username:apitoken"**.
The chosen username requires **Overall/Administer permission** in order to restart Jenkins. Otherwise you will obtain the following error:
```bash
ERROR: q-username is missing the Overall/Administer permission
```
### Setup a new Pipeline Job in Jenkins
1. Jenkins job -> New Item -> "job name" Pipeline -> Pipeline SCM: git , Repository: "this repo", **Script Path: cleanup.Jenkinsfile**
2. Jenkins job -> New Item -> "job name" Pipeline -> Pipeline SCM: git , Repository: "this repo", **Script Path: daily_restart.Jenkinsfile**
3. Trigger a first manual build of the created job to load configuration from Jenkinsfile
## Technical details
### Where Jenkins stores build files
* **Jenkins Master nodes**: jenkins master nodes archive all the legacy build artifacts in **$JENKINS_HOME/jobs/YourProjectName/builds**.
* **Jenkins Slave nodes**: jenkins slaves keep the latest build in their corresponding Workspace in **/home/cloud/jenkins_slave/workspace/YourJobName**.
### Jenkins reload-configuration after cleaning up old builds
Once old build folders have been deleted, jenkins' index of builds needs to be updated by reloading its configuration (fast) or by restarting jenkins (slow). This **"reload-configuration"** of Jenkins **is finally not run in Jenkinsfile cleanup pipeline** as we are proceeding with an automated jenkins restart in its last stage. This restart needs to be done every night and it makes sense to be included in this cleanup job.
### Jenkins restart VS safe-restart
- [jenkins_url]/safeRestart. – This will restart Jenkins after the current builds have completed.
- [jenkins_url]/restart – This will force a restart. Builds will not wait to complete.
**daily_restart.Jenkinsfile** and **cleanup.Jenkinsfile** apply a hard **restart** of Jenkins Master when run in **Main Jenkins Master Enterprise**. A **safe-restart** was initially setup for all Jenkins Master nodes but it could easily last for 2-3 hours in Main Jenkins (sometimes requiring the Jenkins Admin to kill the remaining zombie jobs).
The current configuration is the following:
- **Main Jenkins Master: restart**
- Remaining Jenkins Master nodes: safeRestart
### Unusual characters in foldernames/filenames and IFS env var
IFS env var is modified within this script in order to avoid errors when find command tries to navigate through a Project's folder with spaces in its name. By setting this up we make sure that all the existing folders related to Jenkins Projects are treated correctly and no errors arise when this script is run.
### Summary of lines and parameters found in cleanup script
Item | Parameter | Details
------------ | ------------ |-------------
IFS env var | IFS=$(echo -en "\n\b") | IFS lines discard any space and meta char in a folder's name
find command | -xdev | Don't descend directories on other filesystems
find command | -mtime n | File's data was last modified n*24 hours ago
find command | -delete | removes a file , not valid with directories
find command | -mindepth 1 -maxdepth 1 -type d | no recursive search for directories inside the current directory
find command | -print0 | ASCII NUL character to separate the entries in the file list that it produces (Unusual characters in filenames)
xargs command | -0 | assume that arguments are separated with ASCII NUL instead of white space
df command | cut -d" " -f2- | first column output is removed as device name is too long to be displayed
xargs command | -t | Used in cleanup-script.sh to show a list of old build folders to be deleted. Used in Jenkinsfile to get traces of deleted data.
## Troubleshooting an OpenShift POD running Jenkins Master node
This chapter is necessary to understand how to deal with a trouble Jenkins Enterprise POD (unresponsive and/or with more restarts than the ones triggered by this daily restart).
### Solutions to Known Errors Matrix Table
The below matrix table aims to help with solutions to well known incidents:
Error | Solution or Workaround |
------------ | ------------ |
Jenkins Enterprise is unresponsive or extremely slow | Force a POD Restart by **A)** Scaling the POD to 0 and then to 1 from Openshift GUI, or by **B)** killing the Java Process of Jenkins: **kill 7**|
Restart of Jenkins Enterprise and its POD is taking too long > 1 hour | Deploy Jenkins Microservice: **A)** "oc rollout latest jenkins", or **B)** clicking on 'Deploy' button in Openshift GUI|
### Container State. Pod exitCode errors
- Exit codes seen when using oc get pods or kubectl get pods are unix shell exit codes (echo $?).
- Google "bash error code 137", "bash error code 143", etc.
- Signal 15 is a SIGTERM (see "kill -l" for a complete list). It's the way most programs are gracefully terminated, and is relatively normal
behaviour.This indicates system has delivered a SIGTERM to the processes. This is usually at the request of some other process (via kill()) but could also be sent by your process to itself (using raise()). This signal requests an orderly shutdown of process or system itself. **The real question is "Who/what is sending the SIGTERM?"**
Exit Code Number | Meaning | Example | Comments |
------------ | ------------ | ------------ | ------------ |
128 +n | Fatal error signal "n" | kill -n $PPID of script | $? returns 128 + n|
137 (128 + 9) | Fatal error signal "9" (SIGKILL) | kill -9 $PPID of script | $? returns 137 (128 + 9)|
143 (128 + 15) | Fatal error signal "15" (SIGTERM) | kill -15 $PPID of script | $? returns 143 (128 + 15)|
#### List of SIGTERM signals
```bash
user@jumphost:~> kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL
5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE
9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2
13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT
17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU
25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH
29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN
35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4
39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12
47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14
51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10
55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6
59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
63) SIGRTMAX-1 64) SIGRTMAX
```
#### No errors arise after a new POD deployment
1. No errors are seen when Jenkins Microservice has been deployed again via OpenShift GUI (deploy button) or via "oc rollout latest jenkins_id". Deployment number will be increased by 1 (like deployment #20, or "revision number 20" in 'oc rollout history'). Restart Count is 0:
```
Container jenkins
State:
Running since May 10, 2019 11:24:11 AM
Ready:
true
Restart Count:
0
```
#### Kubernetes/Openshift exit code 137
- Container with Out of Memory Kill. OutOfMemory error in Jenkins ("OOM" error code). Container/POD is using too much memory and is killed automatically (new restart triggered)
- Java Process killed manually with a "kill -9 [PID]" (as explained above)
#### Kubernetes/Openshift exit code 143
- When scaling down number of pods, the following error is reported in the OpenShift console, even though the pod is stopped correctly:
```
The container spring-boot did not stop cleanly when terminated (exit code 143).
```
Root Cause: This issue is related to the exit codes returned from the JVM when it receives signals such as SIGINT, SIGTERM, etc. The JVM default is to return an exit code of 128+signal-id. E.g for SIGTERM one would see an exit code of 143 (128+15). This causes Red Hat OpenShift Container Platform to display a warning message on pod scale down stating that that the container did not stop cleanly (when in actual fact, it did).
- This error also arises when Jenkins is restarted:
- A manual or automated restart of Jenkins also implies a new Container/POD restart.
- Container **restartCount** is increased
- A Jenkins Restart is triggered automatically by **Jenkinsfile-daily-restart** found in this repo (Jenkins cli).
- A Jenkins Restart can also be accomplished from the Jenkins GUI: This is not available when Jenkins is overloaded and unresponsive.
- **Simple workaround to force a Jenkins Restart:** In the following example we force a Container/POD restart by killing Jenkins Java process with a **"kill PID"** (SIGTERM signal, instead of SIGKILL signal with "kill -9"):
```bash
user@jumpserver:~/> oc get pods | grep ^jenkins-2
jenkins-20-1d4vm 1/1 Running 0 45m
user@jumpserver:~/> oc rsh jenkins-20-1d4vm
sh-4.2$ ps ux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1019710+ 1 0.0 0.0 4324 560 ? Ss 11:24 0:00 /bin/tini -vv
1019710+ 7 109 0.5 61154992 5864592 ? Sl 11:24 49:43 java -Djava.aw
1019710+ 938 0.0 0.0 13520 2076 ? Ss+ 11:27 0:00 /bin/sh
1019710+ 3037 0.0 0.0 13520 2000 ? Ss+ 11:31 0:00 /bin/sh
1019710+ 7723 0.0 0.0 13516 2064 ? Ss+ 11:44 0:00 /bin/sh
1019710+ 16464 2.0 0.0 13516 1936 ? Ss 12:09 0:00 /bin/sh
1019710+ 16474 0.0 0.0 49060 1828 ? R+ 12:09 0:00 ps ux
sh-4.2$ kill 7
```
### Openshift oc cli examples
- oc get pods
- **oc get pods -o wide**
- oc describe pod jenkins-as-a-service-73-07xwx
- oc describe svc jaas-test
- **oc get svc**
- oc set env pods --all --list
- **ps -eo pid,lstart,cmd | fold -w 130** (checking Jenkins Java process with arguments and start time)
```bash
user@jumpserver:~> oc rsh jenkins-19-0n5jm
sh-4.2$ ps -eo pid,lstart,cmd | fold -w 130
PID STARTED CMD
1 Sat Apr 28 00:54:42 2019 /usr/bin/dumb-init -- /usr/libexec/s2i/run
7 Sat Apr 28 00:54:42 2019 java -XX:+UseParallelGC -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:GCTimeRatio=4 -XX:Adap
tiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -Duser.home=/var/lib/jenkins -XX:MaxMetaspaceSize=512m -Djavamelody.application-
name=JENKINS -Dfile.encoding=UTF8 -jar /usr/lib/jenkins/jenkins.war --prefix=/myjenkins
45744 Sun Apr 29 15:12:43 2018 /bin/sh
47707 Wed May 9 16:51:16 2018 /bin/sh
54481 Wed May 9 18:20:04 2018 /bin/sh
54656 Wed May 9 18:22:41 2018 ps -eo pid,lstart,cmd
54657 Wed May 9 18:22:41 2018 fold -w 130
```
- oc get pod jenkins-19-0n5jm
- POD Status: **oc get pod jenkins-19-0n5jm -o yaml**
- In the following example, LastState:
- startedAt 2019-05-02T05:40:30Z (05:40:30 UTC -> 07:40:30 CEST)
- finishedAt 2019-05-03T04:38:08Z (04:38:08 UTC -> 06:38:08 CEST)
- CurrentState startedAt: ...
```bash
containerStatuses:
- containerID: docker://9ad7f3a23d6328405090a380ed01305a9e21fefca91a49021db9b3848f208d09
image: myip:5000/prod/jenkins
imageID: docker-pullable://myip:5000/delivery-prod/jenkins@sha256:8f5d31f51cef106df8e68ed432e190f9f6bf0e9307d3f35490584cc4db808c0d
lastState:
terminated:
containerID: docker://de0a32ddab499573fe7cd79745751b6eb2d1e3d84e4c0f529929985ed5c9947a
exitCode: 137
finishedAt: 2019-05-03T04:38:08Z
reason: Error
startedAt: 2019-05-02T05:40:30Z
name: jenkins
ready: true
restartCount: 9
state:
...
```
- **POD Status: oc get pod jenkins-20-1d4vm -o yaml**
- **In the following example, LastState:**
- **startedAt:** 2019-05-14T03:01:40Z (3:01:40 UTC -> **5:01:40 CEST , daily restart**)
- **finishedAt:** 2019-05-14T12:06:03Z (12:06:03 UTC -> **14:06:03 CEST**)
- **CurrentState:**
- **startedAt:** 2019-05-14T12:06:09Z (12:06:09 UTC -> **14:06:09 CEST**)
```bash
containerStatuses:
- containerID: docker://ed4e7b3c2e427d0f5e3940bafe21737173a00df3f09e6853f53b85a47e023c30
image: myip:5000/delivery-prod/jenkins
imageID: docker-pullable://myip:5000/delivery-prod/jenkins@sha256:8f5d31f51cef106df8e68ed432e190f9f6bf0e9307d3f35490584cc4db808c0d
lastState:
terminated:
containerID: docker://92f7db3fbb442d1609abdad498051355266ef0072e97f226e2f9374932671044
exitCode: 137
finishedAt: 2019-05-14T12:06:03Z
reason: Error
startedAt: 2019-05-14T03:01:40Z
name: jenkins
ready: true
restartCount: 5
state:
running:
startedAt: 2019-05-14T12:06:09Z
hostIP: myip
phase: Running
podIP: ip-addr
startTime: 2019-05-10T09:24:04Z
```
- **oc get dc**
```bash
user@jumphost:~/> oc get dc
NAME REVISION DESIRED CURRENT TRIGGERED BY
cjoc 11 1 1
jenkins 19 1 1
jenkins-1 27 0 0
jenkins-2 20 1 1
jenkins-3 1 0 0
jenkins-4 0 1 0
jenkins-5 9 1 1
jenkins-6 14 1 1
jenkins-7 9 1 1
nexus 7 1 1
sonar6 4 1 1
sonar6-postgresql 1 1 1 config,image(postgresql:9.5)
```
- oc deploy --latest jenkins (deprecated, use "oc rollout jenkins" instead)
```bash
user@jumphost:~/> oc deploy --latest jenkins
Flag --latest has been deprecated, use 'oc rollout latest' instead
Started deployment #20
Use 'oc logs -f dc/jenkins' to track its progress.
user@jumphost:~/>
```
- **oc rollout latest jenkins**: This is like clicking on "deploy" button on Openshift GUI when we want to deploy a POD. **PROCEED WITH THIS IN CASE A RESTART OF JENKINS IS STUCK FOR MORE THAN 1 HOUR**. Deployment number will be increased by 1 (like deployment #20, or "revision number 20" in 'oc rollout history').
- **oc logs -f dc/jenkins**
- oc rollout history dc/jenkins
```bash
user@jumphost:~/> oc rollout history dc/jenkins
deploymentconfigs "jenkins"
REVISION STATUS CAUSE
1 Complete manual change
2 Complete manual change
4 Complete manual change
5 Complete manual change
6 Complete manual change
7 Complete manual change
8 Complete manual change
9 Failed The deployment was cancelled by the user
10 Complete manual change
11 Failed manual change
12 Failed The deployment was cancelled by the user
13 Failed The deployment was cancelled by the user
14 Failed The deployment was cancelled by the user
15 Failed manual change
16 Failed The deployment was cancelled by the user
17 Failed manual change
18 Failed manual change
19 Complete manual change
20 Running manual change
```
- **Force the termination of POD:**
```bash
$ oc delete pod jenkins-10-6095v --force=true --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "jenkins-10-6095v" deleted
```
- **oc describe quota**
- oc describe quota quota-48gi -n delivery-prod ("oc describe quota" in default delivery-prod Project)
```bash
user@jumphost:~/> oc describe quota
Name: quota-48gi
Namespace: delivery-prod
Resource Used Hard
-------- ---- ----
requests.cpu 22750m 30
requests.memory 36467264Ki 48Gi
```
- oc status
- **oc status -v** (warnings can be seen)
- **oc get all**
- **oc logs -f po/jenkins-19-0n5jm | less -S**
- oc scale --replicas=0 dc jenkins (Change the number of pods in a deployment)
- oc scale --replicas=3 dc jenkins (Change the number of pods in a deployment)
- **oc rollout history po/jenkins-19-0n5jm** (viewing a deployment)
- **oc rollout history dc/jenkins --revision=19**
- **oc get builds**
- oc edit: Edit a resource on the server
- oc new-build: Create a new build configuration
- oc start-build: Start a new build
- oc cancel-build: Cancel running, pending, or new builds
- oc import-image: Imports images from a Docker registry
- oc tag: Tag existing images into image streams
- **oc debug jenkins-52-167c9** (Launch a new instance of a pod for **debugging**)
- oc explain: Documentation of resources
- oc explain deploy (Deployment enables declarative updates for Pods and ReplicaSets)
- oc exec: Execute a command in a container
- oc cp: Copy files and directories to and from containers.
### Openshift GUI examples
- We can see the following output in OpenShift GUI after a jenkins restart (which implies a POD restart, with this POD's "restartCount" increased by 1):
```
Status:
Running Deployment:
jenkins, #19
IP:
ipaddr.ip
Node:
nodeaddr.net (ipaddr)Restart Policy:
Always
Container jenkins
State:
Running since May 4, 2019 6:37:11 AM
Last State
Terminated at May 4, 2019 6:37:01 AM with exit code 143 (Error)
Ready:
false
Restart Count:
10
Debug in Terminal
```
# Jenkins Slaves Cleanup Scripts
It can be easily automated via a jenkins job that triggers the following shell script:
* ```bash
mv /home/cloud/jenkins_slave/workspace /home/cloud/jenkins_slave/workspace.old && rm -Rf /home/cloud/jenkins_slave/workspace.old &
mv /global/apps/cdbuilt/build_server/maven/<myproject>/local-repo-maven-3 /global/apps/cdbuilt/build_server/maven/<myproject>/local-repo-maven-3.old && rm -Rf /global/apps/cdbuilt/build_server/maven/<myproject>/local-repo-maven-3.old &
```
* Alternative: Workspace cleanup plugin: https://plugins.jenkins.io/ws-cleanup
# References
* [Jenkins.io: **Jenkins CLI Authentication**](https://jenkins.io/doc/book/managing/cli/)
* [Stackoverflow: How to restart jenkins manually](https://stackoverflow.com/questions/8072700/how-to-restart-jenkins-manually)
* [**Cloudbees Support Guideline: Cleanup and disk space management**](https://support.cloudbees.com/hc/en-us/articles/215549798-Deleting-Old-Builds-Best-Strategy-for-Cleanup-and-disk-space-management)
* [GNU.org: Deleting files with find](https://www.gnu.org/software/findutils/manual/html_node/find_html/Deleting-Files.html)
* [Dzone.com: **Declarative Pipeline Refcard**](https://dzone.com/refcardz/declarative-pipeline-with-jenkins)
* [Cloudbees: Declarative Pipeline Quick Reference](https://www.cloudbees.com/sites/default/files/declarative-pipeline-refcard.pdf)
* [Kubernetes Docs](https://kubernetes.io/docs/home)
* [Kubernetes.io: **Pod Lifecycle**](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)
* [Openshift: Application memory sizing](https://docs.openshift.com/container-platform/3.9/dev_guide/application_memory_sizing.html)
* [**Openshift: Openshift Container Platform 3.9 CLI Reference**](https://docs.openshift.com/container-platform/3.9/cli_reference/index.html)
* [**Openshift: Basic CLI Operations - Object types**](https://docs.openshift.com/enterprise/3.2/cli_reference/basic_cli_operations.html#object-types)
* [**Youtube: Introduction to OpenShift Enterprise CLI**](https://www.youtube.com/watch?v=iRrAYnDSF4c)
* [**redhat.com Video: OpenShift Enterprise 3.2 - Creating your first application from the CLI**](https://access.redhat.com/videos/2481041/)
* [Openshift: Exit Code 137](https://blog.danman.eu/openshift-exit-code-137/)
* [**redhat.com: Openshift. Diagnosing an OOM Kill**](https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html/developer_guide/dev-guide-application-memory-sizing#diagnosing-an-oom-kill)
* [redhat.com: Scaling down pods with a running Java process results in a warning message (exit code 143)](https://access.redhat.com/solutions/3100021)
* [redhat.com: Bash exit codes with special meanings](https://access.redhat.com/solutions/196563)
* [redhat.com: What is 'Signal 15' ?](https://access.redhat.com/solutions/737033)
* [redhat.com: How to find out who sent the 'signal-15'](https://access.redhat.com/solutions/360993)
* [Stackoverflow: How to debug container images using openshift](https://stackoverflow.com/questions/41771430/how-to-debug-container-images-using-openshift)
* [Stackoverflow: How to restart pod in OpenShift?](https://stackoverflow.com/questions/49562433/how-to-restart-pod-in-openshift)
* [Reddit.com: jenkinsci](https://www.reddit.com/r/jenkinsci/)
Reference in New Issue View Git Blame Copy Permalink