The complete CI/CD Setup (Part 5 - Pull Request Testing and Deployment)
In this series of blog posts I’ll describe how to setup on a small environment a complete Continuous Integration and Continuous Deployment pipeline, which components are required and how to deploy them and how to make this a bit more general for additional applications.
- In the first part I’ve made an overview and described the components that I’ll deploy.
- In the second part we have deployed the components that are required for the pipeline.
- In the third part we have created our first Continuous Integration and Continuous Deployment pipeline
- In the fourth part we have created tests via Postman and executed them automatically after each Deployment
- In this part we will find a way to deploy automatically each new Pull Request to allow testers to validate the running application
Simple Pull Request Deployment
First of all we need to find a way to identify if a build is for a commit or a Pull Request. Therefore we can print out the available environment variables via the following Jenkins command:
echo sh(returnStdout: true, script: 'env')
There is an environment variable called CHANGE_BRANCH. If this one is present the job is executed as a Pull Request. So if we now extend our if-condition to not only deploy and test when we are running a master-Branch job but also if the CHANGE_BRANCH environment variable is present we are already able to deploy a new version on each Pull Request.
def label = "ci-cd-${UUID.randomUUID().toString()}"
podTemplate(label: label,
serviceAccount: 'helm',
containers: [
containerTemplate(name: 'maven', image: 'maven:3.6-jdk-11-slim', ttyEnabled: true, command: 'cat'),
containerTemplate(
name: 'docker',
image: 'docker',
command: 'cat',
ttyEnabled: true,
envVars: [
secretEnvVar(key: 'REGISTRY_USERNAME', secretName: 'registry-credentials', secretKey: 'username'),
secretEnvVar(key: 'REGISTRY_PASSWORD', secretName: 'registry-credentials', secretKey: 'password')
]
),
containerTemplate(name: 'helm', image: 'alpine/helm:2.11.0', command: 'cat', ttyEnabled: true),
containerTemplate(name: 'newman', image: 'postman/newman', ttyEnabled: true, command: 'cat')
],
volumes: [
hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')
]) {
node(label) {
stage('Checkout project') {
def repo = checkout scm
def dockerImage = "ci-cd-guide/spring-boot"
def dockerTag = "${repo.GIT_BRANCH}-${repo.GIT_COMMIT}"
def registryUrl = "registry.home.koudingspawn.de"
def helmName = "spring-boot-${repo.GIT_BRANCH}"
def url = "${helmName}.home.koudingspawn.de"
def certVaultPath = "certificates/*.home.koudingspawn.de"
container('maven') {
stage('Build project') {
sh 'mvn -B clean install'
}
}
if (env.CHANGE_BRANCH?.trim() || "${repo.GIT_BRANCH}" == "master") {
container('docker') {
stage('Build docker') {
sh "docker build -t ${registryUrl}/${dockerImage}:${dockerTag} ."
sh "docker login ${registryUrl} --username \$REGISTRY_USERNAME --password \$REGISTRY_PASSWORD"
sh "docker push ${registryUrl}/${dockerImage}:${dockerTag}"
sh "docker rmi ${registryUrl}/${dockerImage}:${dockerTag}"
}
}
container('helm') {
stage("Deploy") {
sh("""helm upgrade --install ${helmName} --namespace='${helmName}' \
--set ingress.enabled=true \
--set ingress.hosts[0]='${url}' \
--set ingress.annotations.\"kubernetes\\.io/ingress\\.class\"=nginx \
--set ingress.tls[0].hosts[0]='${url}' \
--set ingress.tls[0].secretName='tls-${url}' \
--set ingress.tls[0].vaultPath='${certVaultPath}' \
--set ingress.tls[0].vaultType='CERT' \
--set image.tag='${dockerTag}' \
--set image.repository='${registryUrl}/${dockerImage}' \
--wait \
./helm""")
}
}
container('newman') {
stage("Test") {
sh("""
newman run \
Demo-App-Collection.postman_collection.json \
--env-var url=https://${url} \
-n 2
""")
}
}
}
}
}
}
Notify the Pull Request
The next step is to inform our Pull Request, that there is a new testing environment available that can be checked. Therefore we extend our pod again and add now a curl container, that sends a HTTP Post every time the application is deployed:
containerTemplate(name: 'curl', image: 'tutum/curl', ttyEnabled: true, command: 'cat', envVars: [
secretEnvVar(key: 'GITEA_TOKEN', secretName: 'gitea-credentials', secretKey: 'accesstoken')
])
To allow Jenkins to write a comment to our Pull Request we must first generate a gitea api token. In the code above we mount this token into the container to use it during our HTTP Post. Therefore we click in Gitea on the arrow next to our profile and on Settings. This opens the settings page with our profile information, there we click on Applications and generate a New Token by writing the Token Name and clicking on Generate Token. Now you should see until you change the page the API token that is required to write the comment to the Pull Request:
Now we can save this Token in Vault and mount it via Vault-CRD to the Kubernetes cluster:
$ vault write kubernetes-secrets/gitea-credentials accesstoken=a0b9341bca7cabc5fc379cdfd2af382f44ec0b73
Success! Data written to: kubernetes-secrets/gitea-credentials
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: gitea-credentials
namespace: jenkins
spec:
path: "kubernetes-secrets/gitea-credentials"
type: "KEYVALUE"
Now Jenkins is able to mount the accesstoken during the build job and use it in our curl command. In this version of sourcecode we are using java.net.URI, maybe Jenkins disallows this, so you must allow it in the Sandbox. The link to it will be shown when the error occures inside the console.
if (env.CHANGE_BRANCH?.trim()) {
container('curl') {
def uri = new java.net.URI(env.CHANGE_URL);
def slashIndex = env.CHANGE_URL.lastIndexOf('/');
def issueIndex = env.CHANGE_URL.substring(slashIndex + 1);
def giteaApiUrl = "${uri.getScheme()}://${uri.getHost()}/api/v1/repos/${dockerImage}"
stage('Notify pull request') {
def pullRequestUrl = "${giteaApiUrl}/issues/${issueIndex}/comments?access_token=\$GITEA_TOKEN"
sh("curl -X POST \"${pullRequestUrl}\" -H 'accept: application/json' -H 'Content-Type: application/json' -d '{ \"body\": \"Successfully created environment for pull request: https://${url}\"}'")
}
}
}
When we now generate our first Pull request (maybe with this change) the application gets deployed and in the Pull Request a comment is added that shows the url to the newly generated environment:
Delete the environment after the Pull Request is closed
When we now close a Pull Request (because it gets merged or rejected) there is still the running Pull Request version of the application, but how do we delete this version then? The main problem here is, Gitea doesn’t inform Jenkins about this change, so we have no chance to delete the environment via the gitea plugin.
But there is another way, we can simply define an additional Webhook inside of Gitea that triggers each time when something happens a Webhook. Then Jenkins gets informed and deletes the environment. Therwefore we will use the Generic-Webhook-Trigger-Plugin, we already installed during our component setup in Part 2
First we create the new Jenkins job that receives the Webhook and runs the helm task to delete the running application:
def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label,
serviceAccount: 'helm',
containers: [
containerTemplate(name: 'helm', image: 'alpine/helm:2.11.0', command: 'cat', ttyEnabled: true)
]) {
node(label) {
// closed is equal to closed and also merged
if ("${env.PULL_REQUEST_ACTION}" == "closed") {
container('helm') {
def namespace = "${env.PULL_REQUEST_REPOSITORY}-${env.PULL_REQUEST_SOURCE_BRANCH}"
sh("helm del ${namespace} --purge")
}
}
}
}
And we configure a Build Trigger via the Generic Webhook Trigger:
There we define multiple relevant environment variables that the Generic Webhook Trigger can fill via the Gitea Hook request body:
- PULL_REQUEST_REPOSITORY: Contains the name of the Repository
- PULL_REQUEST_SOURCE_BRANCH: Contains the source Branch of the Pull Request
- PULL_REQUEST_ACTION: The performed action inside the Pull Request
After this we specify a Token that must be included in the Request to authenticate the Gitea Webhook request. And then we define an Optional filter to only handle Webhooks that are closing the Pull Request, both merge and also deny actions are closing events.
But how do we configure now Gitea to send the Webhook when a Pull Request gets closed (merged or rejected)? To make this programatic we can simply extend our Jenkinsfile. First we add the Token required to execute the Generic Webhook Trigger to Vault to mount it again to our Kubernetes Cluster:
$ vault write kubernetes-secrets/jenkins-credentials accesstoken=7ce550cc-13b0-4f38-a5cc-fa52a82c2c54
Success! Data written to: kubernetes-secrets/jenkins-credentials
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: jenkins-credentials
namespace: jenkins
spec:
path: "kubernetes-secrets/jenkins-credentials"
type: "KEYVALUE"
After this we can now extend our Jenkinsfile to mount this secret into the curl container:
containerTemplate(name: 'curl', image: 'tutum/curl', ttyEnabled: true, command: 'cat', envVars: [
secretEnvVar(key: 'GITEA_TOKEN', secretName: 'gitea-credentials', secretKey: 'accesstoken'),
secretEnvVar(key: 'JENKINS_TOKEN', secretName: 'jenkins-credentials', secretKey: 'accesstoken')
])
And write the logic to create the Webhook in Gitea:
stage('Create delete Webhook if not already exists') {
def jenkinsTriggerUrl = "${env.JENKINS_URL}generic-webhook-trigger/invoke"
def webhookUrl = "${giteaApiUrl}/hooks?access_token=\$GITEA_TOKEN"
def responseBody = sh(returnStdout: true, script: "curl -X GET \"${webhookUrl}\" -H 'accept: application/json'")
if (!"${responseBody}".contains("${jenkinsTriggerUrl}")) {
def addWebHookBody = """{
\\\"type\\\": \\\"gitea\\\",
\\\"config\\\": {
\\\"content_type\\\": \\\"json\\\",
\\\"url\\\": \\\"${jenkinsTriggerUrl}?token=\$JENKINS_TOKEN\\\"
},
\\\"events\\\": [
\\\"pull_request\\\"
],
\\\"active\\\": true
}"""
sh("curl -X POST \"${webhookUrl}\" -H 'accept: application/json' -H 'Content-Type: application/json' -d \"${addWebHookBody}\"")
} else {
echo "Webhook already exists"
}
}
Now each time the pipeline gets executed during a Pull request we check if there is already a Webhook available in Gitea to inform Jenkins about Pull Request events. If the Webhook is already there we do nothing, otherwise we configure Gitea to send this events and authenticate via the jenkins token we configured in the Generic Webhook Trigger.
When we now close a Pull Request because it gets Merged or Rejected a Jenkins Job gets executed to delete the Pull Request environment automatically.
Conclusion
In Parts 3-5 of the blog post series we now developed with Jenkins a Continuous Integration and Continuous Deployment pipeline that integrates very well into Gitea. It allows us to run on each new Commit a build and on each new Pull Request an automatic deployment to an ephemeral environment that gets automatically deleted when the Pull Request gets closed.
In the last Part of the series we will now refactor the pipeline to make it more general, so other applications that require the same workflow can use it as shared Library.
