Customized OpenShift Pipelines with IBM Garage Cloud Native Toolkit
The sample OpenShift Pipelines that come out of the box with Cloud Pak for Applications (CP4Apps) is a good starting point to get started with DevOps. When you need more integrations into your pipelines you will need to create your own pipelines extending the CP4Apps, or if you just want to have more control over the promotion of the application thru different stages like test, staging, production using GitOps. The environment is configured with the IBM Garage Cloud Native Toolkit with tools to support the custom pipelines.
- Extend, Build & Deploy Kabanero Pipelines
- How to add our custom pipelines to your cluster
- How to use artifactory-package-release-update pipeline
- How to use git-package-release-update pipeline
- Create tekton webhook
Developers that use Kabanero pipelines often times have to extend these pipelines to do certain tasks that do not come in the out-of-the-box Kabanero pipelines. These tasks may include code coverage, or use third party applications like Pact Broker, Sonarqube or Artifactory to full-fill software requirements. Currently, there are not many methods to manage and version control your Kabanero pipelines, and the goal of this repository is to help you get going.
You will learn how to add our custom pipelines to your dedicated cluster. In addition, not required you will also learn how to manage and version control your kabanero pipelines.
You will learn how to package, host your pipelines in different environments such as Git or Artifactory and use these pipelines to automate the process of updating the Kabanero custom resource to a respective host where your Kabanero pipelines exist.
We built our pipelines on top of out of box kabanero java-spring-boot2-build-deploy-pl for Spring boot and nodejs-express-build-deploy-pl for nodejs pipeline. Below are the additional tasks we have in our pipelines at the moment.
- Testing: This task runs all the unit tests required for the application.
- Sonar Scan: This task performs sonar scan which helps to detect the code smells, vulnerabilities, bugs etc in your application code.
- Pact Contract Testing: This task performs contract testing. It shares consumer driven contracts and verifies the results using Pact.
- Health checks: This task checks the roll out status of your deployment once it is deployed to the cluster and verifies the application health.
- GitOps: This task will update the the deployment manifests residing in gitops-dev repository.
This repository includes a set of custom pipelines
|artifactory-package-release-update||Compress custom pipelines, upload compressed pipelines to artifactory, and updates the Kabanero Custom Resource|
|clone-storefront-ms-push-repos-to-org||Given a github org name, clone storefront microservices and deploy them to the github org|
|git-package-release-update||Compress custom pipelines, create a github release, upload compressed pipelines to the release, and update the Kabanero Custom Resource|
|storefront-nodejs||This custom pipeline is built on top of nodejs-build-deploy-pl (out of box kabanero pipeline for nodejs applications). It contains additional tasks such as test, sonar-scan, pact-broker, health checks, and gitops tasks|
|storefront-springboot||This custom pipeline is built on top of java-spring-boot2-build-deploy-pl (out of box kabanero pipeline for java spring boot applications). It contains additional tasks such as test, sonar-scan, pact-broker, health checks, and gitops tasks|
Install the following CLI’s on your laptop/workstation:
Openshift 4.3.5 with CloudPak for Apps
- Tools Kit created by the Catalyst Team
You can add the custom pipelines to your cluster we created by following the steps below.
Login to your cluster and change to the kabanero namespaceoc login --token=your-token --server=your-serveroc project kabanero
Download our latest release by visiting https://github.com/ibm-garage-ref-storefront/pipelines-server/releases
default-kabanero-pipelines.tar.gzfile to download the zip file onto your Downloads.
Then run the following command to obtain the checksum value. You will use this checksum value later.> shasum -a 256 default-kabanero-pipelines.tar.gz4c05cb2d593af43a3e4c6818ada1afc4e550e3c6bc08bbbf486eb2ab6ce37274 default-kabanero-pipelines.tar.gz
Edit your Kabanero Custom Resource and add our pipelines> oc edit kabanero -o yaml
Add the following:stacks:pipelines:- https:url: the-url-to-download-the-zip-fileid: pipeline-managersha256: the-checksum-value-you-generated
For example we currently are on release v38.0 and our Kabanero custom resource looks like the following:stacks:pipelines:- https:url: https://github.com/kabanero-io/kabanero-pipelines/releases/download/0.6.1/default-kabanero-pipelines.tar.gzid: defaultsha256: 64aee2805d36127c2f1e0e5f0fc6fdae5cef19360c1bb506137584f3bd0988cc- https:url: https://github.com/ibm-garage-ref-storefront/pipelines-server/releases/download/39.0/default-kabanero-pipelines.tar.gzid: pipeline-manager
Save your changesesc:wq
You might get an error such as
Unable to save custom resource...you just need to update the
resourceVersionkey with the latest time stamp.
Verify the custom pipelines have been added to your cluster as shown below:> tkn pipelines list...artifactory-package-release-update-pl 1 week ago --- --- --- ---git-package-release-update-pl 1 week ago git-package-release-update-pl-run-8fpcq --- --- ---clone-storefront-ms-push-repos-to-org-pl 1 week ago --- --- --- ---storefront-nodejs-pl 1 week ago storefront-nodejs-pl-run-7gjbm --- --- ---storefront-springboot-pl 1 week ago storefront-springboot-pl-run-glsxs --- --- ---
You are now ready to use our custom pipelines.
You can use this pipeline to package, release your pipelines onto Artifactory. This pipeline is for developers that are extending Kabanero pipelines to create custom tasks or pipelines and need a solution to version control your pipelines.
Fork the devops-pipelines repository
Deploy Artifactory on your Openshift cluster
Generate an API Key.
Update Artifactory config map artifactory-config.yaml and update the
artifactory_key. Once done, run the following commands:oc project kabanerocd ./configmapsoc apply -f artifactory-config.yaml
Go to the pipelines directory make any modifications you want to do to any of the pipelines, or include your own.
Create your pipeline by running the following command:cd pipelines/experimentaloc apply --recursive --filename pipelines/experminetal/artifactory-package-release-update/
Go to the dashboard and verify that the
artifactory-package-release-update-plhas been added to the Tekton dashboard
Go to section Create tekton webhook to create your web hook.
Go to your forked repository and make a change, and your Tekton dashboard should create a new pipeline run as shown below: Where the
git-sourceis defined as the pipeline resource with key [url] and value [github repo url]
The end result should look like the following:
You can use a pipeline to automate the process of extending, packaging and releasing your pipelines via a Git Release. The process is very similar to the section above.
- Fork this repository devops-pipelines
Add your custom pipelines or modify an existing one If you inspect
./pipelines/you can create a new folder for each new pipeline you have and follow a similar structure as below.echo pwd./devops-pipelines/pipelines├── experimental│ ├── README.md│ ├── abc│ │ ├── bindings│ │ │ ├── abc-pl-pullrequest-binding.yaml│ │ │ └── abc-pl-push-binding.yaml│ │ ├── configmaps
Now drag and drop your pipelines and tasks to any of these folders,
You must update the
secretwe provided for you. But first, create another repository such as
devops-server. In this repo
devops-serveryou will be hosting your pipelines as Git releases. Do not forget to create a README.md file.
pipelines/stable/git-package-release-update/configmapsand update the
pipeline-server-configmap.yamlapiVersion: v1kind: ConfigMapmetadata:name: pipeline-server-configmapnamespace: kabanerodata:repo_org: your-github-username-or-orgrepo_name: your-github-repo-where-you-will-host-pipelinesimage_registry_publish: 'false'
Update the secret in
pipelines/stable/git-package-release-update/secrets/apiVersion: v1kind: Secretmetadata:name: pipeline-server-gitnamespace: kabanerotype: kubernetes.io/basic-authdata:password: your-git-token-encodedusername: your-git-username-encoded
Now run the following command to be able to retrieve resources for the
kabanero-pipelineservice account.oc adm policy add-cluster-role-to-user view system:serviceaccount:kabanero:kabanero-pipeline
Create web hook for the devops-pipelines repository you created on step 1.
Deploy your pipeline, tasks, event bindings and trigger templates by running the following command in the devops-pipelines repo you created on step 1:oc apply --recursive --filename pipelines/stable/git-package-release-updategit add .git commit -m "adding new pipelines..."git push
Your output should be the following:
If you go to the pipelines-server repo you created on step 2, you should see a new release with your zip files as shown below:
Now inspect your Kabanero Custom Resource to ensure your
default-kabanero-pipelines.tar.gzgot added to the
pipelineskey value pair.oc get kabanero -o yamlstacks:pipelines:- https:url: https://github.com/ibm-garage-ref-storefront/pipelines-server/releases/download/1.0/default-kabanero-pipelines.tar.gzid: pipeline-managersha256: 8fe10018016e5059640b1a790afe2d6a1ff6c4f54bf3e7e4fa3fc0f82bb2207d
The pipelines that you added to the pipelines-server repository should now be visible on the tekton dashboard as shown below:
Now you can reuse these pipelines across your organization! If your cluster comes down you now have a backup of your pipelines.
You need to create an access token on the tekton dashboard or cli in the kabanero namespace. Earlier you created a github token on the github dashboard. You will need to get that token or generate another one and paste it below.
Web hook Settings:Name: devops-demo-kabanero-pipelinesRepistory-url: your forked repo url goes hereAccess Token: Token you generated previously
Target Pipeline SettingsNamespace: kabaneroPipeline: Choose artifactory-package-release-update-pl or git-package-release-update-plService Account: PipelineDocker Registry: us.icr.io/project-name or docker.hub.io/projectname