This repository holds the configuration of the Wikimedia Foundation Inc. Jenkins jobs. It is meant to be used with a python script written by the OpenStack Foundation: Jenkins Job Builder.
When you tweak or add jobs, follow the documentation maintained on mediawiki.org: https://www.mediawiki.org/wiki/CI/JJB
For more about the Jenkins Job Builder software and how to use it, refer to the upstream documentation: https://docs.openstack.org/infra/jenkins-job-builder/
First create a jenkins_jobs.ini file e.g.
[job_builder] allow_duplicates=True [jenkins] user=USERNAME password=API_TOKEN url=https://integration.wikimedia.org/ci/ query_plugins_info=False
Make sure you have tox and python 3 installed, you can then run Jenkins job builder using:
$ ./jenkin-jobs <arguments>
Generate from the YAML an XML file for each Jenkins job, in the output/
directory:
$ ./jenkins-jobs test ./jjb/ --config-xml -o output/
Generate from the YAML an XML file for a single Jenkins job, and output it to stdout:
$ ./jjb-test 'wip-job'
Once happy with your changes to the Jenkins Job Builder definition, commit them and review the difference:
$ ./utils/jjb-diff.sh
A CHANGELOG output will be emitted which you can copy paste to the commit message (git commit --amend
).
Update in production CI a single Jenkins job:
$ ./jenkins-jobs --conf jenkins_jobs.ini update ./jjb/ 'updated-job' (or) $ ./jjb-update 'updated-job' Note: push JJB updates to CI and verify them *before* merging the patch.
Update in production CI all Jenkins jobs matching a filter:
$ ./jenkins-jobs --conf jenkins_jobs.ini update ./jjb/ 'updated-jobs*' (or) $ ./jjb-update 'updated-jobs*' Note: push JJB updates to CI and verify them *before* merging the patch.
Delete from production CI a single Jenkins job:
$ ./jenkins-jobs --conf jenkins_jobs.ini delete ./jjb/ 'obsolete-job' (or) $ ./jjb-delete 'obsolete-job'
To test the configuration, we use tox and you need at least version 1.9+ (bug T125705) to run the test suite. Running tox
in the main dir of your local clone runs the tests.
See https://www.mediawiki.org/wiki/Continuous_integration/Allow_list
Once the change is merged, use the ./fab
helper to deploy your CI config change to production CI:
$ ./fab deploy_zuul
After making the relevant Dockerfile.template
changes and adding the changelog entry, you can locally build and test the new version of the image by running:
$ docker-pkg -c dockerfiles/config.yaml build --select '*/my-image:*' dockerfiles/
Where "my-image" is the name of the docker image. This will try to pull any parent images from the registry if missing locally, and will build them as-needed if not found.
Or, to fetch or build all images:
$ docker-pkg -c dockerfiles/config.yaml --info build dockerfiles/
Once the image is built, you can use the debug-image tool to test it interactively:
$ ./dockerfiles/debug-image my-image/
(If you cd into dockerfiles you can tab-complete on the name of docker images.)
If you are changing an image that is used by other images, you must cascade your changes so that you are the one to deal with issues, not a later user. Run:
$ docker-pkg -c dockerfiles/config.yaml --info update --reason "Reason to make this change" --version 1.2.3 updated-root-image dockerfiles/
Once the change is merged, use the ./fab
helper to build and publish your image to Wikimedia's docker registry:
$ ./fab deploy_docker
In rare cases, you may need:
$ ./fab docker_pull_image <imagename>
To spot where the JJB definitions are using outdated docker images, use:
$ ./utils/docker-updates