ci-fairy - a CLI tool

ci-fairy is a commandline helper tool for some commonly performed tasks, primarily those executed by the CI. ci-fairy is optimized to be run from a CI script where its arguments will be hardcoded and change very little. It does not do a lot of error handling, expect to see exceptions where things go wrong.

Installation

As the ci-fairy is likely used within a CI job, installation via pip3 directly from git is recommended:

pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates@12345deadbeef

The @12345deadbeef suffix specifies the git sha to check out. The git sha should match the sha of the included templates.

Or where a local checkout is available:

pip3 install .

Alternatively, you may directly invoke the ci-fairy tool from within the git tree without prior installation. Note that pip3 will take care of all dependencies, for local invocations you must install those manually.

Use of the GitLab CI Environment

ci-fairy will make use of the predefined environment variables set by GitLab CI where possible. This includes but is not limited to CI_PROJECT_PATH, CI_SERVER_URL, and CI_JOB_TOKEN.

In most cases ci-fairy does not need arguments beyond what is specific to the interaction with the project at the time.

Authentication

Authentication is handled automatically through the $CI_JOB_TOKEN environment variable. Where the ci-fairy tool is called outside a CI job, use the --authfile argument to provide the path to a file containing the value of a GitLab private token with ‘api’ access.

For example, if your private token has the value abcd1234XYZ, authentication can be performed like this:

$ echo "abcd1234XYZ" > /path/to/authfile
$ ci-fairy --authfile /path/to/authfile <....>

Where --authfile is used within a CI job, specificy the token as a GitLab predefined environment variable of type “File”. The “Key” is the file name given to --authfile, the value is the token value of your private token.

Linting

GitLab has an online linter but it requires copy/pasting the .gitlab-ci.yml file into the web form. ci-fairy makes this simpler:

$ cd /path/to/project.git
$ ci-fairy lint

ci-fairy will find the .gitlab-ci.yml file and upload it to the CI linter and print any errors.

Deleting registry images

Note

The CI_JOB_TOKEN does not have sufficient permissions to delete images, --authfile is required for this task.

To delete an image from your container registry, use the delete-image subcommand. This subcommands provides three modes - deleting a specific image tag, deleting all but an image tag or deleting all images.

ci-fairy delete-image --repository fedora/30 --all
ci-fairy delete-image --repository fedora/30 --exclude-tag 2020-03.11.0
ci-fairy delete-image --tag "2020-03-*"

During testing, use the --dry-run argument to ensure that no tags are actually deleted.

Templating .gitlab-ci.yml

The .gitlab-ci.yml file can become repetitive where multiple jobs use the same setups (e.g. such as testing a build across different distributions). ci-fairy provides a command to generate a .gitlab-ci.yml (or any file, really) from a Jinja2 template, sourcing the data from a YAML file.

For example, let’s assume the YAML file .gitlab-ci/config.yml with a list of distributions:

distributions:
  fedora: [32, 31]
  ubuntu: [19.04]

packages:
  fedora:
    needed: ['gcc', 'valgrind']
    use_qemu: false
  ubuntu:
    needed: ['curl', 'wget', 'valgrind']
    use_qemu: true
  debian:
    needed: ['curl', 'wget']

And the template .gitlab-ci/ci.template that uses those to generate jobs (see the Jinja2 documentation for details):

{% for distro in distributions %}
{% for version in distributions[distro] %}
.job.{{distro}}.{{version}}:
   extends: .fdo.container-build@{{distro}}
   variables:
      FDO_DISTRIBUTION_VERSION: "{{version}}"
      FDO_DISTRIBUTION_PACKAGES: "{{" ".join(packages[distro].needed)}}"
{% endfor %}
{% endfor %}

Now let’s generate the .gitlab-ci.yml file:

$ ci-fairy generate-template

and our file will look like this:

.job.fedora.32:
   extends: .fdo.container-build@fedora
   variables:
      FDO_DISTRIBUTION_VERSION: "32"
      FDO_DISTRIBUTION_PACKAGES: "gcc valgrind"

.job.fedora.31:
   extends: .fdo.container-build@fedora
   variables:
      FDO_DISTRIBUTION_VERSION: "31"
      FDO_DISTRIBUTION_PACKAGES: "gcc valgrind"

.job.ubuntu.19.04:
   extends: .fdo.container-build@ubuntu
   variables:
      FDO_DISTRIBUTION_VERSION: "19.04"
      FDO_DISTRIBUTION_PACKAGES: "curl wget valgrind"

Note

Without arguments, ci-fairy generate-templates always uses the files .gitlab-ci/config.yml and .gitlab-ci/ci.template and (over)write the file .gitlab-ci.yml.

This command does not need a connection to the GitLab instance.

Templating any file

While the defaults generate a .gitlab-ci.yml file, it’s possible to invoke ci-fairy with any YAML or template file:

$ ci-fairy generate-template --config somefile.yml mytemplate.tmpl

ci-fairy doesn’t care about the template file type, you can use any text file as template source.

To allow using the same YAML file for multiple templates, ci-fairy allows for the selection of the root node. For example:

$ cat template.tpl
qemu needed? {{use_qemu}}
$ ci-fairy generate-template --root=/packages/fedora --config distributions.yml template.tpl
qemu needed? true
$ ci-fairy generate-template --root=/packages/ubuntu --config distributions.yml template.tpl
qemu needed? false