.. _ci-fairy:

ci-fairy - a CLI tool
=====================

``ci-fairy`` is a commandline helper tool for some commonly performed tasks,
primarily those executed by the CI. ``ci-fairy`` is optimized to be run from
a CI script where its arguments will be hardcoded and change very little.
It does not do a lot of error handling, expect to see exceptions where
things go wrong.


Installation
------------

As the ``ci-fairy`` is likely used within a CI job, installation via
``pip3`` directly from git is recommended: ::

   pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates@12345deadbeef

The ``@12345deadbeef`` suffix specifies the git sha to check out.
The git sha should match the sha of the :ref:`included templates <templates_including>`.

Or where a local checkout is available: ::

   pip3 install .

Alternatively, you may directly invoke the ``ci-fairy`` tool from within the
git tree without prior installation. Note that ``pip3`` will take care of
all dependencies, for local invocations you must install those manually.


Use of the GitLab CI Environment
--------------------------------

``ci-fairy`` will make use of the `predefined environment variables set by
GitLab CI
<https://docs.gitlab.com/ee/ci/variables/predefined_variables.html>`__ where
possible. This includes but is not limited to ``CI_PROJECT_PATH``,
``CI_SERVER_URL``, and ``CI_JOB_TOKEN``.

In most cases ``ci-fairy`` does not need arguments beyond what is specific
to the interaction with the project at the time.

Authentication
--------------

Authentication is handled automatically through the ``$CI_JOB_TOKEN``
environment variable. Where the ``ci-fairy`` tool is called outside a CI
job, use the ``--authfile`` argument to provide the path to a file
containing the value of a `GitLab private token
<https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html>`__ with 'api' access.

For example, if your private token has the value ``abcd1234XYZ``,
authentication can be performed like this: ::

    $ echo "abcd1234XYZ" > /path/to/authfile
    $ ci-fairy --authfile /path/to/authfile <....>

Where ``--authfile`` is used within a CI job, specificy the token as a
`GitLab predefined environment variable
<https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables>`__
of type "File". The "Key" is the file name given to ``--authfile``, the
value is the token value of your private token.


Linting
-------

GitLab has an online linter but it requires copy/pasting the
``.gitlab-ci.yml`` file into the web form. ``ci-fairy`` makes this simpler:
::

    $ cd /path/to/project.git
    $ ci-fairy lint

``ci-fairy`` will find the ``.gitlab-ci.yml`` file and **upload it to the CI
linter** and print any errors.


Deleting registry images
------------------------

.. note:: The ``CI_JOB_TOKEN`` does not have sufficient permissions to
          delete images, ``--authfile`` is required for this task.

To delete an image from your container registry, use the ``delete-image``
subcommand. This subcommands provides three modes - deleting a specific
image tag, deleting all but an image tag or deleting all images. ::

   ci-fairy delete-image --repository fedora/30 --all
   ci-fairy delete-image --repository fedora/30 --exclude-tag 2020-03.11.0
   ci-fairy delete-image --tag "2020-03-*"

During testing, use the ``--dry-run`` argument to ensure that no tags are
actually deleted.


Templating .gitlab-ci.yml
-------------------------

The ``.gitlab-ci.yml`` file can become repetitive where multiple jobs use
the same setups (e.g. such as testing a build across different
distributions). ``ci-fairy`` provides a command to generate a
``.gitlab-ci.yml`` (or any file, really) from a `Jinja2 template
<https://jinja.palletsprojects.com>`__, sourcing the data from a YAML file.

For example, let's assume the YAML file ``.gitlab-ci/config.yml`` with a
list of distributions:

.. code-block:: yaml

    distributions:
      fedora: [32, 31]
      ubuntu: [19.04]

    packages:
      fedora:
        needed: ['gcc', 'valgrind']
        use_qemu: false
      ubuntu:
        needed: ['curl', 'wget', 'valgrind']
        use_qemu: true
      debian:
        needed: ['curl', 'wget']

And the template ``.gitlab-ci/ci.template`` that uses those to generate
jobs (see the `Jinja2 documentation <https://jinja.palletsprojects.com>`__ for details):

.. code-block::

    {% for distro in distributions %}
    {% for version in distributions[distro] %}
    .job.{{distro}}.{{version}}:
       extends: .fdo.container-build@{{distro}}
       variables:
          FDO_DISTRIBUTION_VERSION: "{{version}}"
          FDO_DISTRIBUTION_PACKAGES: "{{" ".join(packages[distro].needed)}}"
    {% endfor %}
    {% endfor %}



Now let's generate the ``.gitlab-ci.yml`` file: ::

    $ ci-fairy generate-template

and our file will look like this:

.. code-block:: yaml

    .job.fedora.32:
       extends: .fdo.container-build@fedora
       variables:
          FDO_DISTRIBUTION_VERSION: "32"
          FDO_DISTRIBUTION_PACKAGES: "gcc valgrind"

    .job.fedora.31:
       extends: .fdo.container-build@fedora
       variables:
          FDO_DISTRIBUTION_VERSION: "31"
          FDO_DISTRIBUTION_PACKAGES: "gcc valgrind"

    .job.ubuntu.19.04:
       extends: .fdo.container-build@ubuntu
       variables:
          FDO_DISTRIBUTION_VERSION: "19.04"
          FDO_DISTRIBUTION_PACKAGES: "curl wget valgrind"

.. note:: Without arguments, ``ci-fairy generate-templates`` always uses
          the files ``.gitlab-ci/config.yml`` and ``.gitlab-ci/ci.template`` and
          (over)write the file ``.gitlab-ci.yml``.

This command does not need a connection to the GitLab instance.

Templating any file
...................

While the defaults generate a ``.gitlab-ci.yml`` file, it's possible to
invoke ``ci-fairy`` with any YAML or template file: ::

    $ ci-fairy generate-template --config somefile.yml mytemplate.tmpl

``ci-fairy`` doesn't care about the template file type, you
can use any text file as template source.

To allow using the same YAML file for multiple templates, ``ci-fairy``
allows for the selection of the root node. For example: ::

    $ cat template.tpl
    qemu needed? {{use_qemu}}
    $ ci-fairy generate-template --root=/packages/fedora --config distributions.yml template.tpl
    qemu needed? true
    $ ci-fairy generate-template --root=/packages/ubuntu --config distributions.yml template.tpl
    qemu needed? false