Template conda jobs
This merge request adds the capability to extend build_conda_env
and publish_conda_env
in CI pipelines that include them.
Use case
As a developer I would like to install additional systems dependencies in the docker image
used to build a conda environment. I would like to be able to extend build_conda_env
and
adjust it as needed.
Proposed solution
Systems deps are typically installed in a before_script
step. This MR introduces the following changes.
conda.yml
A hidden .before_script
has been added to conda.yml
. This allows to reference and extend it in pipelines that include conda.yml
and conda-dist.yml
.
A global before_script
is still available and exposed by default. This allows, for instance, for backward compatibility in this repo's .gitlab-ci.yml
.
conda-dist.yml
Added a .build_conda_env
hidden job that implements the building and packing logic.
A public build_conda_env
extends it to ensure consistent behaviour.
The use of both .build_conda_env
and build_conda_env
was necessary to avoid circular deps in downstream pipelines.
A similar approach has been implemented for .publish_conda_env
and publish_conda_env
. Technically
we don't need it, but possibly it might be good to be consistent (convention).
Example
The CI pipeline at https://gitlab.wikimedia.org/repos/research/knowledge-gaps/-/blob/5-add-ci-integration-package-conda-env-extend-template/.gitlab-ci.yml#L66 overrides the included
build_conda_env
with an ad-hoc one that installs required system deps:
# Defining build_conda_env in this scope takes precedence over global,
# and allows to override the included name.
build_conda_env:
extends: .build_conda_env
# Override the before_script defined .build_conda_env
before_script:
# This reference adds the global before script declared in conda.yml
- !reference [.setup_conda, before_script]
- apt update # we probably don't need this, since we apt update in .setup_conda
#
- apt install -y libkrb5-dev libsasl2-dev gcc g++