This merge request adds the capability to extend build_conda_env and publish_conda_env
in CI pipelines that include them.
Use case
As a developer I would like to install additional systems dependencies in the docker image
used to build a conda environment. I would like to be able to extend build_conda_env and
adjust it as needed.
Proposed solution
Systems deps are typically installed in a before_script step. This MR introduces the following changes.
conda.yml
A hidden .before_script has been added to conda.yml. This allows to reference and extend it in pipelines that include conda.yml and conda-dist.yml.
A global before_script is still available and exposed by default. This allows, for instance, for backward compatibility in this repo's .gitlab-ci.yml.
conda-dist.yml
Added a .build_conda_env hidden job that implements the building and packing logic.
A public build_conda_env extends it to ensure consistent behaviour.
The use of both .build_conda_env and build_conda_env was necessary to avoid circular deps in downstream pipelines.
A similar approach has been implemented for .publish_conda_env and publish_conda_env. Technically
we don't need it, but possibly it might be good to be consistent (convention).
Example
The CI pipeline at https://gitlab.wikimedia.org/repos/research/knowledge-gaps/-/blob/5-add-ci-integration-package-conda-env-extend-template/.gitlab-ci.yml#L66 overrides the included
build_conda_env with an ad-hoc one that installs required system deps:
# Defining build_conda_env in this scope takes precedence over global,
# and allows to override the included name.
build_conda_env:
extends: .build_conda_env
# Override the before_script defined .build_conda_env
before_script:
# This reference adds the global before script declared in conda.yml
- !reference [.setup_conda, before_script]
- apt update # we probably don't need this, since we apt update in .setup_conda
#
- apt install -y libkrb5-dev libsasl2-dev gcc g++