Skip to content

Template conda jobs

Gmodena requested to merge template-conda-jobs into main

This merge request adds the capability to extend build_conda_env and publish_conda_env in CI pipelines that include them.

Use case

As a developer I would like to install additional systems dependencies in the docker image used to build a conda environment. I would like to be able to extend build_conda_env and adjust it as needed.

Proposed solution

Systems deps are typically installed in a before_script step. This MR introduces the following changes.

conda.yml

A hidden .before_script has been added to conda.yml. This allows to reference and extend it in pipelines that include conda.yml and conda-dist.yml.

A global before_script is still available and exposed by default. This allows, for instance, for backward compatibility in this repo's .gitlab-ci.yml.

conda-dist.yml

Added a .build_conda_env hidden job that implements the building and packing logic. A public build_conda_env extends it to ensure consistent behaviour.

The use of both .build_conda_env and build_conda_env was necessary to avoid circular deps in downstream pipelines.

A similar approach has been implemented for .publish_conda_env and publish_conda_env. Technically we don't need it, but possibly it might be good to be consistent (convention).

Example

The CI pipeline at https://gitlab.wikimedia.org/repos/research/knowledge-gaps/-/blob/5-add-ci-integration-package-conda-env-extend-template/.gitlab-ci.yml#L66 overrides the included build_conda_env with an ad-hoc one that installs required system deps:

# Defining build_conda_env in this scope takes precedence over global,
# and allows to override the included name.
build_conda_env:
  extends: .build_conda_env
  # Override the before_script defined .build_conda_env
  before_script:
    # This reference adds the global before script declared in conda.yml
    - !reference [.setup_conda, before_script] 
    - apt update # we probably don't need this, since we apt update in .setup_conda
    #
    - apt install -y libkrb5-dev libsasl2-dev gcc g++
Edited by Gmodena

Merge request reports