Skip to content

CI pipeline revamp

Aqu requested to merge T318346_add_linting_in_ci into main

This patch is about revamping our CI pipeline for:

  • our build (container images),
  • linting,
  • tests,
  • and publishing (deb).

The main features are:

  • It adds linting to the wmf_airflow_common & analytics_test folders.
  • It proposes some Python code autoformatting tools (black & isort). They are also used for linting.
  • It adds Dockerfile linting in CI.
  • It marks some unit tests as slow and not being run locally. (HTTP get on a server)
  • The pipelines are using the trusted workers provided by Releng. Then we can stop maintaining our custom Docker worker.
  • It's using Kokkuri to launch buildctl to build container images.
  • The built environment is cached in a container image. That makes the running time shorter for us for linting & unit tests:
    • lint 6minutes => 6s
    • pytest 7minutes => 50s
  • It uses lib mamba to resolve Conda dependencies.
  • Bump from buster to bullseye for the building image.
  • Fix Python version to ~3.10.
  • Moves most of the project configuration from setup.cfg to pyproject.toml.

Questions:

  • Should we merge those CI pipelines changes to make it the standard in workflow-utils?
  • Should we update conda-analytics CI?

Some patches follows with code updates to pass the linting: !418 (merged) & !421 (merged) . (This merge request is expected to fail at CI)

Bug: T318346

Bug: T326569

Bug: T326570

Edited by Aqu

Merge request reports