This patch is about revamping our CI pipeline for:
- our build (container images),
- linting,
- tests,
- and publishing (deb).
The main features are:
- It adds linting to the wmf_airflow_common & analytics_test folders.
- It proposes some Python code autoformatting tools (black & isort). They are also used for linting.
- It adds Dockerfile linting in CI.
- It marks some unit tests as slow and not being run locally. (HTTP get on a server)
- The pipelines are using the trusted workers provided by Releng. Then we can stop maintaining our custom Docker worker.
- It's using Kokkuri to launch buildctl to build container images.
- The built environment is cached in a container image. That makes the running time shorter for us for linting & unit tests:
- lint 6minutes => 6s
- pytest 7minutes => 50s
- It uses lib mamba to resolve Conda dependencies.
- Bump from buster to bullseye for the building image.
- Fix Python version to ~3.10.
- Moves most of the project configuration from setup.cfg to pyproject.toml.
Questions:
- Should we merge those CI pipelines changes to make it the standard in workflow-utils?
- Should we update conda-analytics CI?
Some patches follows with code updates to pass the linting: !418 (merged) & !421 (merged) . (This merge request is expected to fail at CI)
Bug: T318346
Bug: T326569
Bug: T326570