Skip to content

T292741 implement ci checks

Gmodena requested to merge T292741-implement-ci-checks into multi-project-dags-repo

This PR implements a set of basic CI checks as described in T292741.

Checks

A successful CI build should pass linting and testing steps.

Linting

Run lint manually inside a docker container with make lint.

The linting step currently treats errors as warnings: the build won't stop when failures are detected.

We lint with flake8 and the following (conservative) settings:

  • McCabe complexity threshold: 10
  • maximum allowed line length: 127 (Default PEP8: 79)
  • check for syntax errors or undefined names

Tests

Run test manually, inside a docker container, with make test. Test results and coverage are reported to standard output:

---------- coverage: platform linux, python 3.7.11-final-0 -----------
Name                           Stmts   Miss  Cover
--------------------------------------------------
spark/__init__.py                  0      0   100%
spark/instances_to_filter.py      12      0   100%
spark/raw2parquet.py              21     21     0%
spark/schema.py                    7      0   100%
spark/search_table.py             22     22     0%
spark/transform.py                42     17    60%
--------------------------------------------------
TOTAL                            104     60    42%

======================== 2 passed, 9 warnings in 9.05s =========================

Automation and Github mirroing

Gitlab Pipelines are currently unavailable in Wikimedia's instance. To automate CI, for demo purposes, we propose to mirror this repo to Github, and execute tests on a Github runner

./github/workflow/build.yml implements a CI workflow that runs on every push. Output is available at https://github.com/gmodena/wmf-platform-airflow-dags/actions/workflows/build.yml?query=branch%3AT292741-implement-ci-checks.

Companion PRs

Edited by Gmodena

Merge request reports