Skip to content

Automate testing with tox.

Gmodena requested to merge 5-add-ci-integration into main

This WIP merge request addresses point 1 of issue #5 (closed): Run unit tests.

It provides a basic skeleton for automating code checks. It's based on the cookiecutter template that Platform uses to bootstrap data pipelines.

Gitlab's CI Pipeline status can be found at https://gitlab.wikimedia.org/repos/research/knowledge-gaps/-/pipelines

Note: Marking as draft because I want to start a conversation to validate direction. This change is not prescriptive, and feedback is very much welcome.

I adopted the same type of code checks we use in https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines, but I relaxed the strictness a bit.

A CI Pipeline is triggered after each push and merge request and will run the following checks:

  1. Unit tests (stored under tests/).
  2. Linting, with flake8, runs against the knowledge_gaps module (rules are defined in the tox.ini file)
  3. Type checking, with mypy, runs against the knowledge_gaps module (rules are defined in tox.ini file)

Only unit tests are required to pass to mark a pipeline as successful, flake8/mypy are allowed to fail (but will raise a warning).

Changes

  • add the capability to automate pytest / mypy / flake8 via tox.
  • added a Gitlab CI pipeline config (.gitlab-ci.yml).
  • unit tests have been moved into a toplevel tests/ dir.

Known issues

  • Tests are failing. But I'd rather address a fix in a dedicate merge request.
  • some code under interactive/ is not valid Python. There's some notebook magic (%%load_ext) that will raise a syntax error when parsed by mypy/flake8. For know I'd be keen in leaving interactive code alone, under the assumption the it will be refactored. @aikochou @bmansurov does this track with you? How do you plan to structure this code base?
Edited by Gmodena

Merge request reports