Skip to content

Add mypy checks.

Gmodena requested to merge T293382-add-typing into multi-project-dags-repo

This PR integrates mypy checks for the image-matching project as as described in [SPIKE] Investigate Different CI Checks.

Checks can be triggered via make mypy, and have been added to the gitlab pipeline config.

Some (minor) fixes to error detected during the implementations of these checks are included.

What changes with this PR

ImageMatching Spark pipelines had already type annotations. This PR add some initial integration with mypy to enforce type checking at project build time. This PR contains some fixes to type errors, and false positives, discovered during the integration.

Before:

$ mypy spark
spark/transform.py:5: error: Cannot find implementation or library stub for module named "schema"
spark/transform.py:6: error: Cannot find implementation or library stub for module named "instances_to_filter"
spark/search_table.py:34: error: "Column" not callable
spark/raw2parquet.py:3: error: Cannot find implementation or library stub for module named "schema"
spark/raw2parquet.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
spark/raw2parquet.py:32: error: Argument "header" to "options" of "DataFrameReader" has incompatible type "bool";
expected "str"

After:

$ mypy spark/
Success: no issues found in 6 source files
Edited by Gmodena

Merge request reports