Merged requested to merge T293382-add-typing into multi-project-dags-repo
This PR integrates mypy checks for the
image-matching project as as described in
[SPIKE] Investigate Different CI Checks.
Checks can be triggered via
make mypy, and have been added to the gitlab pipeline config.
Some (minor) fixes to error detected during the implementations of these checks are included.
What changes with this PR
ImageMatching Spark pipelines had already type annotations. This PR add some initial integration with mypy to enforce type checking at project build time. This PR contains some fixes to type errors, and false positives, discovered during the integration.
$ mypy spark spark/transform.py:5: error: Cannot find implementation or library stub for module named "schema" spark/transform.py:6: error: Cannot find implementation or library stub for module named "instances_to_filter" spark/search_table.py:34: error: "Column" not callable spark/raw2parquet.py:3: error: Cannot find implementation or library stub for module named "schema" spark/raw2parquet.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports spark/raw2parquet.py:32: error: Argument "header" to "options" of "DataFrameReader" has incompatible type "bool"; expected "str"
$ mypy spark/ Success: no issues found in 6 source files