Skip to content

GitLab

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • P platform-airflow-dags
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar

🚧 Under Construction! Runners available in /repos; issues & wikis disabled. Support: #wikimedia-gitlab, #GitLab on Phabricator. Request project space here.

  • Gmodena
  • platform-airflow-dags
  • Merge requests
  • !5

Merged
Created Oct 27, 2021 by Gmodena@gmodenaOwner

Add mypy checks.

  • Overview 0
  • Commits 1
  • Pipelines 1
  • Changes 5

This PR integrates mypy checks for the image-matching project as as described in [SPIKE] Investigate Different CI Checks.

Checks can be triggered via make mypy, and have been added to the gitlab pipeline config.

Some (minor) fixes to error detected during the implementations of these checks are included.

What changes with this PR

ImageMatching Spark pipelines had already type annotations. This PR add some initial integration with mypy to enforce type checking at project build time. This PR contains some fixes to type errors, and false positives, discovered during the integration.

Before:

$ mypy spark
spark/transform.py:5: error: Cannot find implementation or library stub for module named "schema"
spark/transform.py:6: error: Cannot find implementation or library stub for module named "instances_to_filter"
spark/search_table.py:34: error: "Column" not callable
spark/raw2parquet.py:3: error: Cannot find implementation or library stub for module named "schema"
spark/raw2parquet.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
spark/raw2parquet.py:32: error: Argument "header" to "options" of "DataFrameReader" has incompatible type "bool";
expected "str"

After:

$ mypy spark/
Success: no issues found in 6 source files
Edited Nov 01, 2021 by Gmodena
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: T293382-add-typing