Add cookiecutter_replay to gitignore.

See merge request gmodena/platform-airflow-dags!27

Onboard sample-project data pipeline

See merge request gmodena/platform-airflow-dags!25

Update unit test.

Updated expected output after default path
changes.

See merge request gmodena/platform-airflow-dags!24

T295360 more path normalisation

Align conventions with documentation.

See merge request gmodena/platform-airflow-dags!23

Set dag owner.

The property is required for a dag to be
picked up by the scheduler, and displayed in DAG UI.

See merge request gmodena/platform-airflow-dags!22

The property is required for a dag to be
picked up by the scheduler, and displayed in DAG UI.

Normalise path layout in factories and cookiecutter

This MR fixes some inconsistencies between dags boilerplate and the
cookiecutter template:

* the expected venv location has moved one level up in the deployed project home
* project dir (cookiecutter config) is not part of pipelines home; we let boilerplate chain dirs together.

See merge request gmodena/platform-airflow-dags!21

T295360 datapipeline scaffolding

This merge request adds a cookiecutter template to scaffold new data pipelines as described in https://phabricator.wikimedia.org/T295360.

This template provides
* Integration with our tox config (mypy/flake8/pytest)
* A PySpark job template
* A pytest template for pyspark code
* An Airflow dag template to help users getting started.

# Structure changes

The project directory largely follows `image-matching`'s strcuture. Notable changes are:
* Python code has been moved under `pyspark`
* Python code is pip installable. This allows to package deps at build time, and ease spark deployment (e.g. we don't need to pass each module like `--files schema.py` - imports will be resolved from the `venv`).

# How to test
checkout the `T295360-datapipeline-scaffolding` branch and run

 A new datapipline can be created with:                                                                                  
 ```                                                                                                                     
 make datapipeline                                                                                                       
 ```                                                                                                                     
                                                                                                                          
 This will generate a new directory for pipeline code under:                                                             
 ```bash                                                                                                                 
 your_data_pipeline                                                                                                      
 ```                                                                                                                     
                                                                                                                         
 And install an Airflow dag template under                                                                               
 ```                                                                                                                     
 dags/your_data_pipeline_dag.py                                                                                          
 ```

From the top level directory, you can now run `make test-dags`. The command will check 
that `dags/your_data_pipeline_dag.py` is a valid airflow dag. 
The output should look like this:
```
make test-dags

---------- coverage: platform linux, python 3.7.11-final-0 -----------
Name                                    Stmts   Miss  Cover
-----------------------------------------------------------
dags/factory/sequence.py                   70      3    96%
dags/ima.py                                49      5    90%
dags/similarusers-train-and-ingest.py      20      0   100%
dags/your_data_pipeline_dag.py             19      0   100%
-----------------------------------------------------------
TOTAL                                     158      8    95%

=========================== 8 passed, 8 warnings in 12.75s ===========================
______________________________________ summary ____________
```

See merge request gmodena/platform-airflow-dags!16

Install openjdk in Github action.

Conda vendored openjdk shows flaky
behaviour with the rest of the build
pipeline.

This change installs adopt openjdk
directly on the host system.

See merge request gmodena/platform-airflow-dags!18

Conda vendored openjdk shows flaky
behaviour with the rest of the build
pipeline.

This change installs adopt openjdk
directly on the host system.

Install openjdk in base image

See merge request gmodena/platform-airflow-dags!17

Modify pipeline to be compatible with algo_v2 changes

See merge request gmodena/platform-airflow-dags!13

* Modified spark scripts and sql tables to be consistent with algo_v2
changes
* Added search table hql
* Updated tests to be compatible with algo changes
* Updated ima dag to no longer use raw data file sensor and uploader

Fix tar archive generation.

See merge request gmodena/platform-airflow-dags!15

Fixes relative import issues when running code checks
in docker containers.

Fix deploy-local-build target

See merge request !14

Fix typo in README.md

See merge request gmodena/platform-airflow-dags!12

Test job failures

See merge request gmodena/platform-airflow-dags!11

Ci demo cleanup

See merge request gmodena/platform-airflow-dags!10

T292741 implement ci checks w tox

See merge request gmodena/platform-airflow-dags!9

T292741 implement ci checks

See merge request !8