Webrequest time-series pipeline
- pipeline to aggregate cost metrics of webrequests traffic into a time-series dataset, with configurable grouping columns and window size
- method to create lancedb tables with vector search indices, backed by ceph s3 storage
- transformation to run similarity search using lacedb distributed by spark
- transformation to use push-down filters for read datasets partitioned year/month/day/hour
- update readme for airflow dev on k8s
- unit tests
Edited by Fabian Kaelin