Webrequest time-series pipeline

  • pipeline to aggregate cost metrics of webrequests traffic into a time-series dataset, with configurable grouping columns and window size
  • method to create lancedb tables with vector search indices, backed by ceph s3 storage
  • transformation to run similarity search using lacedb distributed by spark
  • transformation to use push-down filters for read datasets partitioned year/month/day/hour
  • update readme for airflow dev on k8s
  • unit tests
Edited by Fabian Kaelin

Merge request reports

Loading