wikidata: index RDF graph with qlever

Add three new tasks to hydrate a local mounted volumed on dse-k8s, run qlever-index in an ephemeral pod, and copy the index binary files into the data pipeline s3 bucket for distribution to the wdqs backend service. A medata index file is generated that includes a timestamp indicating the cut-off time for the data contained in the dumps/index. Since we control when the dump process starts, we use the DAG start timestamp instead of the dateModified value in the metadata file. A latest.json manifest needs to be generated in the index S3 bucket, with a pointer (prefix) the wikidata snapshot (datestamp) of the dag run. Kubernetes infrastructure Under the hood we use a PV with node affinity to two high mem/storage hosts in the k8s cluster.

Bug: T428235

Merge request reports

Loading