Add extract_embeddings cli

DCausserequested to merge
T414070-embeddings into main

A job that:

  • parses enterprise structured content
  • extract passages
  • fix section names using cirrus dumps
  • extract embeddings using spark-nlp
  • attempt to re-use pre-computed embeddings from past partition

Bug: T414070

Edited by DCausse

Merge request reports