Skip to content
GitLab

Consume events from page-change.

Merged Gmodena requested to merge T311084-consume-page-change-stream into main

@dcausse @otto @tchin @lbowmaker

Closes: T311084

Refactor the Mediawiki Stream Enrichment job to consume events from the consolidated page-change stream. The following change have been introduced:

  1. Replace schemas defined by cases classes with a Row object.
  2. Integrates WMF boilerplate for stream configuration.
  3. Simplifies the Async enrichment function logic.

An entry point to review code could be enrichment/src/test/scala/EnrichmentSuite.scala.

Dependencies on pending work

  1. events should be produced in the page-change stream
  2. wikimedia-event-utilities 1.2 should be released with https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/810387 merged
  3. Page change schema has been finalised
  4. Enriched page change schema has been finalized.

TODO:

  1. Implement a checkpointing and offset management strategy,
Edited by Gmodena