Page_view: Adds wiki_id from sitematrix.

This MR aims to add the field wiki_id derived from the data we can get from the sitematrix endpoint.

To avoid doing HTTP calls and having I/O waiting during this pipeline, the sitematrix is only called once per task at start up.

We don't expect many changes in the sitematrix, but they could happen. To be able to refresh the lookup table without breaking the application, we will refresh the lookup table after finding some misses.

There's no async process or threads involved in refreshing the lookup table, as we don't expect to refresh it often, I thought that would introduce a lot of code that won't be used many times. This is working in sync, so when the lookup table is being refreshed, the worker will have a small delay (1 ~ 2 seconds) that block the pipeline.

To avoid running this refresh many times, we have a threshold and a cooldown. After the threshold is reached, it will try to refresh the table, unless we are still in the colddown time, so we avoid doing too many requests in case of a failure and having thousands of misses for any reason.

In summary:

  • Table is built at startup
  • Table is built when a Task Manager starts (after OOM, K8s allocating resources or whatever)
  • Table is only refresh if:
    • Number of missing wiki_ids is bigger than threshold
    • AND
    • Colddown time is passed

We are also producing a metric about the number of "wiki_id" misses so we can monitor how often they happen.

Bug: T425624 Bug: T427925

Edited by JavierMonton

Merge request reports

Loading