Instrument eventgate-wikimedia with OTEL (!16) · Merge requests · repos / data-engineering / Eventgate Wikimedia · GitLab

How to register an account on GitLab. Due to spam, new accounts are locked until approved by an admin or the approver bot. Your GitLab account gets automatically approved within one hour if you are a member of Trusted Contributors in Gerrit, or a member of the Trusted-Contributors group in Phabricator and linked your Developer account ("LDAP") to your Phabricator account. If none of these apply, file an unlock request to expedite access. Only if you are WMF staff: Create your GitLab account, have "Wmf" group membership in IDM, link your Developer account ("LDAP") to your Phabricator account, and get automatically approved.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Why:

We would like to propagate the tracing context across the entirety of our production job queue infrastructure.
For this to work, eventgate needs to be instrumented with OTEL so that it can inject the tracing context into the headers of Kafka job messages.

What:

Wire up auto-instrumentation via the OTEL SDK.
Wrap the Kafka publish step in a span.
Check for a new optional append_tracecontext_headers stream configuration flag and inject the tracing context into message headers if set. This will be set for job-related streams. Note that setting headers won't actually work until node-rdkafka-factory!5 (merged) is merged and the new library version is picked up.