Compress the data in the znode to go bypass the 1MB znode data limit (!2) · Merge requests · repos / sre / Kafkakit Prometheus Metricsfetcher · GitLab

How to register an account on GitLab. To prevent spam, new accounts are locked until approved by an admin or the approver bot. You can also file an unlock request to expedite access.

Support: mw:GitLab, how to host a project on GitLab, #wikimedia-gitlab on libera.chat, #GitLab on Phabricator.

Brouberol requested to merge gzip into main Oct 06, 2023

Zookeeper has a 1MB data limit per ZNode, and the data we're writing is highly compressible. As topicmappr can already peak at the data and decompress it if necessary (https://github.com/DataDog/kafka-kit/blob/master/kafkazk/zookeeper.go#L446), we can safely gzip the broker and partition data.