Skip to content

Skip content field merging from endpoint with null sink

Ebernhardson requested to merge work/ebernhardson/null-sink-skip-content into main

We are seeing throughput issues and want to check if the problem is related to the size of the UpdateEvent after the content is merged in. It seems plausible that flink is tuned for small individual events, but we are increasing their size to perhaps a hundred kb, and sometimes a few MB. By running the full pipeline but skipping the content fields we should be able to see if the size of the output events is causing issues.

If successfull expect to see the reduced or eliminated backpressure, and more consistent usage of the async http client pool when the fetch operator is reported 100% busy.

Only applies when using the null sink.

Merge request reports