Skip to content

local_pipeline: issue concurrent parallel requests.

Gmodena requested to merge enrich-with-minibatch into main

Use microbatching and a thread pool local to a process function operator to perform concurrent http requests.

This implementation uses threads to perform I/O bound async callbacks.

This MR implements an experimental Process function, that does not yet integrate the error reporting topic.

Changes have been tested on YARN, but need a new release to be tested in DSE k8s. We might revert once tests have completed.

Bug: https://phabricator.wikimedia.org/T332948

Edited by Gmodena

Merge request reports