Skip to content

flink - Add support for SSE EventSource HTTP endpoints

Ottomata requested to merge sseSourceFunction into main

This is especially useful for local development if you need a streaming source of events, but don't want to set up a Kafka cluster. This source should not be used for production jobs.

Also add 'print' SinkDescriptor support to make it possible to configure printing to stdout via the same config interface as other sinks.

This allows for a stream_manager config like:

stream_manager:
  source:
    connector: "sse"
    stream: "mediawiki.revision-create:1.1.0"
  sink:
    connector: "print"
    stream: "mediawiki.revision-create:1.1.0"

to consume from stream.wikimedia.org and print to stdout, using all the Flink internals.

Depends on https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/911962, which once merged should be released as eventutilities-flink 1.2.7.

Edited by Ottomata

Merge request reports