Skip to content

Add schema version to stream_manager.

Gmodena requested to merge T327866-add-stream-schema-version into main

Bug T327866

Add schema version to stream_manager source/destination targets, and to EventDataStreamFactory.

get_json_schema() now returns the correct version, instead of always fetching latest.

Versioning

This change breaks compatibility. Clients should be updated.

Api changes

  • stream_manager requires an explicit version for both source and destination streams.
  • EventDataStreamFactory factory methods allow for 'latest'.
  • An EventStream data class has been added to the flink module.

EventStreamDescriptor

EventStreamDescriptor data class has been added to the flink module. The idea is to encapsulate stream name and version in the same boilerplate object. This used internally in EventDataStreamFactory factory methods to avoid method arguments proliferation. And EventStreamDescriptor can be instantiated by passing name/version to the constructor as EventStreamDescriptor(name="some.stream", version="1.1.0") or by parsing a string EventStreamDescriptor.from_string("some.stream:1.1.0").

Note that while EventStream.from_string provides basic string parsing, naming and versioning validation is delegated to the Java library.

For more details see docstring and unit tests under tests/test_flink.py.

Related MRs:

cc / @otto @tchin @dcausse

Edited by Gmodena

Merge request reports