Skip to content

Add local stream config (1/3)

Gmodena requested to merge T326565-add-testing-base into main

Bug: T326565

Add local stream config and event schemas.

This is part 1 of 3 MRs that should improve testability and coverage of eventulitilies-python. There are three areas to cover:

  1. Add basic functionality for configuring streams from a local filesystem (!15 (merged) )
  2. Add row to dict, dict to row converters and move flink specific funtionality to a submodule (!16 (merged) )
  3. Factor I/O functions out of stream_manager and add tests. (!17 (merged))

Changes are incremental, and so are MR diffs. The branch in MR !17 (merged) contains everything. I appreciate it's a large surface so I'd be happy to give a walkthrough.

Future work

The following work is planned once we have established a testing base to track regression. They are out of scope for T326565.

  1. Make Sink/Source and Flink configurable.
  2. Add support for schema versions.
  3. Add support for error topics (sideoutput).
  4. Align with Java eventutilities latest releases (e.g. support for nested EventRowTypeInfo).

cc / @otto @tchin @dcausse

Edited by Gmodena

Merge request reports