Add local stream config (1/3)
Bug: T326565
Add local stream config and event schemas.
This is part 1 of 3 MRs that should improve testability and coverage of eventulitilies-python
. There are
three areas to cover:
- Add basic functionality for configuring streams from a local filesystem (!15 (merged) )
- Add row to dict, dict to row converters and move flink specific funtionality to a submodule (!16 (merged) )
- Factor I/O functions out of
stream_manager
and add tests. (!17 (merged))
Changes are incremental, and so are MR diffs. The branch in MR !17 (merged) contains everything. I appreciate it's a large surface so I'd be happy to give a walkthrough.
Future work
The following work is planned once we have established a testing base to track regression. They are out of scope for T326565.
- Make Sink/Source and Flink configurable.
- Add support for schema versions.
- Add support for error topics (sideoutput).
- Align with Java eventutilities latest releases (e.g. support for nested EventRowTypeInfo).