Init catalog tables with a kafka watermark.

Merged Gmodena requested to merge gmodena/flink-wmf-event-catalog:add-kafka-watermak into main

Bug: This MR initialises TableDescriptors with Kafka watermarks.

The change adds a kafka_timestamp virtual column to schemas accessed through the catalog. kafka_timestamp is a watermark field with a delay of 10 seconds. Both settings are eventutilities defaults. We might want to tune this in the future (or expose a property to let users tune column name and delay) but for now they seem a sensible starting point.

kafka_timestamp carries "event time" Flink semantics (rowtime) as described in

An example of tubmle window query on a watermaked table created with this patch can be found at:

Note that while the virtual column can be projected from the schema, it will not be persisted with INSERT statements, thus not affecting existing sink functionality. See for details.

Merge request reports