Skip to content

Source data should be read from files.

Gmodena requested to merge T326731-fix-ds-from-data into main

Bug: T326731

Fixes a test failure detected in T326731:

FAILED test_pipeline.py::test_enrich_from_data - py4j.protocol.Py4JError: An error occurred while calling z:java.net.URI.create. Trace:
...
py4j.protocol.Py4JError: An error occurred while calling z:java.net.URI.create. Trace:
E                   org.apache.flink.api.python.shaded.py4j.Py4JException: Method create([class java.util.HashMap]) does not exist

This was caused by the wrong input being passed to fileDataStream.

When streaming input with from_data(), the payload is serialised to a temporary file. This file should be read be input to a subsequent call to from_files().

Here the source_data instance variable is used to store either an object (from_data input), or a list of files (from_files input).

cc / @otto @tchin

Merge request reports