Created by: gmodena
This WIP adds the capability to extract a python script from the notebook, and save it as build artifact in Github. I open this PR to collect feedback / discuss possible approaches to releasing the notebook code in a format consumable in non-interactive mode.
The changes add a new
py target to
make py) that runs
nbconvert on the algo notebook. The resulting script can be executed on spark with the canonical
spark-submit [...] algorithm.py.
I wanted to experiment with a flow that, after a successful build, publishes the resulting artifact to a generally available location.
I've been toying with CI and added
make py to the github workflow. With this change we'd generate a new script at each PR / git push, that is available under the Artifacts pane of the Action UI.
Unfortunately there is no straightforward way to generate the artifact URL for remote consumption.
Next I want to explore the capabilities of our internal Gitlab (pipelines are enabled), and eventually move the codebase there.