Skip to content

Start jupyter with conda vendored spark.

Gmodena requested to merge (removed):setup-conda-pyspark into main

This MR streamlines our spark3 install and init.

Changed the jupyter notebook startup script to use the Spark distro installed in the tumult conda environment.

Since SPARK_HOME points to a directory in PYTHONPATH, we keep relevant config files in the user owned conf/ directory.

@htriedman I did some poor man's manual integration testing in stat1006 and these changes work for me. Somehow, this time, I'm unable to tag you as reviewer in Gitlab.

Edited by Gmodena

Merge request reports