The source project of this merge request has been removed.
Start jupyter with conda vendored spark.
This MR streamlines our spark3 install and init.
Changed the jupyter notebook startup script to use the Spark distro installed in the tumult conda environment.
Since SPARK_HOME points to a directory in PYTHONPATH,
we keep relevant config files in the user owned conf/
directory.
@htriedman I did some poor man's manual integration testing in stat1006 and these changes work for me. Somehow, this time, I'm unable to tag you as reviewer in Gitlab.