Spark: Redesign API and remove session timeouts
Created by: nshahquinn
-
Breaking change: the
get_sessionandget_custom_sessionfunctions have been renamed tocreate_sessionandcreate_custom_sesson. They will now stop any existing session before creating the new session. This means that the returned session will always reflect the passed settings; previously, the settings were silently ignored if a session already existed. Use the newget_active_sessionfunction if you want to non-destructively retrieve the active session. -
Breaking change: the deprecated "raw" format and the non-deprecated
formatparameter have been removed from therunfunction. -
Breaking change: the
runfunction no longer has the ability to specify Spark settings, as thesession_typeandextra_settingsparameters have been removed. If a session already exists, it will be used. Otherwise, a default "yarn-regular" session will be created and used. - Previously, in some cases, the package automatically closed sessions after 30 minutes of apparent inactivity. It no longer does this.
- Breaking change: the
get_application_id,cancel_session_timeout,stop_session, andstart_session_timeoutfunctions have been removed. These were just intended for internal use, so it's unlikely that you will have to change code.
- Breaking change: the