Update: Migrate to Spark3
Several of our notebooks (Both notebooks here ) and python script (SPC Corpus Generation ), utilize Spark2
, which is no longer supported [source].
Need to create an MR to perform the migration to Spark3
.
Goal:
- Modify each affected scripts/notebooks as per new syntax and conventions