Upgrade to Spark3.
In this MR we:
- Upgrade and clean conda dependencies to pickup Spark3.
- Fix the
tox
config file to only run for python 3.10. - Fix
.gitlab-ci.yml
to simplify test runs. The automated test runs were broken (probably my fault), so we fix that too.
All tests now run successfully.
Edited by Xcollazo