Upgrade to Spark3.

In this MR we:

• Upgrade and clean conda dependencies to pickup Spark3.
• Fix the tox config file to only run for python 3.10.
• Fix .gitlab-ci.yml to simplify test runs. The automated test runs were broken (probably my fault), so we fix that too.

All tests now run successfully.