Allow spark_kwargs to be passed to all table maintenance tasks.
In this MR we:
- Modify the table maintenance mechanism to allow passing
spark_kwargs
andspark_conf
to all table maintenance tasks. - We bump default resources for all maintenance tasks, as they are prone to fail with the cluster's default minimal resources.
- We override default resources for the case of data deletion for table
wmf_dumps.wikitext_inconsistent_rows_rc1
.
Bug: T378122
Edited by Xcollazo