Skip to content

Fix ivy issue on mediawiki dumps, and some cosmetic fixes.

Xcollazo requested to merge fix-ivy-issue into main

This MR solves an issue found in prod with Ivy package pulling in which file permissions made the job fail sporadically:

Exception in thread "main" java.io.FileNotFoundException: /tmp/ivy_spark3/cache/resolved-org.apache.spark-spark-submit-parent-c7a250eb-dd57-4233-a8a1-02f29c30ad5d-1.0.xml (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:71)
	at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:63)
	at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:553)
	at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:184)
	at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:259)
	at org.apache.ivy.Ivy.resolve(Ivy.java:522)
	at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1454)

Additionally, we make some cosmetic fixes to dumps_merge_visibility_events_to_wikitext_raw_dag.py.

Bug: T344709 Bug: T340861

Edited by Xcollazo

Merge request reports