Skip to content

artifact improvements and fixes

Ottomata requested to merge artifact_size_compare into main

Use fsspec open contextmanager

This is just better in general, but also works around a bug in arrow_hdfs.

Add file size comparison when syncing caches

This is slightly more robust than checking just for existence.

Changes:

  • size is now an abstract method on ArtifactLocator
  • size on FsArtifact{Cache,Source} uses fsspec.fs.stat to get size
  • ArtifactCache.put no longer takes a force param; if called it will always write.
  • Artifact.cache_put still respects force param, and now also uses source and cache sizes to decide if put should be called

#305868

Edited by Ottomata

Merge request reports