Skip to content

Introducing versioned artifact cache

Aleksandar Mastilovic requested to merge feature/Add-Support-For-Versioned-Caches into main

In this MR we introduce a new kind of cache - a versioned cache.

This cache stores copies of artifacts in two separate directories. One directory is named by a timestamp at the time of caching, in the YYYYMMDDHHmmss format, and the other directory is named "current".

So, where a normal cache would take an artifact like this:

rootfs://artifact_root_dir
├─ artifact.file

and place it in cache like this:

cachefs://cache_root_dir/
├─ artifact.file

the versioned cache will produce a directory layout as follows:

cachefs://cache_root_dir/
├─ 20241004121212/
│  ├─ artifact.file
├─ current/
│  ├─ artifact.file

This MR also changes the way the library uses fsspec API - instead of opening a stream to write to caches, the library now uses fsspec's copy method that can work on both files and directories.

Merge request reports