Introducing versioned artifact cache
In this MR we introduce a new kind of cache - a versioned cache.
This cache stores copies of artifacts in two separate directories. One directory is named by a timestamp at the time of caching, in the YYYYMMDDHHmmss format, and the other directory is named "current".
So, where a normal cache would take an artifact like this:
rootfs://artifact_root_dir
├─ artifact.file
and place it in cache like this:
cachefs://cache_root_dir/
├─ artifact.file
the versioned cache will produce a directory layout as follows:
cachefs://cache_root_dir/
├─ 20241004121212/
│ ├─ artifact.file
├─ current/
│ ├─ artifact.file
This MR also changes the way the library uses fsspec API - instead of opening a stream to write to caches, the library now uses fsspec's copy
method that can work on both files and directories.