zea.data.chunk_cache

zea.data.chunk_cache

On-disk cache of compressed HDF5 chunks fetched over the network.

Streaming re-fetches the same bytes every time: reading the same 5 frames of an hf:// file three times costs the same ~0.9 s three times. The chunks are immutable and we already fetch at chunk granularity, so they cache exactly.

Keyed by the file’s content hash (HF hands us lfs.sha256 on open, for free) rather than its URL, because an hf:// resolve URL points at a mutable ref: re-uploading a file changes what /resolve/main/ returns while the URL stays the same. Keying on content means a re-upload simply misses, instead of silently serving stale bytes.

Compressed bytes are stored, not decoded arrays: it is the network that costs, and decoding is fast and parallel (see zea.data.chunk_reader).

Module Attributes

ENABLED	Cache is enabled unless ZEA_CHUNK_CACHE=0.
MAX_BYTES	Byte budget for the whole cache.
PRUNE_EVERY	Prune this often (in writes) rather than on every one — the check has to stat the tree.

Functions

cache_for(details[, root])	A cache for the file described by details (an fsspec info mapping).
clear([root])	Remove every cached chunk.
network_bytes()	Bytes fetched over the network by streamed hf:// reads since the process started.
prune([root, max_bytes])	Delete least-recently-used chunks until the cache fits in max_bytes.

Classes

CachedFile(fileobj, cache[, block_size])	A read-only file object that serves h5py's metadata reads from the chunk cache.
ChunkCache(content_id[, root])	Compressed chunks of one file, on disk, addressed by (offset, size).

class zea.data.chunk_cache.CachedFile(fileobj, cache, block_size=None)

Bases: object

A read-only file object that serves h5py’s metadata reads from the chunk cache.

Opening a streamed file re-reads its HDF5 metadata (superblock, group and chunk B-trees) over the network every time — and HF’s CDN intermittently stalls ~3 s on the first range request to a cold object, which is then most of the cost of an open. Those metadata bytes are as cacheable as the chunks are, so they go in the same cache.

Reads are served in aligned blocks, since h5py’s metadata reads are many and small; a block is the unit that gets stored. Chunks and blocks share one keyspace safely: a key is (offset, size) into an immutable file, so the same key always means the same bytes.

block_size defaults to the streamed file object’s own block size, so a cached open and an uncached one fetch the same ranges — a smaller block here would split each of the underlying reads into several requests.

close()

Return type:

None

property closed: bool

details

this is where the chunk fetcher reads the file’s content hash from.

Type:

Passed through

keeps_blocks

Reads keep the blocks they fetch (in the cache).

location

the chunk fetcher reuses the file’s CDN URL.

Type:

Passed through

read(length=-1)

Return type:

bytes

read_at(offset, size)

Read size bytes at offset through the cached blocks, without moving the file position. Safe to call from several threads.

Return type:

bytes

readable()

Return type:

bool

seek(offset, whence=0)

Return type:

int

seekable()

Return type:

bool

tell()

Return type:

int

writable()

Return type:

bool

class zea.data.chunk_cache.ChunkCache(content_id, root=None)

Bases: object

Compressed chunks of one file, on disk, addressed by (offset, size).

contains(offset, size)

Whether this chunk is cached, without reading it.

Return type:

bool

get(offset, size)

The cached bytes of this chunk, or None on a miss.

Return type:

bytes | None

put(offset, size, data)

Store this chunk. Failure to cache is never an error — it just costs a re-fetch.

Return type:

None

zea.data.chunk_cache.ENABLED = True

Cache is enabled unless ZEA_CHUNK_CACHE=0.

zea.data.chunk_cache.MAX_BYTES = 10737418240

Byte budget for the whole cache. Overrun is pruned oldest-first (by access time).

zea.data.chunk_cache.PRUNE_EVERY = 64

Prune this often (in writes) rather than on every one — the check has to stat the tree.

zea.data.chunk_cache.cache_for(details, root=None)

A cache for the file described by details (an fsspec info mapping).

None when caching is off, or when nothing in details identifies the content. A weaker key (the path, say) would go stale on re-upload, and a stale chunk cache is worse than no chunk cache: it corrupts reads silently.

Return type:

ChunkCache | None

zea.data.chunk_cache.clear(root=None)

Remove every cached chunk.

Return type:

None

zea.data.chunk_cache.network_bytes()

Bytes fetched over the network by streamed hf:// reads since the process started.

Take the difference of two readings to measure a single operation, e.g. to drive a progress display. Reads served from this cache are not counted.

Return type:

int

zea.data.chunk_cache.prune(root=None, max_bytes=10737418240)

Delete least-recently-used chunks until the cache fits in max_bytes.

Returns the bytes freed. Uses access time, so chunks a training run keeps re-reading survive while one-off reads age out.

Return type:

int