zea.data.hf_remote

zea.data.hf_remote

Resolve streamed hf:// files to their CDN URL once, instead of on every read.

Each request to a Hub /resolve/ URL counts toward the Hub rate limit (see https://huggingface.co/docs/hub/rate-limits). The Hub answers it with a redirect to a signed CDN URL, and requests to that CDN URL are not rate limited. Before this module, every chunk read went through /resolve/ (about 8 requests per raw-data frame), so multi-worker training quickly hit HTTP 429.

We now do what huggingface_hub.hf_hub_download() does: one HEAD request per file via huggingface_hub.get_hf_file_metadata(), then range requests directly to the returned location, without the Authorization header (the token is for the Hub, not the CDN).

• The signed URL is treated as opaque. It is reused for URL_TTL seconds and resolved again when it expires or the CDN rejects it. A branch or tag can point at new content after an upload, so opening a file by one reuses a cached URL only as long as the repo listing is cached (ZEA_HF_CACHE_TTL).

• Re-resolving uses the commit of the first resolve, so the file cannot change while open.

• Resolved URLs are cached in memory and on disk, so all threads and processes on a machine (e.g. one process per GPU) share them. The disk cache files are only readable by the user.

Module Attributes

URL_TTL	Seconds to reuse a resolved URL.
REFRESH_COOLDOWN	Seconds after a refresh that did not give a new URL (it failed, or the URL is the same, e.g. a Hub URL) during which a request rejected with that URL does not ask the Hub again.
REFUSED_STATUS	HTTP statuses the CDN returns for an expired or invalid URL.
SHARED_BLOCK_BYTES	Byte budget of the in-memory blocks shared by all open handles of the same file (see _SharedBlocks).

Functions

clear()	Clear all cached URLs (in memory and on disk) and shared file blocks.
open_remote(hf_path[, revision, repo_type, ...])	Open an hf:// file for streaming reads.
resolve(hf_path[, revision, repo_type, refresh])	Return the CDN URL of an hf:// file, from cache when possible.

Classes

RemoteFile(location, block_size[, keep_blocks])	Read-only file object over a SignedURL, passed to h5py.
Resolved(url, size, etag, commit, send_auth, ...)	The CDN URL and metadata of one hf:// file, as returned by the Hub.
SignedURL(hf_path[, revision, repo_type])	The current CDN URL of one open file.

Exceptions

FileChangedError

The content of a file changed on the Hub while it was open.

exception zea.data.hf_remote.FileChangedError

Bases: OSError

The content of a file changed on the Hub while it was open.

zea.data.hf_remote.REFRESH_COOLDOWN = 30.0

Seconds after a refresh that did not give a new URL (it failed, or the URL is the same, e.g. a Hub URL) during which a request rejected with that URL does not ask the Hub again.

zea.data.hf_remote.REFUSED_STATUS = frozenset({401, 403, 404, 410})

HTTP statuses the CDN returns for an expired or invalid URL.

class zea.data.hf_remote.RemoteFile(location, block_size, keep_blocks=True)

Bases: object

Read-only file object over a SignedURL, passed to h5py.

Reads whole blocks, so h5py’s many small metadata reads become a few requests, and keeps them in _SharedBlocks, shared with other handles of the same file. With keep_blocks=False it reads exactly the requested bytes instead. Requests go through HTTPFetcher for retries and URL refresh.

close()

Return type:

None

property closed: bool

details

fsspec-style file info. The chunk cache uses the etag as its key.

keeps_blocks

Whether reads keep the blocks they fetch, so reading one ahead pays off.

read(length=-1)

Return type:

bytes

read_at(offset, size)

Read size bytes at offset without moving the file position. Thread-safe.

Return type:

bytes

readable()

Return type:

bool

seek(offset, whence=0)

Return type:

int

seekable()

Return type:

bool

tell()

Return type:

int

writable()

Return type:

bool

class zea.data.hf_remote.Resolved(url, size, etag, commit, send_auth, resolved_at)

Bases: object

The CDN URL and metadata of one hf:// file, as returned by the Hub.

commit: str | None

etag: str | None

property fresh: bool

resolved_at: float

send_auth: bool

Send the Hub token with requests to url. Only true if url is on the Hub itself.

size: int

url: str

zea.data.hf_remote.SHARED_BLOCK_BYTES = 268435456

Byte budget of the in-memory blocks shared by all open handles of the same file (see _SharedBlocks).

class zea.data.hf_remote.SignedURL(hf_path, revision=None, repo_type='dataset')

Bases: object

The current CDN URL of one open file. Refreshed when it expires or is rejected.

Shared by the h5py file object (RemoteFile) and the chunk reader (HTTPFetcher) of the same file.

current()

Return (url, headers), refreshing the URL first if it is older than URL_TTL. May make a request, so do not call it from an event loop.

The refresh is early (signed URLs last longer than URL_TTL), so if it fails the current URL is used. If that has expired too, the CDN rejects it and refresh() is called again.

Return type:

tuple[str, dict[str, str]]

property etag: str | None

refresh(stale_url)

Resolve again after stale_url was rejected or expired. Returns whether there is a new URL.

Checks the shared cache first to prevent multiple handles of the same file sending multiple requests, and asks the Hub at most once per REFRESH_COOLDOWN for the same stale_url, so many rejected requests at once send one request.

Return type:

bool

property size: int

zea.data.hf_remote.URL_TTL = 1800

Seconds to reuse a resolved URL. The Hub’s signed URLs currently expire after about an hour. If that changes, a rejected URL is resolved again anyway.

zea.data.hf_remote.clear()

Clear all cached URLs (in memory and on disk) and shared file blocks.

Return type:

None

zea.data.hf_remote.open_remote(hf_path, revision=None, repo_type='dataset', block_size=None, cache_type=None)

Open an hf:// file for streaming reads.

cache_type="none" reads exactly the requested bytes; any other fsspec cache type reads and keeps whole blocks (see RemoteFile).

Return type:

RemoteFile

zea.data.hf_remote.resolve(hf_path, revision=None, repo_type='dataset', refresh=False)

Return the CDN URL of an hf:// file, from cache when possible.

Parameters:

• hf_path (str) – An hf://org/repo/path/to/file path to a single file.

• revision (str | None) – Branch, tag or commit hash. Defaults to the default branch.

• repo_type (str) – One of "dataset", "model" or "space".

• refresh (bool) – Skip the cache and ask the Hub again, e.g. after the CDN rejected the cached URL.

Returns:

The CDN URL, size, etag and commit of the file.

Return type:

Resolved