Warn when a restricted Hugging Face download resolves no HUGGING_FACE_TOKEN

[MaxGhenis]

Summary

Diagnostic follow-up to #529 (the fix for gated repos itself is #538). download_huggingface_dataset now emits a UserWarning when the repo requires authentication (model_info reports private=True, or raises RepositoryNotFoundError) and get_or_prompt_hf_token() returned None. The warning names the repo, says no HUGGING_FACE_TOKEN was available, that huggingface_hub's own cached token (HF_TOKEN or the hf auth login file) will be used if present, and that a 401 that follows (RepositoryNotFoundError or GatedRepoError) means neither was set or the token is not approved for the repo.

Behaviour is otherwise unchanged, deliberately (per #422): token=None is still passed through to hf_hub_download, there is no raise, no prompt, and core does not read HF_TOKEN. The fallback is huggingface_hub's own: in 1.4.1, utils/_headers.py::get_token_to_send calls get_token() when token is None, and utils/_http.py::hf_raise_for_status maps a 401 to GatedRepoError (X-Error-Code: GatedRepo) or RepositoryNotFoundError.

The warning uses stacklevel=2, so it points at the caller (for example Dataset.download_from_huggingface), and it is not emitted for public repos, which also pass token=None. The module's import-time warnings.catch_warnings() block is a no-op and is not relied on.

Tests

• New TestNoTokenWarning in tests/core/tools/test_hugging_face.py: the warning fires for private-flag and not-found lookups, across token unset, token empty, and only-HF_TOKEN-set environments non-interactively, and for an empty interactive prompt, with token=None passed on and no prompt; no warning for public repos or when a token was resolved from the environment or a prompt; message content and stacklevel are asserted.
• Mutation check: with the source change reverted, the 7 warning-asserting tests fail and the 5 no-warning guards pass.
• uv run pytest tests/core/tools/test_hugging_face.py: 27 passed. uv run pytest tests/core: 708 passed, 1 skipped, 1 xfailed. uvx ruff format --check . and uvx ruff check .: clean.
• Not run: make test (full suite with coverage and reruns) and make documentation; CI covers them and no docs pages change.

Overlap with #538 and #539

Both are still open, so this branch is from master (9125de0). Checked locally by merging each into this branch:

• Fix vacuous test for private Hugging Face repo download without a token #539 (parametrised test_download_private_repo_no_token): merges cleanly, 29 passed. Its three cases now emit this warning and tolerate it (there is no filterwarnings = error setting in the repo). TestNoTokenWarning asserts the warning for the same three environments, so Fix vacuous test for private Hugging Face repo download without a token #539's test does not need to.
• Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos #538 (gated repos): one textual conflict, in the if is_repo_private: block. Resolution: keep Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos #538's requires_authentication predicate and nest the warning under it. With that resolution, 33 passed.

    authentication_token: str | None = None
    if requires_authentication:
        authentication_token = get_or_prompt_hf_token()
        if authentication_token is None:
            warnings.warn(...)  # body unchanged from this PR

Documentation review

• Documentation changes: a Warns: section in the download_huggingface_dataset docstring, and the towncrier fragment changelog.d/warn-hf-no-token.changed.md. No docs/ page describes this function's token handling, so nothing else needed updating.
• Impact: low (a warning, no control-flow change). Confidence: high (huggingface_hub behaviour read from the installed 1.4.1 source; both lookup paths are tested).
• Known gaps: none here. After Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos #538 lands, the docstring's "requires authentication" wording already covers gated repos.

🤖 Generated with Claude Code