Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos

[MaxGhenis]

Fixes #529.

Summary

download_huggingface_dataset decided whether to send a token by testing ModelInfo.private only. policyengine/policyengine-uk-data-private has been public but gated (private=False, gated="manual") since 31 July 2026, so the helper passed token=None to hf_hub_download and the gate returned 401 GatedRepoError even when a gate-approved HUGGING_FACE_TOKEN was in the environment. The Hub only falls back to its own HF_TOKEN variable when no explicit token is passed, which is why PolicyEngine/policyengine-uk#1817 had to export both names as a workaround (see the comment in .github/workflows/code_changes.yaml and pr_code_changes.yaml there).

Behaviour change

A repo now requires authentication when private is true or gated is truthy. ModelInfo.gated is False for ungated repos, "auto" or "manual" for gated repos, and None when the field is absent, so a truthiness test covers every state. In that case the existing get_or_prompt_hf_token() supplies the token from HUGGING_FACE_TOKEN (or an interactive prompt on a TTY).

Unchanged:

• Public, ungated repos still download with token=None and never prompt.
• In a non-interactive environment without secrets, the token is still None rather than an empty string (the Handle empty HUGGING_FACE_TOKEN gracefully #422 behaviour), so Dependabot-style CI keeps working for public repos.
• The RepositoryNotFoundError path still treats the repo as private.
• On a TTY with HUGGING_FACE_TOKEN unset, a gated repo now prompts for a token exactly as a private repo always has. Pressing Enter passes token=None, and huggingface_hub then applies its own implicit token (HF_TOKEN or the login file) if one is configured.

Verified live: GET /api/models/policyengine/policyengine-uk-data-private returns private: false, gated: "manual", and an unauthenticated HEAD on a resolve URL returns 401.

Tests

New tests in tests/core/tools/test_hugging_face.py:

• test_download_gated_public_repo_passes_env_token[manual|auto] mocks model_info returning private=False, gated=... and asserts the token read from HUGGING_FACE_TOKEN by the real get_or_prompt_hf_token reaches hf_hub_download. Both fail on master and pass here.
• test_download_private_flag_repo_passes_env_token covers private=True returned by model_info (the existing private test only covers the 404 path).
• test_download_gated_repo_non_interactive_without_token asserts CI without secrets gets token=None and no prompt.
• test_download_public_ungated_repo_never_prompts[False|None] guards against widening the predicate into a prompt on every public download.

Checks run locally:

• uvx ruff format --check . and uvx ruff check . pass; ruff check --select F821 is clean for the changed files, and mypy on hugging_face.py drops from four errors to the one pre-existing version: str = None error at line 49.
• uv run pytest tests: 702 passed, 4 skipped, 1 xfailed.
• policyengine-core test policyengine_core/country_template/tests -c policyengine_core.country_template: 39 passed.

Not run: make documentation (no docs changed; CI runs it).

Related

• Issue: download_huggingface_dataset sends no token for public+gated repos (private is the wrong predicate) #529
• Incident in policyengine-uk: CI HUGGING_FACE_TOKEN failing with 401 on private dataset downloads policyengine-uk#1816
• Workaround that this makes unnecessary: Send HF_TOKEN so gated dataset downloads authenticate policyengine-uk#1817
• Earlier token handling: Handle empty HUGGING_FACE_TOKEN gracefully #422

🤖 Generated with Claude Code