Analysis Framework

This repository evaluates the LEET-Arg benchmark dataset, which contains statement-level legal reasoning questions, expert rationales, model responses, and evaluator scores.

Repository Contents

• run_benchmark.py: Runs the LEET-Arg questions against a Hugging Face causal language model and stores parsed responses in results/.
• prompts.py: Defines the English reasoning prompt used by the benchmark.
• model_configs/: Per-model inference settings.
• benchmarks/LEET_Arg_Questions_cleaned_and_rationale_by_statement.json: The current cleaned LEET-Arg question set: 93 questions and 301 statement units.
• benchmarks/LEET_Arg_Model_Responses.json: Responses and LLM-as-a-Judge evaluations for seven models across the LEET-Arg questions.
• tools/clean_leet_arg.py: Rebuilds statements for known segmentation issues in a source dataset.
• tools/validate_leet_arg.py: Validates statement structure, source consistency, punctuation, and suspicious content.
• docs/: Research notes and evaluation reports.

Setup

First-time setup

1. Install Python 3.10 or newer and create a virtual environment:

python3 -m venv .venv
source .venv/bin/activate

2. Install the runtime dependencies:

pip install torch transformers accelerate pyvene transformer-lens pydantic

3. Install the Hugging Face command-line tools. The hf command is included with the Hugging Face Hub package:

pip install -U huggingface_hub
hf --help

4. Create or sign in to a Hugging Face account, then authenticate locally:

hf auth login

When prompted, paste a Hugging Face access token with permission to read models. Keep the token out of source files and shell history where possible.

The repository supports Linux and macOS. Use the activation command appropriate for your shell, for example source .venv/bin/activate.fish for Fish.

Troubleshooting Hugging Face imports

This benchmark uses text-only models. If Transformers reports Could not import module 'LlamaConfig' and the traceback ends with torchvision::nms, remove the unused incompatible vision package from the active environment:

pip uninstall -y torchvision

If you want to clean up models in your disk since we are testing one model at a time, use the hf CLI. For listing all saved models in the cache, use hf cache list and hf rm <model> to remove any of them.

Downloading models from Hugging Face

The default model is meta-llama/Llama-3.2-1B-Instruct. It is gated, so first visit the model page, accept its license, and wait for access to be approved. Then download it into a local directory:

hf download meta-llama/Llama-3.2-1B-Instruct \
  --local-dir models/Llama-3.2-1B-Instruct

Public models can be downloaded in the same way. For example:

hf download microsoft/Phi-4-mini-instruct \
  --local-dir models/Phi-4-mini-instruct

You can either let Transformers download and cache a model automatically by passing its Hugging Face ID:

python run_benchmark.py --model meta-llama/Llama-3.2-1B-Instruct

or use a previously downloaded local directory:

python run_benchmark.py --model models/Llama-3.2-1B-Instruct

For local directories, the model configuration lookup falls back to the default Llama settings unless a matching entry exists in model_configs/. Copy and adapt an existing file there if the model requires different dtype, trust, seed, or generation settings. Model downloads can be several gigabytes, so ensure sufficient disk space and use a GPU or other supported accelerator when available.

LEET-Arg Benchmark

Running questions with Ollama

On your Chameleon GPU instance, install Ollama for Linux using its official installer and check that the NVIDIA GPU is available:

nvidia-smi
curl -fsSL https://ollama.com/install.sh | sh

Interrupt the process if it stalls after creating the user. Start the Ollama server in a terminal and leave it running:

ollama serve

In a second terminal on the same instance, download the model and run the script from the repository root:

ollama pull llama3.2:3b
python3 run_ollama_questions.py

If the installer has already started Ollama as a system service, you can use that server instead of starting a second ollama serve process.

The script reads the cleaned LEET-Arg JSON file by default, prepends the requested answer-format prompt to every question, and submits them to Ollama one at a time. It limits each response to 3,000 generated tokens and writes a Markdown file named for each question ID in testing/baseline/. Each file includes the original question, the answer choice and its supplied explanation, and the model response. Existing files with matching IDs are replaced when rerunning the script.

Use --input, --output-dir, --model, or --url to override the defaults. For example, to test one question from a different JSON file, specify that file and use a temporary output directory:

python3 run_ollama_questions.py \
  --input benchmarks/LEET_Arg_Questions_cleaned_and_rationale_by_statement.json \
  --output-dir /tmp/ollama-question-test

When the Python script runs on a different machine from Ollama, use an SSH tunnel rather than exposing Ollama's unauthenticated API port publicly. For example, with SSH local port forwarding from your workstation, leave the script's default --url http://localhost:11434.

Test setup

For each benchmark question, the model receives only:

1. The base system prompt defined in prompts.py.
2. The individual LEET-Arg question from benchmarks/LEET_Arg_Questions_cleaned_and_rationale_by_statement.json.

The model is not given the dataset answer, the expert rationale, other model responses, or demonstrations from other questions. The generated response is parsed for its answer and saved with the question in results/.

Run all questions with Llama 3.2 1B Instruct:

python3 run_benchmark.py

Run only questions whose IDs start with 2021_:

python3 run_benchmark.py --model meta-llama/Llama-3.2-1B-Instruct --year 2021

Results are written to results/<model-signature>.json. Existing results are retained and new responses are appended by default. Use --overwrite to clear that model's result file before running:

python3 run_benchmark.py --model microsoft/Phi-4-mini-instruct --year 2021 --overwrite

Each stored record retains the source question and includes model_answer and model_rationale, parsed from the model's Answer-<choice>. response. The output terminal will not display the rationale for readability purposes, storing it directly in the file.

Testing

See issue #24 for details on testing, in particular, the comment on the testing procedure.

LEET-Arg Dataset

The cleaned question file contains 93 questions from 2021-2025 and 301 statement-level tasks. The response file contains entries for the same question IDs and includes these models:

claude_opus4, claude_sonnet4, deepseek_r1, gemini-2.5-pro, o3, o3-pro, and o4-mini.

Validate the current cleaned dataset:

python tools/validate_leet_arg.py \
  --input benchmarks/LEET_Arg_Questions_cleaned.json

The validator checks sequential statement keys, missing statements, suspiciously short or leaked instruction text, punctuation, and relaxed source consistency. The source-cleaning utility is intended for rebuilding a cleaned file from an original dataset:

python tools/clean_leet_arg.py \
  --input path/to/LEET_Arg_Questions.json \
  --output benchmarks/LEET_Arg_Questions_cleaned.json

Evaluate saved model responses and create the comparison CSV:

python3 results/evaluate_results.py

Limitations

The LEET-Arg files contain model-generated answers and evaluator judgments, which should be treated as benchmark data rather than verified legal advice. Model availability, hardware, Hugging Face permissions, and model-specific settings can affect reproducibility.

See docs/ for LEET-Arg research context and evaluation notes.