Repository navigation

Add provider API catalogue and inference smoke-test utilities - #8

Open

sfloess wants to merge 128 commits into

mainfrom

research/provider-api-inference-smoke

Open

Add provider API catalogue and inference smoke-test utilities#8
sfloess wants to merge 128 commits into
mainfrom
research/provider-api-inference-smoke

Commits on Nov 1, 2026
- Add provider API inference smoke-test harness
  sfloess
  committed
  9a1c325
- Add provider catalogue inventory utility
  sfloess
  committed
  2561a5e
- Document provider API research utilities
  sfloess
  committed
  02da0ee
- Ignore local provider API research outputs
  sfloess
  committed
  edda044
- Require positive metadata for chat candidate classification
  sfloess
  committed
  c853eb9
- Classify incomplete reasoning responses in provider smoke tests
  sfloess
  committed
  026275a
- Filter task-specific models from chat candidate shortlist
  sfloess
  committed
  f8bfb27
- Document normalized reasoning evidence and smoke budget
  sfloess
  committed
  4984968
- Add exact model selection to provider smoke tests
  sfloess
  committed
  70db050
- Correct truncated response classification and retain smoke evidence
  sfloess
  committed
  8bf881c
- Extract response classification for deterministic tests
  sfloess
  committed
  6540bf1
- Restore normalized usage fields before classification
  sfloess
  committed
  a9526a8
- Add deterministic smoke response classification tests
  sfloess
  committed
  e4b89a4
- Document corrected classification and append-only evidence
  sfloess
  committed
  6f0efdd
- Identify provider smoke-test HTTP client explicitly
  sfloess
  committed
  631d12d
- Test explicit provider HTTP client headers
  sfloess
  committed
  ea66724
- Add DeepInfra representative inference candidate
  sfloess
  committed
  86e45f4
- Test DeepInfra endpoint configuration
  sfloess
  committed
  81b6303
- Document DeepInfra smoke-test candidate
  sfloess
  committed
  30bff47
- Add NVIDIA NIM representative inference candidate
  sfloess
  committed
  9616530
- Test NVIDIA NIM endpoint configuration
  sfloess
  committed
  123aab9
- Document NVIDIA NIM smoke-test candidate
  sfloess
  committed
  2492ee1
- Add Hugging Face Inference Providers smoke candidate
  sfloess
  committed
  77e37d2
- Add safe all-configured provider batch mode
  sfloess
  committed
  9f8d6b7
- Extract representative provider selection
  sfloess
  committed
  c394bf6
- Test all-configured batch selection
  sfloess
  committed
  f3453cc
- Document all-configured provider batch mode
  sfloess
  committed
  3da4fb8
- Add Mistral and Cohere smoke tests; refresh Gemini candidate
  sfloess
  committed
  eec6644
- Test refreshed free-tier candidates and deprecated NVIDIA endpoint
  sfloess
  committed
  bdede87
- Remove obsolete Gemini 2.5 smoke candidate
  sfloess
  committed
  4d9b0f9
- Document new free-tier providers and deprecated NVIDIA endpoint
  sfloess
  committed
  f6357e8
- Improve provider 402 diagnostics
  sfloess
  committed
  4c632d2
- Test provider billing error classification
  sfloess
  committed
  a82fe7e
- Document sanitized provider error diagnostics
  sfloess
  committed
  a9cf7e4
- Add read-only provider error inspection and free DeepSeek routes
  sfloess
  committed
  e1127b1
- Cover free DeepSeek provider routes
  sfloess
  committed
  01b854b
- Document no-cost DeepSeek routes and error inspection
  sfloess
  committed
  4506920
- Support conventional provider credential environment variables
  sfloess
  committed
  aaee23d
- Test provider credential aliases and precedence
  sfloess
  committed
  5f4125c
- Document provider credential environment aliases
  sfloess
  committed
  cb07e3d
- Support conventional credentials in catalogue inventory
  sfloess
  committed
  de573cb
- Clarify credential fallback behavior in provider research docs
  sfloess
  committed
  7f6dd92
- Add read-only credential source diagnostics
  sfloess
  committed
  11d3231
- Document read-only credential source diagnostic
  sfloess
  committed
  155064e
- Plan and report every configured provider in batch runs
  sfloess
  committed
  d249322
- Test complete all-provider batch planning
  sfloess
  committed
  6034cb8
- Document complete all-provider evaluation behavior
  sfloess
  committed
  303b5e3
- Clarify provider credential lookup documentation
  sfloess
  committed
  ef17a65
- Discover additional provider credentials from the environment
  sfloess
  committed
  454199d
- Test discovery of unlisted provider API credentials
  sfloess
  committed
  fb1ad2c
- Add Anthropic to credentialed catalogue inventory
  sfloess
  committed
  b30b364
- Report unsupported discovered providers in batch summary
  sfloess
  committed
  e71e986
- Document dynamic provider credential discovery
  sfloess
  committed
  b44e3a0
- Discover and test account-scoped provider credentials
  sfloess
  committed
  21046d8
- Test separate provider account credential discovery
  sfloess
  committed
  f826362
- Document provider account-scoped credential handling
  sfloess
  committed
  0643fa8
- Normalize Think Machines account credential aliases
  sfloess
  committed
  a81b0d3
- Test Think Machines credential namespace alias
  sfloess
  committed
  24b14e1
- Raise default smoke-test output budget to 512 tokens
  sfloess
  committed
  4f8cb13
- Test the default smoke-test output budget
  sfloess
  committed
  f8232be
- Centralize the default output token budget
  sfloess
  committed
  e8305de
- Fix output-budget regression test to use constant
  sfloess
  committed
  5405342
- Fix account identity in direct provider runs
  sfloess
  committed
  6d772b6
- Test account identity in direct provider runs
  sfloess
  committed
  47a13e7
- Add explicit OpenRouter free model account matrix
  sfloess
  committed
  b57867b
- Test OpenRouter free model matrix selection
  sfloess
  committed
  f008e49
- Classify provider rate limits and sanitize errors
  sfloess
  committed
  952dd93
- Test rate-limit classification and error sanitization
  sfloess
  committed
  da620f5
- Document rate-limit diagnostics and error sanitization
  sfloess
  committed
  ad5324e
- Fix sanitizer regression test imports and assertions
  sfloess
  committed
  763bf63
- Harden provider error metadata parsing
  sfloess
  committed
  7af45df
- Classify agentic-only OpenRouter model restrictions
  sfloess
  committed
  bf815d4
- Test agentic harness restriction classification
  sfloess
  committed
  a49be29
- Document agentic-harness-only error classification
  sfloess
  committed
  126de58
- Add provider capability and adapter catalogue
  sfloess
  committed
  c30a412
- Add unified inventory capability planning and evidence reporting
  sfloess
  committed
  d3bfe37
- Fix evidence report grouping and JSONL line endings
  sfloess
  committed
  0624d46
- Test unified provider inventory capability planning and reports
  sfloess
  committed
  5b2eade
- Correct evidence fixture JSONL separators
  sfloess
  committed
  c5d2265
- Harden unified runner option validation
  sfloess
  committed
  98cfc30
- Document unified provider runner inventory and capabilities
  sfloess
  committed
  2614505
- Normalize capability planning evidence metadata
  sfloess
  committed
  c8c3706
- Implement provider-native embedding adapters and execution
  sfloess
  committed
  43c8cba
- Register Voyage Jina and native embedding adapters
  sfloess
  committed
  7eb3b38
- Enable explicit embedding execution and safe planning mode
  sfloess
  committed
  3abb29f
- Test embedding adapters response validation and paid opt-in
  sfloess
  committed
  165580e
- Show capability-only providers as adapter available
  sfloess
  committed
  2c64215
- Account for Google model in embedding URL
  sfloess
  committed
  1e65e7d
- Report embedding-only adapters in inventory and plans
  sfloess
  committed
  4792ea7
- Document native embedding adapters and explicit execution
  sfloess
  committed
  708837b
- Expose per-capability endpoints and guard unsupported capability execution
  sfloess
  committed
  dfd26f9
- Test embedding-only provider inventory and capability plans
  sfloess
  committed
  2330770
- Use current Gemini embedContent config field
  sfloess
  committed
  64e33dc
- Assert Gemini retrieval task config uses current API field
  sfloess
  committed
  59127b8
- Link to current Mistral embeddings API reference
  sfloess
  committed
  c50f945
- Recognize Voyage AI credential names and SDK alias
  sfloess
  committed
  52778e8
- Test canonical Voyage AI credential discovery
  sfloess
  committed
  968e4aa
- Include Voyage AI in all-provider credential planning fixture
  sfloess
  committed
  1ba6eec
- Require explicit opt-in for Jina embedding account credits
  sfloess
  committed
  a0df67b
- Test Jina account-credit candidate requires explicit opt-in
  sfloess
  committed
  89317de
- Document Jina balance findings and require opt-in for retest
  sfloess
  committed
  602dac7
- Classify Jina 403 insufficient balance as billing limitation
  sfloess
  committed
  be4b654
- Test Jina 403 insufficient-balance classification
  sfloess
  committed
  6c68c16
- Add fixed semantic retrieval benchmark dataset
  sfloess
  committed
  b2a6049
- Add semantic retrieval benchmark runner
  sfloess
  committed
  048a664
- Test semantic embedding retrieval benchmark
  sfloess
  committed
  72285d2
- Support package and direct-script imports in embedding benchmark
  sfloess
  committed
  5bad962
- Correct reciprocal-rank regression fixture
  sfloess
  committed
  eff9e88
- Document semantic retrieval benchmark workflow
  sfloess
  committed
  298003e
- Support local corpus datasets and bounded embedding batches
  sfloess
  committed
  04d223f
- Test external benchmark datasets and bounded batching
  sfloess
  committed
  60b53a9
- Document real corpus benchmark dataset input
  sfloess
  committed
  283271d
- Add local private kbase corpus exporter
  sfloess
  committed
  91a7b4e
- Support separate reviewed query and relevance files
  sfloess
  committed
  d582027
- Document local kbase export and reviewed query workflow
  sfloess
  committed
  64e4f19
- Fix Markdown title parsing in kbase exporter
  sfloess
  committed
  d89e273
- Write valid newline-terminated corpus JSON
  sfloess
  committed
  746f7ce
- Align benchmark examples with separate kbase judgments
  sfloess
  committed
  a31c3f9
- Format kbase exporter and annotate loader
  sfloess
  committed
  0909099
- Fix canonical chunking checkout path resolution
  sfloess
  committed
  3c0f3bd
- Make corpus exporter independent of FlossWare chunking repo
  sfloess
  committed
  9d15209
- Test standalone corpus export baseline
  sfloess
  committed
  5385ca1
- Fix and verify exporter baseline tests
  sfloess
  committed
  bf079dd
- Clarify evaluation as independent Ensemble research
  sfloess
  committed
  114a9f9
- Document Evaluation to Ensemble research progression
  sfloess
  committed
  8aad03b
- Remove private corpus and kbase workflow from evaluation research docs
  sfloess
  committed
  8a67b8e
- Remove unnecessary private corpus exporter
  sfloess
  committed
  faf59d4
- Remove tests for deleted private corpus exporter
  sfloess
  committed
  5ef5c13

Commits

Commits on Nov 1, 2026