Use a relative tolerance in is_constant

[avehtari]

This PR was assisted by Claude

Summary

loo has its own is_constant() (R/helpers.R) with an absolute tolerance: abs(max(x) - min(x)) < .Machine$double.eps. Values of small magnitude are then treated as constant, so the result depends on the units of x. This PR makes the tolerance relative to the largest absolute value in x, as stan-dev/posterior#469 did for posterior::is_constant().

Problem

E_loo() uses is_constant() in the function-specific Pareto-k (.E_loo_khat_i()). When x is treated as constant, pareto_k is the k-hat of the ratios alone, and the k-hat of h * r is not computed. For small-magnitude x, pareto_k therefore changes with the scale of x:

library(loo)
LLmat <- example_loglik_matrix()
chain_id <- rep(1:2, each = nrow(LLmat) / 2)
psis_mat <- psis(-LLmat, r_eff = relative_eff(exp(LLmat), chain_id), cores = 1)
set.seed(123)
x <- matrix(rnorm(length(LLmat)), nrow = nrow(LLmat))
k <- E_loo(x, psis_mat, type = "mean", log_ratios = -LLmat)$pareto_k
k_s <- E_loo(1e-20 * x, psis_mat, type = "mean", log_ratios = -LLmat)$pareto_k
sum(abs(k_s - k) > 1e-10)   # master: 19 of 32 differ (up to 0.335); this PR (with posterior#469): 0

Change

• is_constant() in R/helpers.R: x is constant if max(x) - min(x) <= tol * max(abs(x)), or if all values are equal (also all 0 or all -Inf). The code is the same as in posterior#469.
• An infinite spread, for example c(-Inf, 2), is not constant, the same as on master. x with NA gives NA, the same as on master.
• Other behavior change: for values of magnitude above 1, values that differ by about 1 ulp are now constant, for example c(1, 1 + .Machine$double.eps).

E_loo() also calls posterior::pareto_khat() on h * r. With a posterior without stan-dev/posterior#469, that call returns NA for small-magnitude values, so E_loo() needs both changes to give a pareto_k that does not depend on the scale of x.

The other constant check in loo, the tail check in do_psis_i() (R/psis.R), works on log ratios shifted by their maximum. Log ratios do not change when the weights are scaled, so that check is not changed.

Tests

• tests/testthat/test_0_helpers.R: is_constant uses a tolerance relative to the magnitude of x: zeros, small constant values, -Inf, 1-ulp differences, small values that differ, infinite spread, NA.
• tests/testthat/test_E_loo.R: E_loo pareto_k does not depend on the scale of x, for s = 1e-20 and 1e-200. The test is skipped when the installed posterior has the absolute tolerance (posterior 1.7.1 on CRAN), because posterior::pareto_khat() then returns NA for h * r.

With posterior master (which has #469), the new tests give 6 failed expectations without this change and pass with it. With posterior 1.7.1 from CRAN, the is_constant() tests pass and the E_loo() test is skipped.

Tested versions

• loo: this branch, based on master fd910fef.
• posterior: 1.7.1 (CRAN), and master 48243f8.
• R 4.6.1, Linux x86_64.

Copyright and Licensing

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

• Code: MIT (https://opensource.org/licenses/MIT)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.48%. Comparing base (fd910fe) to head (34d70fd).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #407   +/-   ##
=======================================
  Coverage   92.48%   92.48%           
=======================================
  Files          31       31           
  Lines        3139     3141    +2     
=======================================
+ Hits         2903     2905    +2     
  Misses        236      236           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 34d70fd is merged into master:

• ✔️loo_function: 1.07s -> 1.08s [-0.8%, +2.1%]
• ✔️loo_matrix: 1.25s -> 1.25s [-1.32%, +1.12%]
  Further explanation regarding interpretation and methodology can be found in the documentation.