Heap corruption when a compiled NumExpr object is called from several threads with NUMEXPR_NUM_THREADS=1

[kif]

Hi,
I noticed random crashes of python when parallelizing test execution in pyFAI, noticeable when tuning the number of worker and the number of threads per worker:
silx-kit/pyFAI#2948
I had an AI-agent investigating the issue and it arrived to numexpr with a fairly comprehensive bug-report with a reproducer without pyFAI or anything else than numpy. So I believe this is a valid bug:

Summary

Calling a single pre-compiled numexpr.NumExpr object concurrently from several Python threads corrupts the heap when numexpr
runs single-threaded (NUMEXPR_NUM_THREADS=1). The process dies with Segmentation fault, double free or corruption (out),
corrupted size vs. prev_size or malloc(): invalid size (unsorted), the exact symptom varying from run to run.

Setting NUMEXPR_NUM_THREADS to 2 or more makes the problem disappear entirely.

numexpr.evaluate() does not reproduce it; only a shared compiled NumExpr object does.

Reproducer

"""A single precompiled numexpr.NumExpr object shared by several Python threads."""
from multiprocessing.pool import ThreadPool
import numexpr, numpy
 
print(f"numexpr {numexpr.__version__}, nthreads={numexpr.nthreads}", flush=True)
EXPR = numexpr.NumExpr("where(norm == 0.0, dummy, signal / norm)")
 
size = 200_000
signal = numpy.random.random(size)
norm = numpy.random.random(size)
dummy = numpy.float64(0.0)
 
def work(n):
      for _ in range(300):
          EXPR(norm, dummy, signal)       # same compiled object in every thread
      return n
 
with ThreadPool(4) as pool:
      pool.map(work, range(4))
print("OK")

$ NUMEXPR_NUM_THREADS=1 python reproducer.py
numexpr 2.14.2, nthreads=1
Segmentation fault (core dumped)

$ NUMEXPR_NUM_THREADS=2 python reproducer.py
numexpr 2.14.2, nthreads=2
OK
Observed vs expected

• Observed: the interpreter dies with a corrupted heap, 100 % of the runs, usually within a couple of seconds.
• Expected: either the calls are serialised internally and the script prints OK, or the documentation states that a compiled
  NumExpr object must not be shared between threads.

Tested on python 3.13, 3.14 and 3.15, all with GIL (not tested in GIL-free):

• 2 Python threads are enough; 2, 3 and 4 threads all fail 3/3. There is no gradual degradation, which points at a race
  rather than at resource exhaustion.
• Reproduced on two Python versions and two numexpr versions, so this is not a regression of a particular release.
• Both interpreters are regular GIL builds (Py_GIL_DISABLED = 0, sys._is_gil_enabled() is True); free-threading is not
  involved.
• The machine has 503 GB of RAM, 428 GB free: this is not memory exhaustion.

Where the crash surfaces

A gdb backtrace taken at the moment of the abort shows that the faulting thread is doing a perfectly ordinary
numpy.arange():

  Thread 68 "Thread-2 (worke" received signal SIGABRT, Aborted.
  #6  malloc () from /lib/x86_64-linux-gnu/libc.so.6
  #7  default_malloc () from numpy/_core/_multiarray_umath...so
  #8  PyDataMem_UserNEW () from numpy/_core/_multiarray_umath...so
  #9  PyArray_NewFromDescr_int () from numpy/_core/_multiarray_umath...so
  #10 PyArray_ArangeObj () from numpy/_core/_multiarray_umath...so

while another thread is inside NumExpr_run:

  Thread 67 "Thread-1 (worke":
  #9  NumExpr_run(NumExprObject*, _object*, _object*) () from numexpr/interpreter...so

The heap is therefore already corrupted by the time numpy allocates; the faulty write happens earlier, which is why the
reported symptom varies.

Guess at the cause

With nthreads == 1, NumExpr_run presumably evaluates inline in the calling thread and reuses the per-object working buffers,
whereas with nthreads >= 2 the work goes through the thread pool and its per-thread buffers. Two Python threads entering
the same compiled object then write into the same scratch space. This is consistent with the threshold being exactly 2
threads, and with evaluate() being unaffected.

Workaround

Never let numexpr run with a single thread when compiled expressions are shared:

os.environ.setdefault("NUMEXPR_NUM_THREADS", "2")

Beware that numexpr falls back on OMP_NUM_THREADS when NUMEXPR_NUM_THREADS is unset, so OMP_NUM_THREADS=1 alone — a common setting in batch pipelines and in parallel test runners, which serialise each worker to avoid oversubscription — is enough to trigger this.