Python thread state design

[emmatyping]

To ensure access to Python is done soundly, we need to ensure that any code that interacts with the Python interpreter has an attached thread state (unless otherwise allowed). c.f. https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock

PyO3's Python<'py> token seems to model this pretty well, though it is not perfectly sound. I'm not sure if there is a perfectly sound design available today.

Until auto traits and negative traits allow expressing !Ungil, perhaps we can require types in CPython to unsafe impl manually to ensure that the invariants are upheld?

[davidhewitt]

PyO3/pyo3#3640 - even with Ungil auto trait the current design cannot be made fully sound.

Possible other design concerns here include the interplay with internal APIs which want the thread state passed as an argument, the PyO3 Python<'py> token is ZST.

My comment about hypothetical "contexts" in #5 (comment) is still the best idea I have about what a sound long-term solution might be (probably because it's hypothetical and I can pretend contexts have no limitations).

In general I think it'd be very interesting to present this problem to the Rust team and see what their ideas are.

[emmatyping]

In general I think it'd be very interesting to present this problem to the Rust team and see what their ideas are.

Agreed! I've added this to the agenda, I think it would be good to discuss this at the next meeting.

[encukou]

I strongly recommend passing a context argument to all functions. I'd want in this in the C API, but can't because of backwards compatibility.

For functions that need an attached thread state, this should act as an opaque "token" that "guarantees" the state is attached, but internally it should contain the thread state; sometimes that would allow public bindings to call faster internal functions. When "entering Rust land", call an explicit (and fallible) function to get the "current" context. Ideally, we'll start adding context to new C APIs as well so it can be passed more naturally.

For the unsoundness, looks like PEP 788 might help? Consider not wrapping the APIs that it deprecates.

Consider a different type of context for tp_traverse and traversal-safe functions, and potentially other "restricted" situations; ideally these (or some of these) should be callable with a "regular" context as well.