Make it easier to traverse the frame stack for third party tools.

[markshannon]

Profilers and debuggers need to traverse the frame stack, but the layout of the stack is an internal implementation detail.
However can make some limited promises to make porting tools between Python versions a bit easier.

In order to traverse the stack, the offset of the previous pointer needs to be known. To understand the frame, more information is needed.

@pablogsal
@Yhg1s
expressed interest in this.

Linked PRs

• GH-100987: Refactor _PyInterpreterFrame a bit, to assist generator improvement. #100988
• GH-100987: Don't cache references to the names and consts array in _PyEval_EvalFrameDefault. #102640
• GH-100987: Allow non python frames in frame stack. #103010
• GH-100987: Allow objects other than code objects as the "executable" of an internal frame. #105727

[markshannon]

Initially, I propose to refactor the PyInterpreterFrame struct such that it starts:

typedef struct _PyInterpreterFrame {
    PyCodeObject *f_code;
    struct _PyInterpreterFrame *previous;
    ...

Currently f_code must be a code object, but we could generalize it to allow other objects.
For example, the shim frame inserted on entry to _PyEval_EvalFrameDefault could have that field set to None indicating it should be skipped in tracebacks, etc.

The order of f_code and previous doesn't really matter, but have f_code first makes #100719 a bit simpler

[pablogsal]

Let me collect some feedback from maintainers of debuggers and profilers and will comment here the requirements so we can think of solutions.

[markshannon]

@pablogsal Any feedback?

[markshannon]

We can further improve traversal of the _PyInterpreterFrame for debugging and introspection by allowing C extensions to create frames without the rigmarole of creating a code object.

We should rename the f_code field to f_executable, and allow any object.

typedef struct _PyVMFrame {
    PyObject *f_executable;
    struct _PyVMFrame *previous;
} PyVMFrame;

Although tools and the VM should tolerate any object, we should in practice only allow a few classes:

• CodeObject: Implies that the PyVMFrame is a full _PyInterpreterFrame. Only the VM should make this kind of frame
• Builtin function, method descriptor, slot wrapper, etc. The frame represents a call to the given object.
• None: Internal shim. Tools should skip this frame.
• Tuple: First three items should be name, filename, flags where flags determine the meaning of additional entries.

The tuple form is for tools like Cython, Nanobind, etc. Creating a tuple of strs and ints is much simpler and faster than creating a fake code object.

C extension can link themselves into the frame stack at the cost of about 4 memory writes, and 3 reads:

    PyVMFrame frame;
    frame.previous = tstate->current_frame.frame;
    frame.f_executable = &EXECUTABLE_OBJECT;
    tstate->current_frame.frame = &frame;
    /* body of function goes here */
    tstate->current_frame.frame = frame.previous;

We can do this for builtins functions by modifying the vectorcall function assigned to the builtin function/method descriptor.
We would need to benchmark this to see the performance impact, but it will be much cheaper than sys.activate_stack_trampoline()

[pablogsal]

@pablogsal Any feedback?

I have reached out again to tool authors, give me a couple of days to gather comments. Apologies for the delay

[markshannon]

No problem.

[itamarst]

@benfred -^