Convert _ctypes static types (and the "meta class") to heap types

[vstinner]

Feature or enhancement

The _ctypes extension implements types as static types. They should be converted to heap types: see issue gh-103092 for the rationale.

Problem: the PyCStgDict type is kind of special, it's a "metaclass" which creates a new class and then replaces the class namespace (type dict) with a custom "stgdict". See the PyCStructType_new() function.

Previous attempts to convert _ctypes types to heap types:

• April 2023: PR gh-103092: Adapt _ctypes's PyCStgDict type to heap type #103932
• January 2024: PR gh-103092: Port some _ctypes data types to heap types #113630

Linked PRs

• gh-114314: Move types to ctypes_state #114316
• gh-114314: Avoid possibly invalid args when initializing a ctypes meta type #114740
• gh-114314: ctypes: remove stgdict and switch to heap types #116458

[vstinner]

Example with _ctypes.Structure:

$ ./python
>>> import _ctypes
>>> type(_ctypes.Structure)
<class '_ctypes.PyCStructType'>
>>> _ctypes.Structure.__bases__
(<class '_ctypes._CData'>,)
>>> _ctypes.Structure.__base__
<class '_ctypes._CData'>

In Python, such type would be defined with a syntax like:

class Structure(_CData, metaclass=PyCStructType):
    ...

When a subclass is created, PyCStructType_new() is called. Example of subclass:

class MyStruct(_ctypes.Structure):
   ...

In C, PyCStructType_Type can be retrieved from global_state.PyCStructType_Type global variable.

(gdb) p global_state.PyCStructType_Type.tp_new
$12 = (newfunc) 0x7fffea18739f <PyCStructType_new>

(gdb) p global_state.PyCStructType_Type.tp_init
$13 = (initproc) 0x59321c <type_init>

(gdb) p global_state.PyCStructType_Type.tp_name
$14 = 0x7fffea57a230 "_ctypes.PyCStructType"

Main steps of PyCStructType_new():

• Create a new type by calling PyType_Type.tp_new(type, args, kwds) where type is PyCStructType_Type.

• Type members are initialized to:

  • tp_name = "MyStruct"
  • tp_new = GenericPyCData_new() -- inherited from _ctypes.Structure.tp_new
  • tp_init = Struct_init() -- inherited from _ctypes.Structure.tp_init

• Override tp_dict with an instance of ctypes PyCStgDict_Type (which inherits from dict).

Example showing that tp_dict type is PyCStgDict_Type instead of dict:

>>> import _ctypes
>>> import _testcapi
>>> class MyStruct(_ctypes.Structure): pass
... 
>>> d=_testcapi.type_get_tp_dict(MyStruct)  # I added this function locally just for this example
>>> type(d)
<class 'StgDict'>

[vstinner]

First step: merged PR #113857

[vstinner]

See merged PR #113620.

[encukou]

It seems that stgdict is a workaround for the issue PEP-697 is designed to solve: have additional C data on a type.
If no one else is interested in this, I can try my hand at replacing the extra stgdict fields by PyObject_GetTypeData(o, _ctypes.PyCStructType).

[erlend-aasland]

It seems that stgdict is a workaround for the issue PEP-697 is designed to solve: have additional C data on a type. If no one else is interested in this, I can try my hand at replacing the extra stgdict fields by PyObject_GetTypeData(o, _ctypes.PyCStructType).

I agree; I already mentioned this in one of my (now closed) PRs. I think PEP-697 would be perfect for this.

[neonene]

replacing the extra stgdict fields by PyObject_GetTypeData(o, _ctypes.PyCStructType).

@encukou Could you be more specific, please? I posted a PR #114740, which seems to have become a different attempt.

[neonene]

@vstinner "PyCStgDict" words in the title and description are typos?

PyCStgDict is a dictionary subclass rather than a metaclass, I think.