gh-158445: Allocate memory in the heap in Py_GetVersion()

[vstinner]

Py_GetVersion() now allocates memory on the heap, instead of using a static buffer, to no longer truncate the version if it's longer than 299 bytes.

Update Py_GetCompiler() and Py_GetBuildInfo() tests: they are now always a part of sys.version.

• Issue: C API: Detect invalid UCS4 characters in debug mode #158445

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34907310 | 📁 Comparing ef0a101 against main (be87a85)

  🔍 Preview build  

2 files changed

± c-api/interp-lifecycle.html
± whatsnew/changelog.html

[vstinner]

On my Fedora 44, __clang_version__ is short: 22.1.8 (Fedora 22.1.8-4.fc44). But in issue gh-157368, way longer version has been seen: 22.1.3 (https://github.com/llvm/llvm-project e9846648fd6183ee6d8cbdb4502213fcf902a211). On Windows, PC/pyconfig.h no longer uses __clang_version__. But we may such long clang version on other platforms.

[itamaro]

hey @vstinner, we ran into a side effect of moving the buffer to the heap - some debuggers and core-dump tools find the interpreter version by reading the Py_GetVersion() buffer out of process memory (py-spy for example). With only the 20-byte static_version left, they lose the tag and the "free-threading build" marker. Getting those back means following heap_version into the heap, which is fragile and not always possible.

can we make the static buffer bigger, at least enough to hold the version, tag, and free-threading marker (~64 bytes)? let me know if I should file a separate issue

[vstinner]

py-spy:

    if let Some(&addr) = python_info.get_symbol("Py_GetVersion.version") {
        info!("Getting version from symbol address");
        if let Ok(bytes) = process.copy(addr as usize, 128) {
            if let Ok(version) = Version::scan_bytes(&bytes) {
                return Ok(version);
            }
        }
    }

I understand that in Python 3.15, py-spy copies 128 bytes from the static char version[300]; variable.

Would it be possible for py-spy to (1) call Py_GetVersion() function and (2) copy the return string? I suppose that it's better to not call functions in a debugger to leave the process state unchanged.

some debuggers and core-dump tools find the interpreter version by reading the Py_GetVersion() buffer out of process memory (py-spy for example)

I didn't think about these use cases. That's not how I expected the Python C API to be used. Well, they just inspect memory, they don't even call functions, if I understand correctly.

@itamaro:

can we make the static buffer bigger, at least enough to hold the version, tag, and free-threading marker (~64 bytes)? let me know if I should file a separate issue

Would you recommend to revert my change and add a comment explaining the debugger/crash reporter use cases?

Or just restore the static buffer to its Python 3.15 size (300 bytes)? I suppose that the static buffer name should be restored to version, so debuggers/crash reporters don't have to update their code.

Thanks for rising the issue!

[vstinner]

Or there is another approach, build strings at build time, to avoid the complex code building strings at runtime: #158951.

[itamaro]

I understand that in Python 3.15, py-spy copies 128 bytes from the static char version[300]; variable.

Would it be possible for py-spy to (1) call Py_GetVersion() function and (2) copy the return string? I suppose that it's better to not call functions in a debugger to leave the process state unchanged.

yeah, I think not calling function in the debugged process is the point. for core-dump tools, there isn't even a process around.

I didn't think about these use cases. That's not how I expected the Python C API to be used. Well, they just inspect memory, they don't even call functions, if I understand correctly.

sounds right. like you say, they're inspecting memories, not "using" the C API per se.

Would you recommend to revert my change and add a comment explaining the debugger/crash reporter use cases?

Or just restore the static buffer to its Python 3.15 size (300 bytes)? I suppose that the static buffer name should be restored to version, so debuggers/crash reporters don't have to update their code.

I don't think we need to revert, just restoring the version static buffer should be sufficient!

thanks for looking into this! I didn't look closely at your gh-158951, it looks like a bigger change than just restoring the static buffer, so if you think that's the better way to go, I may be able to test it in the next few days, or maybe next week.