cuda.bindings: support multiple CTK release lines on main

[rwgk]

Summary

Closes #1199. This continues Keith Kraus's original PR #2675, preserving his CUDA 12 source import and initial build, test, and release integration. The replacement became necessary when that PR's temporary base branch was deleted after #2467 merged.

Maintain both supported CUDA ABI-major source trees on main, with short release branches available for stabilization and urgent fixes. Release each bindings line independently, together with its matching cuda-python metapackage.

Package root	Role	Build/test toolkit
cuda_bindings_12/	Maintenance	12.9.1
cuda_bindings/	Current	13.4.2

The roots build the same cuda-bindings distribution and cuda.bindings namespace in separate environments. The CUDA 12 import is synchronized through v12.9.9. After merge, 12.9.x is retained as historical release evidence.

Decisions Implemented

1. Source layout and scope: keep the physical ABI-major root cuda_bindings_12, with lifecycle roles represented separately in the registry. Support exactly one current and one maintenance root with distinct majors. The mapping identifies package paths; it does not promise arbitrary numbers of public release lines.
2. Release branches: integrate shared development on main; cut short stabilization branches from a tested commit when needed. Urgent fixes can start on the frozen branch, with a linked follow-up PR to reconcile the fix back into main. A new manual backport workflow accepts one merged PR and one existing non-default target branch, opens a reviewable PR, and leaves CI approval and merging to the normal process. The historical 12.9.x branch is not resumed.
3. Major versus patch maintenance: major-line maintenance can include explicitly selected API, Python, or platform compatibility work. A frozen patch release admits its approved fixes and release prerequisites. The updated release checklist and new bindings release guide require recording that scope in the checklist and release notes.
4. Independent releases: a bindings tag selects one bindings root and its metapackage, with published Pathfinder and no Core release. Ordinary development CI still tests dependent Core variants. A frozen tested branch provides a release source when unrelated development on main is unfinished; it does not waive required validation. — For details, see this comment below.
5. SCM and packaging: Package versions continue to use standard setuptools-scm, with each bindings line selecting its own tag family. A temporary fallback handles CUDA 12 builds before an eligible tag is reachable. The standalone metapackage carries matching build metadata, and pre-commit/CI checks prevent those copies from drifting. See the SCM and packaging review guide below.
6. Historical releases: tagged registries are authoritative; retain a bounded pre-registry compatibility resolver. New releases require notes in the tagged source. Legacy releases can recover exact-version notes from the matching control-tree root; historical metapackages without separate notes can use matching bindings notes with an explicit warning. Empty notes and missing matching notes fail.
7. Cross-root and generator review: authors and reviewers must assess handwritten changes in both roots. No general byte-equality test establishes semantic applicability. Generated imports record cybind revision, toolkit inputs, commands, and manual adjustments. File seals verify bytes, not generation provenance. Continuous toolkit QA and a broader cybind redesign remain separate work.
8. Test scope: removed five redundant snapshot/static-agreement tests, retained behavioral and negative release/artifact cases, and added concrete metadata-drift, historical-notes, and reachable-tag regressions. The full CI-tool suite now runs in PR CI.

---

SCM and packaging review guide

Click to un/expand

Reviewer question	Intended behavior	Where to inspect
Which tags determine a package's version?	Each root selects its minor family; setuptools-scm handles parsing and version progression. CUDA 12 excludes the old v12.9.0 baseline.	CUDA 13 configuration, CUDA 12 configuration
What happens before an eligible CUDA 12 tag is reachable?	CI uses the configured development fallback plus the commit identifier. Once any matching tag is reachable, the override stops, including on descendants of prerelease or postrelease tags.	The decision function
Why repeat metadata in the metapackage?	Its standalone source distribution lacks the registry and sibling package files, so it carries its own selectors and fallback.	Metapackage setup
What prevents those values from diverging?	A validator checks the registry, package selectors, metapackage metadata, and maintenance Pixi version for agreement.	Consistency validator

The bootstrap regression cases cover no eligible tag, the excluded old baseline, a matching tag and its descendants, and an unreachable tag.

The key review question is: Does the custom fallback reliably get out of the way once standard Git-derived versioning can work?

---

The durable operating procedure is RELEASE-bindings.md, with shared ownership and rollover guidance in ci/README.md. The contributor CI overview now uses a maintained conceptual Mermaid diagram.

Implementation and Review Guide

• Imports CUDA 12 source, tests, examples, docs, packaging, and Pixi lockfile. The existing coredump lifetime fix remains an intentional handwritten overlay.
• Routes dependency-aware CI, wheels, sdists, tests, docs, and releases through the selected root; uses in-tree PEP 517 backends for independent source builds.
• Validates complete release artifact matrices, excludes test wheels, preserves release suffixes, and requires exact-tag/exact-SHA successful CI for publishing.
• Keeps release source and workflow control revisions distinct. A current control registry cannot replace a modern release's tagged registry.
• Removes branch-sourced artifacts and the old automatic 12.9.x backport policy. Adds explicit manual automation for selected stabilization branches.
• Removes the temporary .lycheeignore. The three new canonical main/cuda_bindings_12 URLs must be rechecked after merge.

The six-layer review aid #2960 remains useful for the large import and integration. Its final tree matches the earlier head 59ff7bd88756bda02164daefb0a71daff47b1d2e; it predates the September 29 completion commits. Read those additional commits here for the final policy, metadata/versioning safeguards, and historical-note fixes. Intermediate review-aid layers are not independently deployable PRs. #2737's final tree is authoritative.

Validation of the Final Candidate

Final head: 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38. Its only delta from b00b677be366fd3638c78e26b99aa5b238602646 is four authored documentation files: two explicit section targets, one repaired link, and an orphan marker for the release-note template. A focused production-version Sphinx/MyST render reproduces three warnings before the patch and zero afterward.

• Local CI tools: 191 passed, 55 subtests passed. Full pre-commit passed, with lychee checked separately: 800 successful checks and only the three expected pre-merge canonical-main 404s. Newly added and changed documentation links pass.

• CUDA environment checks report cuda-13-4.conf; no global toolkit switch was made. Native implementation code is unchanged in these completion commits.

• Actual v12.9.7/v12.9.9 source archives pass release resolution and note checks for both components.

• Final-head CI completed green on attempt 2 at 16:07:31 UTC after /ok to test 17e19e25b6b. The first attempt passed 105 jobs; two free-threaded Windows A100 rows failed in unchanged Core code, producing a derived status-gate failure. Both exact rows had passed on current main and the previous PR head. The isolated diagnostic then passed both full rows, including the originally failing tests, with identical complete package lists, Python, driver, and final-head artifacts. The normal failed-job retry passed both rows and the aggregate gate while preserving the other successful jobs. No source changes, test skips, or weakened checks were used to obtain the retry result.

• Security and Bandit completed green on the same final head.

• After marking the PR ready, the complete check rollup has 121 successful checks, three intentional skips, and none pending or failing; this includes auxiliary rehearsal checks. Normal CI itself has 108 successful jobs. The separate pre-commit.ci commit status also passes.

• CUDA 12 package-source rehearsal passed all 48 build, sdist, GPU-test, documentation, and prerequisite jobs with source b00b677be366fd3638c78e26b99aa5b238602646 and disposable controller 77cfb553ed6076c5d6d23f0e6d97f2b63f953c7f. Synthetic local-only v12.9.99 is never pushed. Publishing and docs deployment are disabled; missing synthetic notes are deliberately rejected and real 12.9.9 note routing is checked separately. The final custom audit failed on an incorrect harness expectation for the established stable ~= dependency contract. After correcting that expectation and a harness-only gh invocation, the validation-only follow-up completed green using those successful artifacts. It verifies original-run provenance, both production wheel validators, actual base and [all] metadata, and exact-source archive identity; controller 64d05e466a0f285b1876ffbfe0f281f5a733b6f9 retains the evidence. No product dependency policy changed.

• Production release dry runs use b00b677be366fd3638c78e26b99aa5b238602646 as control revision (release tooling is unchanged in the final docs-only commit) and existing real tags, leaving run ID blank: 13.4.3 bindings, 13.4.3 metapackage, 12.9.9 bindings, 12.9.9 metapackage. All four completed green, including docs and final release-artifact validation; publication jobs were skipped. No production publication or remote release tag is requested.

• Final-source release-doc validation checks the final four-file docs delta, renders source 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38 with unchanged package wheels from the rehearsal, and asserts both repaired HTML links. It completed green, including both HTML link checks. Sphinx warnings decreased from 96 to 93; remaining generated API warning categories match the historical source, alongside duplicate object descriptions in release-note documents.

Historical Evidence and Remaining Boundaries

Earlier full local QA on CUDA 13.4 covered Pathfinder, bindings in normal/PTDS modes, bindings Cython tests, and Core. An earlier isolated CUDA 12.9 build covered maintenance bindings in normal/PTDS modes and Cython tests. Those results belong to their recorded earlier revisions; the final-head CI and rehearsal above are the current evidence.

The previous publication-incapable CUDA 12 rehearsal used source cb95f0d73e10e23d640a9559229d9d94bebde185. It is retained as historical mechanism evidence, not evidence for this final candidate.

The v12.9.9 import contained a stale generated-file seal after #2911. Commit 548b2615 corrected that digest and documented a trailing blank-line normalization without changing executable generated content. This does not establish a fresh cybind regeneration. Generator/consumer qualification remains separate follow-up work.

Known inherited boundaries: two NVRTC generated-doc entries refer to functions absent from the maintenance module; raw-digest archive companions do not use conventional filename-bearing sha256sum -c format. The missing CUDA 12 support anchor is fixed here. README links intentionally retain published 12.9.7 documentation because the 12.9.8/12.9.9 documentation URLs currently return 404.

The Windows CI investigation also confirmed an existing Core VMM slow-growth cleanup issue: explicit cleanup is followed by resetting an owning Buffer handle, attempting deallocation again. The same warnings occur in successful baseline runs, and these Core sources are unchanged by this PR. A separate ownership fix needs retained-view and rollback coverage; the logs do not establish this defect as the cause of either intermittent failure.

Supporting unreleased CTK lines, additional simultaneous public roots, source-tree deduplication, and reorganizing private QA are outside this change.

Merge Status

The implementation decisions, local checks, hosted validation, and author-side thread cleanup are complete. The PR is ready for review: all 45 initially open threads are resolved, and GitHub shows 52 of 52 threads resolved. The existing changes-requested review remains for renewed reviewer consideration and approval; no review was dismissed. After merge, validate the three newly available canonical source links on fresh main. Production tags and publication require the normal separate release approval.

Author-side review disposition map (45 initially open threads)

Review area	Final disposition	Threads
Packaged helpers and named interfaces	CI helpers are an installed package with shared parsing. Purpose-specific environment output and named build/test values replace redundant resolver/interface layers and positional arrays. source, source	3916239618, 3916306849, 3916830273, 3917119898, 3931004805, 3931004808, 4031120234
Package identity, metadata and versions	Keep two distinct major roots and explicit current/maintenance roles. Align registry, minor-family SCM selectors, metapackage and fallback metadata in pre-commit/PR CI; retain standard SCM parsing and progression, required major selection, and canonical release-tag spelling. source, source	3916702314, 3916714594, 3917025344, 3917103010, 3931004811, 4031083254, 4031089226, 4031114469, 4031119779, 4062051063, 4062130293, 4062428956
Per-root CI and toolkit pins	Build/test decisions follow the selected source root and Core ABI variant; tag routing checks ref_type, sdist matrices use the nonempty registry, and Pixi checks include both roots. Normal bindings development still builds both dependent Core variants. source, source	3916813446, 3917038211, 3917048863, 3917080542, 3972240690
Release provenance and artifacts	Resolve package metadata from the tagged source, including legacy YAML/JSON. Identify artifact runs by exact tag and SHA, validate the complete wheel matrix, exclude test wheels, and require explicit published-vs-artifact dependencies. Historical support remains bounded by retained source/control metadata and artifacts. source, source	3916893027, 3917053836, 3917070013, 3917097612, 3972240696, 3972240701, 3972240709, 3990562921
Release and maintenance guidance	Document shared-main development plus independent release stabilization, explicit backport dispatch and main return paths. Replace the obsolete diagram and transient review narrative, pin CUDA 12 installation examples, and preserve development suffixes in docs paths. source, source	3917111123, 3972240718, 3972240724, 4030956358, 4030959650, 4031036494
Cross-root changes and generation	Remove the broad equality checker. Require per-root applicability review and target-specific generation provenance; semantic equivalence is not generally decidable. Retain distinct ignore lists while generation layouts differ. source, source	3917089956, 4030963901, 4031135974, 4063090459
Regression fixes and cleanup	Port CUDA 12 NVML visibility/platform handling, make temporary Git repositories independent of signing configuration, and remove the temporary link-check ignore file. source, source	3972240732, 3972240736, 3972240741

These dispositions describe the implementation and retained design choices. They do not represent reviewer approval.

[rwgk]

/ok to test d618c51

[rwgk]

/ok to test bcb537e

[rwgk]

/ok to test f287aed

[rwgk]

Major refresh 2026-09-27T15:09:12-07:00 (PDT)

Outcome

PR #2737 remains open and Draft. I merged current main, synchronized the maintenance bindings source through the released v12.9.9, addressed the actionable review feedback, and pushed the changes to its existing rwgk/cuda-python:agent/cuda-bindings-12-on-main head. The current head is f287aedf6d03db4864d96c0fbc9550a2a94fe2fe. I posted /ok to test f287aedf6d0 for that head. Hosted CI is green: 117 successful checks, 3 intentional skips, and 0 pending or failed at the final check, including a policy check after the PR-description edit. I updated the PR description to reflect the refreshed source, validation, and open reviewer decisions.

This refresh does not rename the package roots or decide the remaining architecture/policy questions for the reviewers. The temporary .lycheeignore is intentionally still present; per Ralf's earlier instruction, remove it after approval, before final merge CI.

At the final check, main was still f9ed2bd and 12.9.x still 89713a7. GitHub reported the PR mergeable against main, but its review decision remains CHANGES_REQUESTED.

Baseline and integration

The previous PR head was f1f5c8e. Public main advanced to f9ed2bd, including the CUDA 13.4.2 generated-source sync, CUDA Core updates, CI/toolchain and Pixi changes. The maintenance branch advanced to 89713a7, the v12.9.9 tag. Its relevant changes include generator catch-up (#2885), parameter-packer simplification (#2911), pointer/coredump work (#2930), and the FFI/channel-format update (#2934).

I merged main first to preserve its latest current-line packaging and workflow behavior. The eight conflicts were in the wheel workflow, a generated benchmark lock, CI environment routing, the registry, three package metadata files, and metapackage setup. The PR's dual-root routing was retained; main's CIBW and packaging updates were incorporated. Next I copied the maintained CUDA 12 source changes from v12.9.9 as complete files where generated, while retaining the PR's existing coredump-lifetime overlay. The release note and new tests were carried over; branch-specific .github/workflows and root .gitignore changes were not imported. An independent source comparison found the imported release files matched v12.9.9 except the intentional overlay.

The shipped v12.9.9 runtime.pyx.in had two pointer changes but an old generated-file digest. The repository's EOF hook also normalized one trailing blank line. I updated only the embedded digest to match the imported source content; there is no executable-code change in that follow-up commit. The generated-file seal hook now passes.

Feedback and disposition

Review references: Mike's changes-requested review, Leo's release-root naming comment, Brandon's release-version comment, Brandon's cross-root-check comment, and Brandon's shared-main coupling comment. The individual inline threads contain more detail; this table records the engineering disposition.

Reviewer	Feedback	Disposition
Mike	Clarify maintenance ownership and a future 12/13-to-13/14 rollover; update the release checklist.	Durable guidance now lives in the shared CI guide; the maintenance guide was shortened; the checklist refers to the configured supported lines.
Mike	Fix release-upload concurrency, simplify positional workflow extraction, and remove the legacy positional run-ID interface.	Concurrency now distinguishes component, tag, and dry-run; the wheel workflow exports named workplan values; the final caller uses --tag, and positional mode was removed with a regression test.
Leo	Require explicit metapackage CUDA major and avoid invalid/duplicated release-version handling.	CUDA_PYTHON_BUILD_MAJOR is now required; CUDA 12 fallback and both source-tag regexes were tightened, with parity tests. The current root's minor-specific tag regex remains intentional for its release family.
Brandon	Stop producing a CUDA 12 development version below shipped v12.9.9; preserve prerelease/post-release metapackage semantics; reject leading-zero patch tags.	Maintenance fallback is 12.9.10.dev0; .postN and prerelease handling are tested; noncanonical patch spelling such as v13.4.01 is rejected. An isolated CUDA 12 12.9.9.post1 sdist-to-wheel smoke test retained the version and exact cuda-bindings==12.9.9.post1 dependency.
Brandon	Make cross-root handwritten-fix review deterministic, and avoid a CUDA 12 emergency release being blocked by unrelated current-line CI.	The guide requires explicit applicability review but does not claim a general deterministic semantic check. One-root source changes still feed both CUDA Core ABI variants in the monolithic wheel: this is a real shared-main coupling and remains a reviewer decision, not a safe one-line planner fix.
Mike and Leo	Consider role-based root naming and a simpler N-root registry.	Explicit cuda_bindings_12/ and cuda_bindings/ names are retained, consistent with Ralf's earlier preference; public CI still validates exactly one current and one maintenance root. The naming/generalization tradeoff remains an explicit reviewer decision.

The release resolver's historical-tag-tree fallback is retained because removing it would change old release-run behavior; that is a separate compatibility decision. A standalone metapackage sdist cannot assume the repository registry accompanies it, so a small amount of package-local version selection remains. Mike's concern about duplicated tests and Leo's concerns around root naming can be revisited during review without conflating them with the v12.9.9 sync.

Local validation

• A fresh worktree TestVenv was built against /usr/local/cuda-13.4 with qa_bindings_linux_build.sh 13.4; the matching QA test script passed. Its six pytest groups reported: pathfinder 1,615 passed/5 skipped; bindings normal 580 passed/25 skipped; bindings PTDS 580 passed/25 skipped; bindings Cython 9 passed; Core 4,179 passed/242 skipped/3 xfailed; Core Cython 1 passed.
• In a separate fresh /tmp/pr2737-cuda12-venv, CUDA 12.9 pathfinder and maintenance bindings editable builds and all three Cython test extensions compiled successfully. The host was temporarily switched to CUDA 12.9 for tests; a runtime assertion confirmed both cudart and nvrtc loaded under /usr/local/cuda-12.9/; the host was restored to 13.4 afterward.
• The first CUDA 12 test run exposed three test-environment issues: a headless OpenGL smoke test, a stale NVML initialization expectation on a CUDA 13.4-capable driver, and cuFile compatibility state leaked by its boolean-parameter test. The current-root tests already handle the NVML driver transition and use cuFile compatibility mode. I made focused maintenance-test fixes rather than changing binding behavior. The rerun passed in both normal and PTDS modes: 332 passed/31 skipped/5 subtests passed each; Cython: 9 passed.
• This host has no nvidia-fs driver, so the cuFile run exercised compatibility mode. It does not independently prove the native GDS path; hosted GPU coverage and a GDS-capable host remain distinct evidence.
• A final read-only review found that the new NVML driver guard initially ran during test collection, even on Windows where the test is skipped. I moved it into the test body; collection, a focused CUDA 12 GPU run (2 passed/1 expected skip), and the full hook suite passed afterward.
• The complete CI-tool suite passed: 170 tests and 55 subtests. The first attempt lacked ci's declared PyYAML dependency in the fresh TestVenv; after pip install -e './ci[test]', two old fixtures needed adjustment to distinguish an exact tag on HEAD from an older reachable tag. The final suite passed.
• pre-commit run --all-files passed, including generated-file seals, registry/pin consistency, Ruff, mypy, actionlint, and lychee. An earlier lychee attempt hit a transient 503 from the external DLPack documentation site; the retry passed. The pre-commit hook is not installed in this checkout, but the full command was invoked explicitly.
• The pinned Pixi v0.73.0 maintenance lock check passed. The 12.9.9.post1 metapackage sdist-to-wheel smoke passed without a pretend-version override during the wheel build.

Local full logs are under /tmp/pr2737_qa_*_20260927.log and /tmp/pr2737-cuda12-build-2026-09-27.log; these are ephemeral workstation evidence, not committed artifacts. TestVenv/ remains untracked in the worktree for follow-up testing.

Commits added

1. f184331a51d82ec2a492c70315f5e345aa227a6e - merge current main and resolve dual-root conflicts.
2. 1d3e7718614f47b4d3b0029079d0e243bdc0ce3a - synchronize maintenance source through v12.9.9.
3. edbe3c28b8cd008de5f4b817d248183536775fd9 - harden version and tag handling.
4. c4feec16202d01a1bbddae6b8ae8226b01d9ca0b - clarify workflow release routing.
5. a0fd3b39134ea27ffa53bc052cd92b74603a7717 - update maintenance and rollover guidance.
6. 548b2615a3adc746b108fb086a64013f9320c481 - correct the imported runtime-template seal.
7. d618c51013d495572ed4419005ee9fe0c7caf79f - correct SCM test fixtures for exact tags at HEAD.
8. bcb537e9823d9f65dd1329496f0128d55234c915 - stabilize maintenance tests on current GPU hosts.
9. f287aedf6d03db4864d96c0fbc9550a2a94fe2fe - defer the NVML driver guard until test execution.

Hosted CI

Final head: f287aed; trigger comment: #issuecomment-5859806421.

• Main CI: run 36350305385 - passed after a failed-only rerun.
• Pixi lock freshness: run 36350294542 - passed.
• Pixi source-build smoke: run 36350294590 - passed.
• Security: run 36350305796 - passed. The independent Bandit scan passed as well.
• Final PR checks: 117 successful, 3 intentional skips, 0 pending, failed, or cancelled across all 120 registered checks after the description edit. The CI run itself was fully green before that extra policy check.

The first Docs job built/rendered successfully but its link checker received HTTP 429 from an external Conda documentation page, the sole error among 13,395 unique checked links. I reran only failed jobs. The new Docs job passed rendered-link validation, and the aggregate status job passed. No source change was made for this external rate-limit flake.

Remaining reviewer decisions and merge reminders

1. Confirm explicit versioned source roots vs lifecycle-named roots; do not silently combine naming and registry redesign with this refresh.
2. Decide whether shared-main CI coupling is acceptable for emergency CUDA 12 work; the current monolithic Core wheel and two-ABI build mean a maintenance PR can be blocked by a current-line failure. A focused CUDA 12 tag CI does not fully solve review-PR coupling.
3. Decide how much automated cross-root checking is useful. Byte equality is valid only for a deliberately identical subset; handwritten applicability is a review judgment.
4. Decide whether old tag-tree reruns should remain supported; that controls whether the historical resolver fallback can be removed.
5. Before merge: obtain approval, remove .lycheeignore after approval, run final CI, then validate the three previously excluded main/cuda_bindings_12 links after merge.

The older six-layer review branch is an architectural snapshot through f1f5c8e, not tree-identical to the refreshed PR head. The updated PR description now reflects v12.9.9, the 13.4.2 pin, this CI run, the earlier dry-run follow-ups, and the remaining reviewer decisions.

[rwgk]

Pre-merge decisions and short-term follow-ons

PR #2737 is large enough that prolonged incubation is becoming a risk of its own. Its current tree imports the maintained CUDA 12.9 source through v12.9.9, has passed fresh local CUDA 12.9 and 13.4 builds and tests, and has a green hosted CI matrix. Yet main, 12.9.x, cybind, and ctk-next continue to evolve separately. Each delay increases the chance of another source synchronization, generated-code mismatch, merge, and full validation cycle, while the old backport branch remains part of the active workflow. We should settle the few decisions needed for a safe shared-main release model, obtain explicit review approval, complete the final cleanup and CI, and merge promptly. At the same time, we should give the independent generator, release-QA, and maintenance improvements named owners and near-term tracking. Merging soon is a sequencing choice, not a request to waive substantive review or release safeguards.

Status at this assessment

• PR head: f287aedf6d03db4864d96c0fbc9550a2a94fe2fe. At the September 27 check, public main was f9ed2bdaede7b66e6323dc7772a93953af39dfc7, already merged into the PR. The 12.9.x head was the imported v12.9.9 tag. No new sync was required at that moment; recheck immediately before merging.
• The PR is Draft and GitHub reports it mergeable but blocked. Hosted checks have no failures or pending jobs; local CUDA 12.9/13.4 testing is recorded in the major-refresh report. The publication-incapable CUDA 12 release dry run exercised the release mechanism on an earlier source snapshot, not the final v12.9.9 payload.
• There is no approval yet, and Mike's changes-requested review remains active. Many technically unresolved threads are outdated or have been addressed in code; the unresolved-thread count is not a list of distinct blockers.
• A fresh six-layer review branch is available. Its sixth commit has exactly the same tree as the current PR head; it is a narrative review aid, not a second implementation.

Definitely resolve before merging

1. Settle the core architecture and obtain approval. Ask Mike, and Leo on the points he raised, for an explicit decision on the physical root names (cuda_bindings_12/, cuda_bindings/) and the two-status package-root registry. The current implementation need not be renamed merely because an alternative exists, but Mike's requested changes and Leo's naming question cannot be silently deferred as minor follow-ons. Either obtain acceptance of the current model or make a focused revision and validate it. Answer Leo's separate questions about the intentional 13.4 tag-family pin in bindings metadata and standalone metapackage setup; change code only if the explanation exposes a real defect or the reviewer rejects the contract.

2. Agree on the operating policy for urgent CUDA 12 fixes. Brandon's concern is real: a maintenance-only change can encounter failures in the current-line CUDA Core build because one Core wheel serves both ABIs. Before removing routine backports, the owners should explicitly accept that coupling for normal PRs and identify an emergency release escape path if current-line CI is broken. The escape path need not become a new framework inside cuda.bindings: support multiple CTK release lines on main #2737, but it must be credible and documented or consciously accepted as a residual risk. Similarly, confirm the cross-root handwritten-fix review policy: reviewers assess applicability in both roots; a generic byte comparison cannot prove semantic equivalence (Brandon's comment).

3. Close the review loop on the actual current head. Walk the new six-layer branch with Mike and Keith; reply to the still-relevant comments, distinguish superseded threads from outstanding decisions, and obtain the required approval. The existing historical-tag fallback should be accepted or deliberately changed if a reviewer objects; it is already implemented and tested, so speculative simplification is not itself a merge prerequisite. Keep the manual-seal disclosure visible, but do not make a cybind regeneration an implicit condition for approving the checked-in, tested source.

4. Do the final source and CI check. Immediately before merge, compare the PR against the then-current main and 12.9.x heads and integrate relevant new changes. After approval, remove the temporary .lycheeignore as already agreed in review, manually trigger final CI on the exact merge candidate while retaining Draft-mode control if desired, and mark the PR ready at the chosen point. Do not treat the currently green matrix as proof for a later changed head. The three temporarily excluded main/cuda_bindings_12 links can only be checked on fresh main after merge.

The key distinction: items 1 and 2 require an explicit reviewer/owner decision before merge; they do not necessarily require more code in this PR. Items 3 and 4 are the approval and validation gates. The existing CI and dry-run evidence support a prompt merge once those gates are satisfied, but the earlier dry run does not prove the final release payload end to end.

Short-term follow-ons to track

Priority / timing	Work to track	Definition of done
Parallel now; finish promptly	Resolve drift among cybind, ctk-next, and public main. cybind !580 and CUDA 13.4 regeneration #2953 are already open. Review #2953's Cython ABI and Python API compatibility concerns on their own merits; then verify a CUDA 12 regeneration also emits the intended typed kernel-parameter access and valid seals.	A generator/helper combination is validated for each active source line, with generated diffs and ABI/API effects reviewed. No requirement to pin an old cybind revision merely to reproduce #2737.
Immediately after merge; before the first new CUDA 12 release	Rehearse the CUDA 12 tagged-release path from merged main with publication disabled, using the then-current source and release resolver. The previous synthetic-tag dry run predated the v12.9.9 import and later release fixes.	Selection, artifact matrix, provenance, documentation, and upload-disabled behavior are recorded for the merged implementation; no public tag or package is published by the rehearsal.
Next maintenance cycle	Turn the agreed emergency CUDA 12 policy into an exercised runbook or narrowly scoped CI/release escape path if the shared-main coupling is judged too costly. Stop routine 12.9.x backports once #2737 merges, but retain the historical branch as a release record until the team is comfortable with the new procedure.	An urgent maintenance fix can be built, tested, and released with a documented route even if unrelated current-line CI is unhealthy.
Next focused quality PR	Add only justified cross-root checks for files with a documented identical-content invariant, and make the handwritten-applicability reminder easy to follow in review. Do not promise a universal automated semantic-equivalence check.	The checks catch real accidental drift without rejecting legitimate CUDA-version differences; reviewers retain responsibility for applicability.
Small independent fixes	Address the dry-run findings already listed in the PR description: stale CUDA 12 NVRTC generated documentation, the missing support anchor in a release note, and filename-bearing .sha256sum companions if conventional sha256sum -c use is desired.	Each issue has a small tracked fix and focused validation; release-doc and checksum defects do not expand #2737.
Immediately after merge	Validate the three previously excluded links by running pre-commit run lychee --all-files on fresh main, and record the result.	All three canonical main/cuda_bindings_12 links resolve; no temporary ignore file remains.

Useful lower-priority cleanup includes reconsidering the historical-tag fallback if the team drops old-tag reruns, aligning the .gitignore patterns raised in review, and addressing the CI pipeline diagram source tracked in #2740. Broad N-root generalization, same-major unreleased CTK-line support, and wholesale source deduplication should not be smuggled into #2737 or treated as urgent follow-ons without a concrete near-term need.

Proposed coordination

Before declaring #2737 ready, record the architecture and emergency-policy decisions in the PR and assign an owner and target milestone to each high-priority follow-on. That makes the trade explicit to Mike, Keith, Leo, and Brandon: merge the well-tested shared-main foundation promptly, then close the narrower gaps in independent, reviewable changes. If any reviewer judges a listed follow-on indispensable to safe first use, move that specific item into the pre-merge set rather than letting the entire list become an open-ended condition.

[rwgk]

/ok to test 5deacef

[rwgk]

/ok to test 59ff7bd

[rwgk]

This was generated by codex gpt-6-sol ultra

---

@mdboom, following up on your review. Thank you for separating the concrete fixes from the larger design questions. I do not mean to mark the whole review resolved; this is where each point stands after the latest changes.

Your point	Status
Release checklist and the 13-to-14 rollover procedure	a0fd3b3 updates both. 5deacef corrects the rollover guide's stale tag_regex wording after the SCM simplification.
Release-upload concurrency and the dead positional lookup-run-id interface	Addressed in c4feec1. Upload concurrency is keyed by component/tag/dry-run, and callers use the explicit tag option.
Duplicated metapackage version policy	74ef24e removes custom tag_regex handling. The standalone cuda-python sdist lacks the repository registry and sibling pyprojects, so some major-specific selectors/dependencies remain in setup.py; the updated description calls out this residual duplication for an explicit decision.
Physical root naming and the generic registry versus two named roots	Still open. The description now distinguishes the mapping's abstraction cost from the actual public builder, which validates exactly one current and one maintenance root with distinct ABI majors. I'd value your judgment on whether to keep or narrow that model.
Historical-tag fallback	Retained intentionally for supported older tag trees with the required metadata/control pin. The policy of rerunning pre-registry tags is explicitly open in the description.
Tests redundant with CI	No systematic pruning has been done. The updated description asks whether you want that audit before merge or are comfortable tracking it as a follow-up; I am not marking this resolved.

I also made a fresh six-layer review-only PR, whose final tree exactly matches the current #2737 head. It is a reading aid, not an alternative merge path. CI at head 59ff7bd is green, with three intentional skips.

[rwgk]

/ok to test b00b677

[rwgk]

/ok to test 17e19e2

[rwgk]

Work Report — GPT-6-Astra ultra — Autonomous drive to merge readiness

Outcome

The engineering work is complete, and I am comfortable with merging this implementation after the required renewed reviewer approval. PR #2737 is marked ready for review at 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38. Three product commits are pushed, final normal CI is green with 108 successful jobs, release validation is complete, and all 52 discussion threads are resolved. The existing changes-requested review remains intact; the PR has not been merged.

The chosen direction is implemented: maintain the two supported CUDA ABI-major source roots together on main, and use short release branches when a version needs stabilization or an urgent repair. The implementation preserves independent bindings/metapackage releases for each line and documents how fixes return to integration.

Expand for details

Date: September 29, 2026. Requested report start label: 0640 America/Los_Angeles.
Final verification completed at 16:13 UTC / 09:13 America/Los_Angeles.

Starting point and scope

• Initial PR head: 59ff7bd88756bda02164daefb0a71daff47b1d2e.
• Initial main: 36e4d40a664f08ca88b30a50c3553da56d4ef3c2.
• Historical 12.9.x and v12.9.9: 89713a7c8bf61037f6a0375ab611778c21f6f9cd; the live branch still matched this pin during the audit.
• Audited all 52 inline review threads, including 45 initially unresolved threads, and the substantive review summaries.
• Preserved Keith Kraus's original source-import and build/release integration credit from PR cuda.bindings: build 12.9 and 13.x selectively from main #2675.
• Used GitHub App MCP for PR/review/job reads and GitHub CLI where branch, dispatch, pagination, or artifact operations needed it. Local Git and source inspection supplied exact revision evidence.

Added product commits

1. e785620d9bb67c31f405cf54a609101f184d1714 - release-branch policy, manual backport workflow, reviewer decisions in durable documentation, contributor CI overview, missing support anchor, and removal of the temporary link exclusions.
2. b00b677be366fd3638c78e26b99aa5b238602646 - metadata consistency, full CI-tool PR gate, standard SCM progression after reachable tags, historical release-note compatibility, and focused regression coverage.
3. 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38 - two explicit MyST targets, one repaired historical-note link, and an intentional release-note template marked orphan. Only four authored documentation files differ from the implementation/release-rehearsal candidate.

The user explicitly approved retaining the existing fork head as an exception to the standing upstream-branch convention, so all product commits were pushed to rwgk/cuda-python:agent/cuda-bindings-12-on-main. Its existing tracking configuration was preserved. Commits were added; existing history was not amended. No merge or production release was performed.

Decisions resolved

Branches and maintenance scope

main integrates both supported majors. A short release branch can freeze selected changes while development continues, and an emergency can start from a previously released modern-layout tag. Before the first migrated CUDA 12 release, the baseline must be a validated commit containing both roots; the historical single-root branch is not silently resumed.

Major-line maintenance can include explicitly selected API, Python, or platform support. Once a patch release is frozen, only approved fixes and release prerequisites enter that release. The checklist records this scope rather than treating the word "maintenance" as a complete release policy.

The manual backport workflow validates a merged source PR and one existing non-default target branch. The pinned action creates a reviewable PR. Label-derived targets are disabled; there is no automatic merge. Workflow-file changes need appropriate manual credentials, and generated changes need target-specific regeneration. Branch-first emergency repairs require a linked return to main and an applicability decision for both roots.

Durable procedure: RELEASE-bindings.md.

Layout, registry, and package boundaries

Keep the explicit CUDA 12 physical root and separate lifecycle roles. The supported shape is exactly one current and one maintenance major. Retaining the mapping avoids scattering source paths through workflows; it does not implement arbitrary numbers of supported public lines. A future 13/14 rollover must preserve the CUDA 13 source in a separate root before introducing CUDA 14.

Keep explicit minor-family SCM selectors so source builds do not accidentally select a prior minor's tags. Keep the standard setuptools-scm parser. The standalone metapackage's small selector/dependency policy is intentional because its sdist cannot import sibling source trees or repository CI metadata. The new guard checks the duplicated selectors and maintenance fallback against the actual package metadata.

Release independence and historical compatibility

Normal development CI includes affected dependents and can build both Core ABI variants. A bindings release tag selects one bindings root and its matching metapackage, uses published Pathfinder, and does not require another-major or Core release. Stabilization branches isolate unfinished development by freezing a tested source baseline; they do not waive required checks.

The tagged registry is authoritative when present. Historical trees without it retain a bounded compatibility path, with separate source and workflow-control revisions. Expired artifacts and indefinitely retired configurations are not promised to remain releasable.

Cross-root review and cybind

Handwritten changes require explicit applicability review in both roots. Semantic equivalence cannot generally be established by comparing bytes. Generated imports should record the generator revision, toolkit inputs, command, and any manual adjustments. Seals certify bytes, not generator provenance.

Continuous toolkit QA and broader cybind asset/consumer improvements remain independent follow-ups. This change does not need a generator overhaul or QA directory move to establish the new source ownership. The final generated payload remains the reviewed v12.9.9 import plus previously disclosed overlays and seal normalization.

Concrete defects fixed during this drive

1. Metadata drift escaped pre-commit and ordinary PR tests. The hook watched pyprojects, but schema validation stopped inspecting their selectors after the SCM simplification. Added an explicit current-tree metadata validator and ran the full existing CI-tool suite in PR CI. Historical configuration loading remains independent of current-tree validation.
2. Reachable tags could be overridden by a manufactured maintenance version. The old helper compared reachable versions with the fallback and special-cased exact HEAD tags. A descendant of an appropriate earlier post-release tag could incorrectly receive a fabricated next-patch version. The helper now asks Git whether any tag selected by the package's real selector is reachable, then defers to standard SCM progression.
3. A new Git fixture inherited machine-wide signing settings. Disabled signing only inside the temporary test repository, preserving real commit signing.
4. Strict tagged-source notes broke real historical releases. v12.9.7 lacks matching notes in its tagged tree, and v12.9.9 lacks separate metapackage notes. Added an explicit legacy-only fallback to exact-version control notes, using the control registry's root; historical metapackages may use matching bindings notes with a visible warning. Empty notes, wrong-version notes, invalid resolver metadata, and registry-bearing new releases remain strict.
5. Documentation had broken local targets. Added the missing support-page anchor. Comparing the full imported docs with the historical release then exposed two MyST targets and one orphan template warning; the third commit fixes those authored-doc issues. A focused render with production Sphinx/MyST versions reproduces exactly three warnings before the patch and zero afterward, and verifies the HTML links.

Removed five redundant snapshot/static-agreement tests. Retained negative artifact selection, matrix completeness, exact-tag identity, historical-source, and mixed-workplan tests because ordinary happy-path CI does not exercise those failures.

Local validation

• Checked local_cuda.sh status before local test activity; every check reported active cuda-13-4.conf. No global CUDA switch was performed.
• Installed the declared ci[test] dependencies into the existing TestVenv after the first collection attempt reported missing PyYAML.
• Final CI-tool suite: 191 passed, 55 subtests passed; two setuptools-scm deprecation warnings for the existing supported configuration spelling.
• Full required pre-commit run --all-files passed with only lychee skipped after its separate network run. Generated-file seals, metadata/pin checks, Ruff, mypy, actionlint, and other hooks passed.
• Full network lychee: 800 successful checks and exactly three expected pre-merge 404s, all canonical main/cuda_bindings_12 links. Removed .lycheeignore permanently. Separately checked the newly tracked release guide and changed documentation; that check passed.
• Actual archived v12.9.7 and v12.9.9 trees passed the real resolver and release-notes CLIs for both bindings and metapackage. The checks exercise distinct tagged-source and control-tree package roots.
• Inherited-global-signing regression selection passed. SCM tests cover selected regular/prerelease/post-release tags and descendants, unmatched release families, and unreachable tags.
• Official seal validation covered 133 marked files across the two roots.
• Native bindings code did not change during this drive. The CI/tooling/docs changes were validated locally through their own tests, then through hosted build/release execution. Earlier full local native QA belongs to earlier revisions and is not presented as a fresh rebuild of this candidate.

Local logs:

• /tmp/pr2737_final_ci_tools_20260929.log
• /tmp/pr2737_precommit_final_clean_20260929.log
• /tmp/pr2737_precommit_network_20260929.log
• /tmp/pr2737_precommit_docs_final_20260929.log
• /tmp/pr2737-docs-anchor-check/before/render.log and /tmp/pr2737-docs-anchor-check/after/render.log

Hosted validation

The product head advanced to 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38 only for the four authored-doc fixes. Native package code, packaging metadata, CI helpers, and production release workflows are byte-identical to b00b677. Normal final-head CI was requested with /ok to test 17e19e25b6b. The earlier /ok to test b00b677be36 run was superseded and cancelled, not recorded as a product failure.

Final-head check	Result
Normal CI, attempt 2	108 successful jobs, completed September 29 at 16:07:31 UTC
Security	Successful: Pulse and CodeQL pass; suite preflight intentionally skipped
Bandit	Successful

After marking the PR ready, a stable, fully paginated check read confirms 121 successful checks, three intentional skips, and none pending or failing. These 124 checks include auxiliary rehearsal checks; normal CI itself has 108 successful jobs. The separate pre-commit.ci - pr commit status passes. The new assignee/labels/milestone check also passes.

The final-head CI metadata/helper gate independently confirms the same 191 tests and 55 subtests passed. The original complete attempt and the failed-job retry are preserved separately; the following section records the two intermittent Windows failures and the investigation rather than hiding them behind the final green result.

Final-head Windows failure investigation

The first observed normal-CI failure was Windows Python 3.14t / CUDA 13.0.2 wheels / A100 MCDM. The jit_lto_fractal.py subprocess exited with 3221225477 (0xC0000005, native access violation), with empty stdout/stderr and no crash stack. The rest of that Core test run reported 3,902 passed, 486 skipped, and two expected failures.

Compared the exact same row against main baseline 36e4d40 and prior PR head 59ff7bd: both had the example pass and reported 3,903 passed, 486 skipped, and two expected failures. All three used published bindings 13.0.3, CuPy 14.2.0, NVRTC 13.0.88, runtime 13.0.96, and nvJitLink 13.4.92. Core source, Pathfinder source, the example, and its test have identical Git object identities between the main baseline and final head. The CUDA 12 imported bindings are not installed in that row. This evidence supports a focused diagnostic rerun; it does not identify the native crash's root cause or justify a speculative code workaround.

The original attempt completed with 108 jobs: 105 successful, two failed Windows test rows, and their derived Check job status failure. The complete paginated attempt-1 records were saved before any retry (/tmp/pr2737-original-ci-attempt1-jobs-pages.json; independently also /tmp/pr2737-final-pr-ci-attempt1-jobs.json). At 15:44:20 UTC, after confirming the exact PR head and unchanged base, the GitHub App accepted a failed-job-only rerun of the original workflow. No successful build/test row was rerun or removed from coverage.

The failed-job retry completed successfully at 16:07:31 UTC. GitHub reports all 108 jobs successful: 105 successful first-attempt jobs retain their original execution timestamps, while only the two failed Windows rows and the aggregate gate execute again. The CUDA 13.0 retry explicitly passes the fractal test and reports 3,903 Core tests passed, 486 skipped, two expected failures, and 15 warnings in 276.51 seconds. The CUDA 13.4 retry explicitly passes the VMM growth case and reports 4,060 Core tests passed, 409 skipped, four expected failures, and 112 warnings in 379.06 seconds. The product source is unchanged throughout; no test suppression, relaxed assertion, or reduced matrix was used. The original failure causes remain unresolved, with the separately confirmed existing ownership defect described below.

A second job failed in Windows Python 3.14t / CUDA 13.4.2 wheels / A100 MCDM: test_vmm_allocator_grow_allocation[handle_type1] (win32_kmt) received CUDA_ERROR_INVALID_VALUE from cuMemAddressFree while releasing a temporary, noncontiguous reservation. The reserve call had succeeded. The remaining Core result was 4,059 passed, 409 skipped, and four expected failures. The identical test passed on main and the previous PR head, each with 4,060 passed. Core source and test blobs are unchanged; driver 596.36, CuPy 14.2.0, and runtime 13.4.92 match. No common cause is established for these two different errors.

Queued x86 L4 rows delayed completion and GitHub rejected a job rerun while the containing workflow remained active. Read-only runner status was unavailable (repository runner API permission denied). Historical successful baseline jobs had already shown 78-162 minute L4 queue delays, so the observed queue alone did not establish an outage. The test matrix was preserved.

A separate Windows diagnostic run uses controller 7ef419f7eb1a0ee8cc1b690487dc9f9f1cf66a1e on rwgk/pr2737-windows-crash-diagnostic-17e19e2. Its one new wrapper calls the unchanged Windows test workflow at final product source 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38, reuses native artifacts from the normal run, preserves the exact original registry/workplan and two failed matrix rows, and adds only PYTHONFAULTHANDLER=1 and PYTHONUNBUFFERED=1. It has read-only repository permissions and no inherited repository secrets. Third-party dependencies still use the normal production constraints, so their resolved versions are compared in the diagnostic logs.

The diagnostic completed green on its first attempt: all three jobs passed. The CUDA 13.0 row explicitly reports the fractal example PARALLEL PASSED, with 3,903 Core tests passed, 486 skipped, and two expected failures. The CUDA 13.4 row explicitly reports the previously failing VMM growth test PARALLEL PASSED, with 4,060 Core tests passed, 409 skipped, and four expected failures. Both used the ordinary four-thread test configuration. The complete package lists immediately before Core are identical to their original jobs (34 packages for CUDA 13.0 and 44 for CUDA 13.4), as are Python 3.14.7 and driver 596.36. The inherited VMM cleanup warnings remain. These results establish that the two failures are intermittent in unchanged source and matching environments; they do not establish either failure's root cause. Evidence: /tmp/pr2737-win-fractal-diagnosis.md, /tmp/pr2737-windows-diagnostic-versions.json, and the two diagnostic excerpts beside them.

Separate pre-existing Core ownership defect

The failure investigation uncovered and independently confirmed a duplicate deallocation attempt in the existing VMM slow-growth path. Buffer ownership captures the resource and size in a C++ deleter. Slow growth explicitly unmaps and frees the old address, then Buffer._clear() resets the owner, invoking its deallocator again. The deallocator can stop at a failed unmap; this is a duplicate cleanup attempt, not proof of a second successful free. Matching destructor warnings appear in both successful baseline runs as well as the failed run. All implicated source files are byte-identical to the main baseline.

The failing temporary-reservation address differs from the warning addresses, so the logs do not establish an address-reuse interleaving or prove this defect caused that failure. No common cause with the JIT subprocess crash is established.

A separate Core correction should preserve the old mapping and owner while transactionally constructing the new mapping, transfer new-resource ownership exactly once, and cover rollback and retained-view behavior. Simply resetting an ownership flag or explicitly closing the old Buffer can invalidate DLPack exports that retain it. This ownership/API follow-up is outside the branch migration's source changes and is recorded for team follow-up.

Exact-source CUDA 12 rehearsal

Source candidate: b00b677be366fd3638c78e26b99aa5b238602646.
Initial disposable controller: 77cfb553ed6076c5d6d23f0e6d97f2b63f953c7f.
Branch: NVIDIA/cuda-python:rwgk/pr2737-cuda12-release-rehearsal-b00b677.
Initial run: https://github.com/NVIDIA/cuda-python/actions/runs/36579081279

The controller changes only five workflow files in a separate worktree. It pins source checkouts and artifact identities to the product candidate, creates synthetic v12.9.99 only in runner clones, uses read-only permissions, physically removes docs publication steps, and contains no package-publishing path. No remote synthetic tag is created.

The rehearsal deliberately verifies rejection of missing synthetic 12.9.99 notes and independently validates real 12.9.9 notes. It does not manufacture a product release note. The intended payload checks cover 18 release wheels, one matching metapackage wheel, wheel METADATA and the stable metapackage's compatible-release bindings requirements (base and [all]), archive source identity, sdists, GPU tests, and documentation.

All 48 original prerequisites passed: 24 native wheel builds, two sdists, 17 GPU test rows, documentation, three matrix jobs, and source resolution. The original rehearsal's final custom metadata assertion was too strict: it expected one ==12.9.99 requirement, while the established stable-release contract correctly emits base and [all] ~=12.9.99 requirements. Both production wheel validators had already passed. The correction was confined to a validation-only controller that reuses the successful artifacts and verifies the actual package contract; no product code or native rebuild was needed.

The first validation-only attempt, 36583210291, exposed another harness-only issue: hosted gh api rejects combining --slurp with --jq. After changing that command to pipe paginated JSON through jq, 36583421384 completed green with controller 64d05e466a0f285b1876ffbfe0f281f5a733b6f9. It verifies all 48 successful original prerequisites and the single known original audit failure, both production wheel validators, wheel metadata, and the exact-source archive. The original run remains recorded as failed; the successful follow-up establishes the corrected artifact audit separately.

Evidence artifact: pr2737-cuda12-release-evidence, ID 11041026359, 113,805,674 bytes, retained until October 29, 2026. GitHub reports archive SHA-256 3469ec9c8691d19dfceb93880904140be56d613e926e2a9bbb8b5b633b837259. Its manifest records source, artifact-run/controller, validation-run/controller, and payload hashes separately.

Downloaded and independently inspected the evidence: all 19 wheel hashes match, all wheel metadata versions are 12.9.99, and the binary matrix is exactly CPython 3.10, 3.11, 3.12, 3.13, 3.14, and free-threaded 3.14 across Linux x86-64, Linux AArch64, and Windows x86-64. Experimental 3.15/3.15t builds are excluded from publication inputs by the production validator. The source archive hash is 172fcc14468358400b93c9d6cf06f3622f2317d10586195c23f8df91f9ef0e43; archival metadata identifies source b00b677 and local tag v12.9.99. Representative registry, package setup, pyproject, and generated driver blobs match that source commit byte-for-byte. Downloaded manifest and inspection record: /tmp/pr2737-final-release-evidence-36583421384/.

This source rehearsal uses a branch push, so it does not establish tag-event run discovery or publication credentials. The separate production dry runs exercise exact-tag lookup with real retained artifacts.

Production release dry runs

Control revision: b00b677be366fd3638c78e26b99aa5b238602646 on isolated branch rwgk/pr2737-release-control-b00b677.
All dispatches explicitly request release-action=dry-run, leave run-id blank, and leave the docs deployment branch blank.

Component	Existing tag	Successful production dry run
cuda-bindings	v13.4.3	36579384563
cuda-python	v13.4.3	36579488035
cuda-bindings	v12.9.9	36579493491
cuda-python	v12.9.9	36579498745

All four completed successfully, including exact-tag CI discovery, notes validation, docs, wheel-matrix checks, and release-artifact preparation. PyPI/TestPyPI publication was skipped; no GitHub release or docs deployment was created.

Exact source/artifact lineage: v13.4.3 = 972ee3b15f4f120f0f99e9d698a75f0e19331b0e, CI 35802685334; v12.9.9 = 89713a7c8bf61037f6a0375ab611778c21f6f9cd, CI 35803298412. Both existing tags predate the new registry. These runs exercise the candidate control implementation against actual legacy sources; the separate exact-source rehearsal covers the new two-root source layout.

Final-source documentation refresh

Final source: 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38. Controller: fcc602d155e5433ef1f0b5514dd763eb8ae388bb. Run: https://github.com/NVIDIA/cuda-python/actions/runs/36581936586. The two-file disposable controller proves the source delta contains only the four authored docs, reuses unchanged native wheels from the original rehearsal, builds full release docs from the final source, and checks both repaired links in the actual HTML. This is not represented as a native-wheel rebuild at the final docs-only SHA.

All three jobs passed. Sphinx warnings decreased from 96 to 93; the three authored-doc defects are fixed. Remaining generated-API warning categories match the historical v12.9.9 source. Duplicate object descriptions are more numerous because this tree retains more release-note documents. The full docs build is successful, not warning-free.

Review disposition

Resolved all 45 threads that were open at the initial audit after checking the implementation, documented decisions, relevant validation, and fresh discussion state. GitHub now shows 52 of 52 threads resolved, including the seven that were already resolved. The PR body contains a grouped disposition map with permanent source links; the appendix below records all 45 individual decisions. New comments and thread state were checked before mutations. No thread reply or unrelated issue/comment was posted; the only new PR comments were the two expressly requested /ok to test triggers.

Author-side thread resolution is distinct from human approval. No review was dismissed and no approval was represented as granted. The PR was not merged.

Remaining boundaries

• After merge, check the three new canonical main-branch URLs again.
• Two inherited generated NVRTC documentation entries refer to functions absent from the CUDA 12 module. They need a generator-side correction; this drive does not hand-edit sealed generated docs.
• Existing production archive checksum companions contain raw digests, not conventional filename-bearing sha256sum -c input.
• CUDA 12 README links intentionally remain at published 12.9.7 docs: direct network checks found 12.9.7 HTTP 200 and 12.9.8/12.9.9 HTTP 404.
• Short-lived rehearsal/control branches are retained as validation evidence. No existing branch/tag was force-pushed or deleted.
• The manual backport action is statically validated against its pinned implementation; no artificial backport PR was opened solely to exercise a team-facing mutation.

Final repository and PR state

• Product head: 17e19e25b6b20eee6a5c13c8b39befd1b2b11a38; base main remains 36e4d40a664f08ca88b30a50c3553da56d4ef3c2.
• PR is open, ready for review (draft=false), and mergeable without conflicts. GitHub still reports CHANGES_REQUESTED / BLOCKED; renewed reviewer approval remains outstanding.
• All 52 review threads are resolved. No reviewer approval was manufactured or dismissed.
• The tracked working tree is clean. Local agent/cuda-bindings-12-on-main retains tracking of origin/agent/cuda-bindings-12-on-main, and the PR fork head matches the tested product head.
• Three product commits were added and pushed. Four isolated canonical validation branches retain the release-control, CUDA 12 rehearsal/audit, final-docs refresh, and Windows diagnostic evidence. No existing branch/tag was force-pushed or deleted.
• No merge, remote version tag, package publication, GitHub release, docs deployment, or global CUDA toolkit switch was performed.

Appendix: individual review dispositions

The PR description contains the grouped review map. These individual decisions cover the 45 threads that were open at the initial audit; they do not imply reviewer approval.

Review thread	Author-side disposition
3916239618	CI helpers are a real repository-private distribution, installed editable with declared runtime/test dependencies.
3916306849	Removed the standalone release resolver and redundant JSON-to-environment reconstruction. The purpose-specific write-github-env operation validates the selected record; JSON remains only at structured workflow boundaries.
3916702314	An explicit current-tree metadata guard compares registry minor-family selectors with bindings/metapackage SCM metadata and maintenance fallback/Pixi versions; it is separate from historical schema loading.
3916714594	PEP 440 interpretation uses packaging.version.Version; release routing additionally requires canonical tag spelling and rejects local-version upload tags.
3916813446	Bindings/metapackage rebuilds are selected by root. Normal bindings-source changes still build both dependent Core ABI variants; the author does not claim complete CI independence.
3916830273	Shared registry/version helpers live in the installed ci.tools package rather than relying on ad hoc script imports.
3916893027	Retain runtime tagged-source resolution: a dispatch control revision can differ from the source release tag, so precomputing today's registry cannot establish the historical package/layout/toolkit. The supported compatibility boundary is documented.
3917025344	The drift guard now runs directly in pre-commit and per-PR CI, including the complete CI-tool test suite; metadata agreement is no longer enforced only by nightly static assertions.
3917038211	CI passes a release tag only when github.ref_type is tag; branches whose names begin with v do not trigger release routing.
3917048863	Linux and Windows sdist matrices enumerate the validated two-root registry; per-root workplan gates select execution without constructing an empty matrix.
3917053836	All component artifact selection excludes names ending in -tests before download, preventing test wheels from entering publication inputs.
3917070013	Artifact mode requires exactly one local Pathfinder wheel. Published mode is explicit; a missing or duplicate artifact cannot silently fall back to PyPI.
3917080542	The Pixi checker validates every registered bindings root and matching Core CUDA variant; both roots are included in the pre-commit inputs.
3917089956	Removed the broad shared-files equality/symlink checker rather than retaining an incorrect invariant. Shared maintenance guidance now requires semantic applicability review and permits narrowly scoped equality checks only where justified.
3917097612	CUDA_PYTHON_ARTIFACT_NAME is emitted only for local bindings using the registered build toolkit; published bindings no longer invent a local artifact name.
3917103010	The registry explicitly supports exactly one current and one maintenance root with distinct CUDA majors. It centralizes physical paths and roles; arbitrary N-line support is not promised.
3917111123	Replaced the obsolete contributing diagram with the current package-selection, artifact, dry-run and publication flow. Broader diagram elaboration can remain in #2740.
3917119898	Removed the obsolete installer helper. Retained TOML readers use tomllib rather than regex-based TOML parsing.
3931004805	Adopted the proper-package option with editable installation and declared test extras, matching the shared-helper architecture.
3931004808	Removed the generic echo-through package-json/github-env options; write-github-env is a dedicated operation with a concrete consumer contract.
3931004811	The earlier response about regex validation is superseded: the final guard checks the default-SCM selectors, fallback metadata and standalone metapackage agreement, with negative drift cases.
3972240690	Linux and Windows test gates index each row's bindings root and Core ABI variant; mixed bindings-source/Core-test regression coverage is retained.
3972240696	Release validation derives the expected Python/platform/ABI matrix from tagged workflow metadata and rejects missing, extra and duplicate wheel targets.
3972240701	Dependent component releases resolve their bindings dependency from that component's tagged source, using its current role or recognized legacy metadata, rather than today's control registry.
3972240709	The tagged-source resolver probes both versions.yml and versions.json and preserves the historical toolkit pin. Explicit compatibility fallbacks remain bounded; current metadata does not overwrite recognized historical metadata.
3972240718	Both bindings docs builders preserve .dev suffixes when choosing the docs version path, avoiding occupation of a stable version path by a development tag.
3972240724	CUDA 12 installation examples pin the 12.9 line; README/DESCRIPTION links target valid versioned 12.9.7 docs instead of latest CUDA 13 docs. This disposition does not claim those links point to the newest published patch.
3972240732	The CUDA 12 NVML test handles unavailable PCI information, skips Orin/Thor naming mismatches and compares visible CUDA devices against the NVML superset.
3972240736	Temporary repositories disable inherited commit/tag signing in SCM/tag-selection/run-lookup fixtures. The inherited-signing regression selection passed.
3972240741	Removed the tracked temporary .lycheeignore file. Final full pre-commit passed with lychee skipped; a separate network-enabled link scan reported 800 successes and only three expected pre-merge main-path 404s. These are reported, not hidden by a new ignore file.
3990562921	Artifact lookup requires a successful push run for the exact tag and source SHA; same-SHA sibling tags are distinguished. The policy releases each line independently and does not require both versions to share a commit.
4030956358	Removed transient import/review history from the maintenance guide; it now identifies the root, integration policy and lasting shared instructions.
4030959650	The maintenance guide now gives ongoing maintenance/release instructions and links common policy instead of addressing this PR's reviewer.
4030963901	Generation and handwritten applicability instructions are centralized in shared CI guidance and apply to both roots.
4031036494	Replaced the obsolete fixed-12.9 bot with explicit dispatch of a merged PR to one existing release branch. Labels cannot add targets; no automatic merge is enabled. Documented manual conflict/generated/workflow-file handling and return-to-main responsibility.
4031083254	Retain cuda_bindings_12 as a stable major identity; current/maintenance remain explicit lifecycle roles. A prev rename would obscure identity without simplifying the supported two-role contract; major rollover is documented.
4031089226	Keep the current minor in the source-build --match selector so a new target cannot accidentally derive its version from a preceding minor's reachable tags. The metadata guard makes coordinated registry/packaging updates mandatory.
4031114469	CUDA_PYTHON_BUILD_MAJOR is always required; absent or unsupported values fail instead of defaulting to a major.
4031119779	Keep the standalone metapackage's minor-family selector map because its sdist cannot import repository CI metadata. Validate exact agreement against both bindings roots; keep setuptools-scm's standard parser.
4031120234	Build/test environment values now come from named jq object entries instead of a positional read-variable/array correspondence.
4031135974	Retain separate generated-file ignore lists while the CUDA 12 and current runtime generation layouts materially differ; Mike accepted this bounded choice in the discussion. Consolidation can follow actual layout convergence.
4062051063	Aligned the 12.9.10.dev0 fallback across packaging inputs, then corrected the override to apply only when no tag matching the actual package selector is reachable. Matching earlier regular/prerelease/postrelease tags and descendants now keep standard SCM progression.
4062130293	Metapackage builds use standard setuptools-scm tag parsing, preserving prerelease and postrelease suffixes. Real SCM parser integration cases remain.
4062428956	Release selection rejects noncanonical leading-zero spellings before PEP 440 interpretation, avoiding normalization disagreement. Source builds continue to use the standard SCM parser.
4063090459	A general deterministic equivalence check for handwritten code is unavailable. Require explicit cross-root applicability and reviewer verification; generated-file seals establish content integrity, while revision/input records establish generation provenance.

[rwgk]

The historical 12.9.x branch receives no further routine or emergency backports

One thing I realized about this is that if we do need a backport to 12.9, we're starting from a state where the 13.x code may not currently be producing a green run. Based on my read of this that will incur a full 13.x rebuild/test. There's something a little tricky to me about this coupling that feels like it could cause us to block a 12.x fix on 13.x CI jobs just due to the nature of the situation, though I don't have a great suggestion here.

@brandon-b-miller, GPT-6-Astra ultra implemented a practical compromise. In its own words:

Your concern is real: an ordinary CUDA 12 maintenance PR can still encounter CUDA 13/Core CI failures. The proposal retains that integration testing and adds a release path that can proceed from a tested baseline.

The compromise has three parts:

• Keep normal development on main. Bindings changes continue to exercise affected Core consumers, so integration regressions remain visible.
• Use short release branches when needed. Stabilization or urgent fixes can proceed from a tested commit containing the new package layout, isolated from subsequent unrelated changes on main. Branch-first fixes require a linked follow-up PR to reconcile them back into main, with applicability assessed for both bindings roots.
• Scope release-tag CI to the selected line. A bindings tag builds and tests that bindings package and its matching cuda-python metapackage, using published Pathfinder. Core builds, runtime tests, API checks, and docs are excluded from that release run.

This addresses the release-stage coupling and provides a practical way around unrelated breakage on advancing main. It does not eliminate all coupling: PRs still need their normal review and CI, and repository-wide checks—including Core lint/type checks—still run on release tags.

So the intended guarantee is that a bindings release does not require a Core build, test run, or release. It is not a guarantee that every possible Core problem is irrelevant to the process.

[rwgk]

@leofang Following your offline question about whether we should continue #2737, I asked Codex for a deep analysis of the 12.9.x history, related PRs/discussions, and the generator/QA workflow.

Its conclusion: keep #2737’s direction, combined with explicit release branches for stabilization and urgent fixes.

The main reasons:

• CUDA 12.9 is an actively supported major line. It has received new bindings capabilities, Python support, and ecosystem updates. Maintaining it involves substantial ongoing integration work.
• The history shows recurring coordination costs: diverging CI/release infrastructure, cross-branch artifact dependencies, and generated changes requiring target-specific adaptation. Better backport automation helps, but several of these tasks require more than transporting commits.
• Release branches remain useful. They let us freeze a tested baseline and select fixes while development continues on main.

The proposed division is therefore: develop the two supported majors together on main; stabilize and repair particular releases on branches when needed.

The revised PR implements that direction with focused bindings-release CI, a manual backport workflow, and a short release guide. Normal PRs retain Core integration checks, and generated changes still need explicit applicability review.

That gives us a concrete reason to finish #2737 while continuing to improve branch automation and generator coordination.