Make the vendored-to-hosted takeover atomic

[Mikola Lysenko (mikolalysenko)]

Problem

Architecture audit B03: scan/get --mode hosted over a vendored package (the vendored→hosted takeover) reverted the vendored wiring on disk first and only then planned the hosted pin. When the hosted rewriter refused (a lock-level refusal, a missing berry checksum, unavailable wheel metadata, a Poetry 0.x lock, a uv pin-down, ...), the package was left unpatched in both modes. Hand-copied pre-gates for bun, berry, classic, vlt, Gradle, pypi platform wheels and requirements tried to predict those refusals, and they drifted from the rewriters they copied (the root cause behind #945, #723 and the closed #699). The hosted→vendored direction is already atomic (#963).

• B14: hosted writes were non-transactional. Files were written one by one, and an error exit after the takeover still said "nothing was written".
• B37 (takeover part): the dry run counted every takeover as redirected without running the rewriter.
• B15 (takeover list): the disk flow took over cargo/npm/golang/pypi/Gradle maven, but the in-memory engine refused only cargo/npm/golang.

Change

The takeover now runs inside the run's GroupCommit (commands/scan/hosted/takeover.rs):

1. Each vendored revert is staged in the overlay under a savepoint. A revert that fails or keeps drifted wiring is rolled back and refused, as before.
2. The hosted rewrite reads the overlay, so it plans against the reverted project.
3. A staged purl the rewrite does not pin is retracted. The overlay goes back to its pre-revert state, the purl stays vendored byte for byte, the rest are staged again and the rewrite runs again. Each pass drops at least one purl, so the loop ends. The retracted purl is skipped with its cause (an existing skip reason, the rewriter warning that names it, redirect_requirements_takeover_unreachable, the rewriter's lock-level refusal, or redirect_takeover_not_pinned), followed by the new redirect_takeover_kept_vendored warning. Exit 0.
4. The hosted pins and the vendored ledger go into the same overlay and are committed once (journaled). Reverted artifacts are deleted only after the commit (GroupCommit::defer_removals; the per-unit revert removal goes through remove_tree_and_prune, bun workspace tarballs through remove_mirror, and the JVM revert's Gradle/Maven tree files through group_commit::defer_removal). The JVM-owned .gitattributes files and the derived maven-metadata.xml files are now in the captured set, so the Gradle revert's writes stay in the overlay too.
5. --dry-run runs the same steps and drops the overlay instead of committing, so it reports the wet outcome (B37).
6. A hosted run with no takeover also stages its writes, but commits them unjournaled (commit_unjournaled) because hosted mode writes nothing under .socket/vendor/. If one replacement fails, the files already replaced are put back (B14). The few files the group does not capture (the Gradle hosted index and script under .socket/gradle/) are written just before the commit and put back if a later step or the commit fails.
7. B15: hosted::takeover::in_reach is the one predicate both engines use.

Duplicate copies deleted

• Takeover pre-gates that copied rewriter logic: 9 → 0 (bun text/binary, berry lock, berry dep, classic, vlt, Gradle takeover_refusal, pypi platform wheel, requirements reach). preflight_requirements_takeover only explains a retraction now. Review found the Gradle gate was not yet safe to remove, because the JVM revert deleted the vendored tree straight from disk. That is fixed in this PR (see below), so the count stands. preflight_yarn_classic_hosted is now private. After merging main's Fix yarn berry project gates drifting between modes (#628, #629) #657, which moved the berry project gates into the shared formats/yarn/berry_gates, the takeover again checks yarn-berry entries against those gates on the pre-revert project. It calls the rewriter's own preflight_yarn_berry_hosted once per staging pass, so no logic is copied. This is needed because the berry revert re-renders package.json in its majority line ending, so after the revert the rewriter could no longer see the mixed manifest that Hosted yarn berry rewrites a mixed-line-ending package.json that vendored mode refuses #628 refuses. Copied pre-gates: 9 → 0. Shared-gate call sites: 1.
• Takeover-capable lists: 2 → 1 (stages.rs closure + hosted.rs closure → hosted::takeover).
• Dry-run takeover preview path: the engine's TakeoverPreview install-policy preview and the CLI's withheld-and-counted preview are deleted. The dry run is the real rewrite now.
• Stranded-takeover reporting (redirect_takeover_unpatched, partial_failure, unrecorded) is deleted, because a takeover can no longer strand.
• The cargo/golang revert removal copies (remove_tree + prune_empty_vendor_levels) now use the shared remove_tree_and_prune.

Behavior changes (documented in CLI_CONTRACT.md)

• A takeover the rewriter would not pin keeps the package vendored with exit 0. Before, it was stranded with exit 1.
• A failed write or commit changes nothing and exits 1 with "nothing was changed". Before, the takeover and earlier locks were already on disk.
• A dry run over a vendored pnpm project now previews trustLockfile as written to a new pnpm-workspace.yaml, which is what the wet run does: the revert removes the vendor-created file. The old preview said "merged into the existing" one.
• With a yarn-offline-mirror, the yarn classic hosted rewriter no longer also warns redirect_yarn_classic_berry_migration_risk, because the mirror refused every pin and nothing was pinned. That warning came from Fix hosted yarn classic pins missing berry warning (#907) #917 on main. Without this fix, a retracted classic takeover reported the berry warning as its cause instead of redirect_yarn_classic_offline_mirror.
• A vlt takeover over a BOM'd lock is refused by the staged revert itself (vendored_revert_failed) instead of the deleted vlt pre-gate. The state stays the same.

Testing

All commands were run in the worktree with CARGO_INCREMENTAL=0 through the shared limiter, -j4, on the branch after merging origin/main (431b818, head 830726e):

• cargo test -p socket-patch-core --lib: 5608 passed, plus the new mirror test.

• cargo test -p socket-patch-cli --lib: 869 passed.

• cargo test --no-fail-fast -p socket-patch-cli with --test set to each of: coverage_fix_scan_hosted_dryrun_vendored, e2e_golang_hosted_state, covgap_commands_scan_hosted, in_process_redirect, hosted_memory_engine, mode_migration_pypi, in_process_vendor, in_process_vendor_bun_takeover, in_process_vendor_npm_v1_takeover, in_process_vendor_pnpm_takeover, in_process_vendor_pypi_takeover, vendor_eject, vendor_jvm_cli, contract_gradle_codes, gradle_agent_cli, e2e_sbt_vendor, e2e_scala_cli_vendor, covgap_commands_vendor, covgap_commands_vex, e2e_vex_redirect, e2e_redirect_gradle_build, e2e_vendor_gradle_build. All passed. The real-Gradle tests are #[ignore]d.

• After the second merge and the berry gate: --lib, in_process_vendor (121, including Fix yarn berry project gates drifting between modes (#628, #629) #657's berry_takeovers_refuse_before_reverting_the_old_mode), mode_migration_npm (19, real yarn), mode_migration_pypi, mode_migration_cargo, mode_migration_bun, mode_migration_vlt, in_process_alternate_installers, remove, repair, e2e_redirect_gem_stale_install and the takeover files above all passed.

• mode_migration_cargo::vendored_then_hosted_takeover_leaves_pure_hosted (real cargo) and mode_migration_npm::{berry,classic}_vendored_then_hosted_takeover_leaves_pure_hosted (real yarn): passed.

• cargo clippy -p socket-patch-core -p socket-patch-cli --all-targets -- -D warnings: no findings in code this PR changed, but it fails on 20 pre-existing lints in files this PR does not touch, or in lines it did not write (checked with git blame against the merge-base):

  • python_crawler.rs:2760 unused unix_default, and :5375
  • patch/apply.rs:3736
  • jvm_jar.rs (8 × from_ref)
  • redirect/mod.rs:10292 (test)
  • upstream/uv.rs:2097
  • bun_binary.rs:1051
  • maven_repo.rs:2503
  • nuget_feed.rs:2005
  • pnpm_lock.rs:3877
  • yarn_berry_lock.rs:1483
  • yarn_layering_tests.rs:1023,1028
  • The CLI test helpers prebuilt_common/common (module loaded twice, needless borrows).

  With -A unused_variables so the core library compiles, the CLI src/ has no findings.

• Not run locally: the real-Gradle leg e2e_redirect_gradle_build -- --ignored gradle_hosted_ (including gradle_hosted_takeover_refusal_keeps_vendored). This machine has no JDK, so it is left to the Gradle CI leg. The same staged revert is covered in-process by the new core test below.

New or flipped regression tests (each fails without its fix):

• vendor::jvm::gradle::tests::a_staged_revert_changes_nothing_until_its_group_commits covers the review blocker. It vendors a Gradle patch, with and without a sibling version that shares maven-metadata.xml, then stages the real revert in a defer_removals group. Dropping the group (the dry run) or rolling it back (the retracted takeover) leaves the full tree byte-identical, and committing lands exactly the plain revert. Before the fix, the drop case deleted the jar, pom, marker, metadata and both .gitattributes.
• Full-tree snapshots (.socket/ and the vendored artifacts included) now wrap the dry-run takeover in coverage_fix_scan_hosted_dryrun_vendored (pnpm, package-lock, vlt), e2e_golang_hosted_state::hosted_takeover_of_vendored_module_removes_vendored_state (golang, via get --dry-run) and mode_migration_cargo::vendored_then_hosted_takeover_leaves_pure_hosted (cargo). Bun (mode_migration_bun, assert_unchanged) and uv (uv_takeover_without_wheel_metadata_keeps_the_package_vendored, which includes the wheel) already compare the whole state.
• patch::redirect::tests::yarn_classic_offline_mirror_refusal_skips_the_berry_risk_warning. It fixes in_process_vendor::classic_vendored_to_hosted_takeover_refuses_with_offline_mirror, which failed after merging main's Fix hosted yarn classic pins missing berry warning (#907) #917.
• From the first round: mode_migration_pypi::uv_takeover_without_wheel_metadata_keeps_the_package_vendored, a_crash_inside_the_takeover_commit_is_finished_by_the_next_run, ledger_update_failure_changes_nothing, covgap_commands_scan_hosted::ledger_save_failure_after_successful_revert_fails_closed, in_process_redirect::partial_lockfile_write_failure_exits_1_and_writes_no_ledger, hosted_memory_engine::vendored_pypi_takeover_is_refused_like_the_disk_flow, group_commit::tests::deferred_unit_removals_wait_for_the_commit.

mode_migration_npm::berry_vendored_then_hosted_takeover_leaves_pure_hosted now expects a hosted resolutions entry in package.json instead of a pristine one. I checked it on origin/main (431b818, real yarn) in a detached worktree, and it already fails there at mode_migration_npm.rs:1223 ("the berry resolutions entry must be reverted"). Since #465, main's hosted berry pin writes resolutions into package.json. So the old assertion was stale, and this PR does not change that behavior.

Deferred

• B30 (hosted meeting vendored: one pre-rewrite decision for Composer in-place rewrite and the NuGet key collision, mod.rs:5525,5552 vs nuget_feed.rs:219) is not in this PR. Composer/NuGet are not takeover ecosystems, and that decision needs C34.
• Poetry 0.x vendored → hosted takeover un-vendors the package before hosted mode refuses the lock, while --dry-run previews a clean takeover (redirected: 1, exit 0) #945 / uv vendored → hosted takeover strands a package that vendored mode pinned to a different version than uv.lock: the wet run reverts to the unpatched release (exit 1), while --dry-run previews a clean takeover #723 (uv pin-down, Poetry 0.x) should now be retracted generically instead of stranded, but this PR has no fixtures for them. They are not claimed, are not in the agent-status block, and are left to Fix PyPI hosted takeover un-vendoring before refusal (#723, #945) #946.
• The per-ecosystem dry-run snapshot covers pnpm, package-lock, vlt, bun, uv, golang and cargo through the CLI, and Gradle only in-process. yarn classic/berry and the other pypi flavors are still checked on the lock and ledger only. Audit of the in-reach reverts: every artifact deletion in npm-family, pypi, cargo, golang and bun goes through remove_tree_and_prune or remove_mirror (both deferring). JVM was the only backend deleting uncaptured .socket/ files directly.
• Residuals:
  • A crash inside the journaled commit leaves the reverted artifact as an orphan. The next run finishes the commit but does not delete the artifact; vendor --revert's sweep does.
  • The Gradle hosted index and script (.socket/gradle/hosted-index.tsv, socket-patch.hosted.settings.gradle) stay outside the captured set. They are written just before the commit and put back on failure, but a crash in that window leaves them on disk; the captured settings line is what makes them live.
  • The rollout budget can count a takeover that is later retracted.
• Exit policy (Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704) is unchanged: a failed commit keeps the existing hosted error envelope.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes core hosted-mode lockfile and vendor-ledger mutation paths and exit/status contracts; behavior is safer when redirects fail but regressions could affect multi-package takeovers or crash recovery.

Overview
Vendored → hosted takeover is now staged and committed atomically instead of reverting vendored wiring on disk before planning hosted pins. Hosted scan/get open a GroupCommit, run takeover logic in new hosted/takeover.rs, stage each revert under savepoints, plan the hosted rewrite against the overlay, and retract any package the rewriter would not pin (overlay rolled back; package stays vendored with redirect_takeover_kept_vendored, exit 0). Successful runs commit vendored reverts, lockfile redirects, and ledger updates in one step; artifact deletes are deferred until after commit.

Hosted writes are transactional: takeover runs use a journaled commit; hosted-only runs use commit_unjournaled with rollback on partial failure. --dry-run runs the same pipeline and drops the overlay, so previews match wet runs without touching disk.

Removed the stranded-takeover path (redirect_takeover_unpatched, partial_failure exit 1) and ecosystem-specific takeover pre-gates copied from rewriters; hosted::takeover::in_reach is shared with the in-memory engine. CLI_CONTRACT.md documents staged takeover, new skip/warning codes, and updated failure semantics (e.g. ledger save failure leaves the project unchanged).

Tests add full-tree dry-run snapshots and flip expectations for ledger/commit failures and partial lock writes.

^{Reviewed by Cursor Bugbot for commit 406e111. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 406e111. Configure here.}