Vendored requirements.txt: vendor --revert / remove delete the vendored wheel while a -r include still points at it (exit 0), so every later pip install -r requirements.txt fails #867

[agent] Found by the scheduled pip / requirements.txt bug-hunt routine (ledger #309).

Summary

revert_requirements has a guard for vendor lines that drifted: when a vendor line no longer matches its ledger record, the revert leaves it in place, and a residual-reference sweep keeps the .socket/vendor/pypi/<uuid>/ artifact so the line still installs (vendor_revert_residual_reference → vendor_artifact_kept / vendor_revert_kept). That works when the drifted line is still in the file the ledger recorded.

The sweep only scans the files named in the ledger's wiring records, though. If the user moves the vendored line into a -r include (for example while splitting requirements.txt into requirements/base.txt), the revert:

1. skips the recorded requirements.txt:1 record as vendor_revert_line_drifted;
2. sweeps only requirements.txt, which no longer mentions the uuid, so it finds no residual reference;
3. deletes .socket/vendor/pypi/<uuid>/ (and all of .socket/) and exits 0 with status: success.

requirements/base.txt still carries ./.socket/vendor/pypi/<uuid>/six-1.16.0-py2.py3-none-any.whl # socket-patch vendor: six==1.16.0, so pip fails on every install from then on. remove pkg:pypi/six@1.16.0 goes through the same path and gives the same result.

This is also the remedy socket-patch itself prescribes. In the same layout, a superseding patch's re-vendor fails with pypi_requirements_already_vendored ("requirements/base.txt: already routes six … (requirements.txt: the vendor line changed since vendoring); run socket-patch vendor --revert before re-vendoring"). Following that advice breaks the project.

Impact

After a vendor --revert or remove that reports success, pip install -r requirements.txt fails on every machine and in CI with OSError: [Errno 2] No such file or directory: '…/.socket/vendor/pypi/<uuid>/six-1.16.0-py2.py3-none-any.whl'. The wheel is gone, and the ledger entry is too, so neither vendor --revert nor repair can recover it. The user has to restore it from git or hand-edit the include.

Repro (Linux, main 99f61d2, pip 26.2.1 and 24.0 / CPython 3.11)

This uses a local mock of the patch API and vendoring service (batch / by-package / view/<uuid> plus prebuilt_common::mount_view_from_source over the real installed six.py), run through a throwaway driver that wasn't committed.

python3 -m venv venv && venv/bin/pip install six==1.16.0 idna==3.7
export VIRTUAL_ENV=$PWD/venv
printf 'six==1.16.0\nidna==3.7\n' > requirements.txt
F="--json --yes --api-url $MOCK --api-token fake --org test-org --vendor-url $MOCK --patch-server-url $MOCK"
socket-patch scan --mode vendored $F     # exit 0; line 1 -> ./.socket/vendor/pypi/<uuid>/six-…whl  # socket-patch vendor: six==1.16.0

# The user moves the vendored line into an include (pip resolves it exactly as before)
L=$(head -1 requirements.txt); mkdir requirements
printf -- '-r requirements/base.txt\nidna==3.7\n' > requirements.txt
printf '%s\n' "$L" > requirements/base.txt
socket-patch scan --mode vendored $F     # exit 0, "already_vendored: artifact and lockfile wiring already in sync"

socket-patch vendor --revert --json --yes
# exit 0, status success
# events: skipped vendor_revert_line_drifted ("requirements.txt: the vendor line for requirements.txt:1 changed since vendoring; left untouched"), removed
# no vendor_revert_residual_reference; .socket/ is gone
cat requirements/base.txt                # still ./.socket/vendor/pypi/<uuid>/six-1.16.0-py2.py3-none-any.whl …
python3 -m venv fresh && fresh/bin/pip install -r requirements.txt
# ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '…/.socket/vendor/pypi/<uuid>/six-1.16.0-py2.py3-none-any.whl'   (exit 1)

socket-patch remove pkg:pypi/six@1.16.0 --json --yes in the same state gives the same result: exit 0, vendor_reverted plus vendor_revert_line_drifted, and the artifact is deleted.

Control (same file): when the vendored line stays in requirements.txt but has drifted, because the trailing comment was edited or a space was dropped before #, the revert correctly reports vendor_revert_residual_reference, vendor_artifact_kept and vendor_revert_kept, and the wheel stays. So the guard exists. It just doesn't look past the recorded files.

Expected vs actual

The function's own contract (pypi_requirements.rs, doc comment of revert_requirements): "any surviving reference to the vendored uuid dir afterwards raises vendor_revert_residual_reference", and the sweep comment: "a leftover line pointing at the (about to be deleted) uuid dir would break installs." The #786 in-use probe (requirements_entry_in_use) already treats the requirements tree as "the root requirements.txt plus in-root -r includes".

• Expected: the residual sweep covers the same tree (root plus reachable in-root -r includes). The moved line keeps the artifact (vendor_revert_residual_reference, vendor_artifact_kept), so pip keeps installing.
• Actual: only the recorded files are swept. The artifact is deleted under a live reference, with exit 0.

OS × version

First bad version

Not bisected. The sweep has only ever covered reverted (the recorded files).

Suspect code

• crates/socket-patch-core/src/vendor/pypi_requirements.rs:462-470: the residual-reference sweep iterates &reverted, which holds only the files named in entry.wiring. It should also scan the rest of the requirements tree (requirements_include_names(root), the same walk the Vendored requirements.txt after the user removes or bumps a vendored pin: the rescan re-adds the removed package as a "(transitive)" line (exit 0), or exits 1 forever after a bump, and scan --prune never reverts the entry #786 in-use probe uses) before it allows the artifact to be deleted.
• The same tree-wide check probably belongs in plan_rewire (pypi_requirements.rs:~697), whose doc comment says "present verbatim in an editable file of the tree" but only looks in the recorded file. That's why the superseding re-vendor sends the user to vendor --revert in the first place.