Skip to content

Cache the macOS vexctl build in the CI test job - #874

Merged
Mikola Lysenko (mikolalysenko) merged 2 commits into
mainfrom
ci-janitor/cache-macos-vexctl
Oct 5, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 2 commits into
mainfrom
ci-janitor/cache-macos-vexctl

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Every CI run's test (macos-latest) job compiles vexctl, including all of sigstore/cosign, with go install github.com/openvex/vexctl@v0.3.0. It has to compile because the v0.3.0 darwin release binaries are built with go1.22.7, which emits no LC_UUID, and the macOS runner's dyld refuses to load them.

  • Cost: across the last 39 macOS test jobs, Install vexctl took a median of 110s (min 46s, max 168s, mean 106s). These timings come from the step timestamps in GET /actions/runs/<id>/jobs for the most recent 40 completed CI runs. Main and PRs together see about 75 CI runs a day, so this is roughly 2 macOS runner-hours a day of rebuilding the same bytes.
  • Flake source: this compile is the step's only network dependency. Its module verification reads dozens of sum.golang.org tiles, and transient INTERNAL_ERROR stream resets there have failed otherwise-green runs. That is why the step already carries a 5-attempt retry with backoff (see the existing comment in ci.yml).

Root cause

The compiled binary is a pure function of the vexctl version, the Go toolchain that setup-go resolved, and the runner OS/arch. Nothing cached it, so every run rebuilt it.

Fix

Changes are to the test job in .github/workflows/ci.yml only:

  • VEXCTL_VERSION moves to job-level env, so the cache key and the install step read the same value.
  • New step Restore vexctl (macOS) uses actions/cache/restore (SHA-pinned to v5.1.0) on $RUNNER_TEMP/vexctl-bin. The key is vexctl-<version>-go<setup-go resolved version>-<os>-<arch>, so a Go patch bump or a vexctl bump rebuilds.
  • Install vexctl on macOS skips go install on a cache hit. On a miss it compiles exactly as before (same 5-attempt retry), now with GOBIN=$RUNNER_TEMP/vexctl-bin. All three OSes now put the same directory on PATH, and vexctl version runs on every path, so a bad restore fails the step loudly.
  • New step Save vexctl (macOS) runs only on main and only on a miss. This follows the existing "main runs are the only rust-cache writers" policy, so PR runs can't seed the cache.
  • The Linux and Windows paths are unchanged: they still download the sha256-pinned release binary.

The coverage job (Linux) is untouched. The cached binary is about 84 MB uncompressed, well inside the 10 GiB cache budget the Cache cargo comment cares about.

Proof

  • Hit path, run locally: RUNNER_OS=macOS CACHE_HIT=true with a stub binary in $RUNNER_TEMP/vexctl-bin makes the step skip go install, run the restored binary, and write the directory to $GITHUB_PATH.
  • Miss path, run locally (go1.24.7): the step compiles into $RUNNER_TEMP/vexctl-bin/vexctl, vexctl version prints, and the directory lands on $GITHUB_PATH.
  • actionlint: 220 findings before and 220 after, all pre-existing and unrelated (undefined matrix properties in the e2e job). None are in the edited range.
  • Workflow unit tests: scripts/tests/test_ci_e2e_tiers.py, test_ci_vlt_rows.py and test_release.py pass.
  • Expected in CI: PR runs miss the cache until main saves the first entry, so the 110s goes away on the first PR run after this merges. On this PR's own run the macOS compile still happens, with the same timing as today.

Where tests run

No test moves or is removed. tests/e2e_vex.rs still gets a real vexctl on PATH on all three OSes, built from the same version with the same Go pin. The required-check name test (macos-latest) is unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_018DBLy7nL4ekAfds2AURvgc


Generated by Claude Code


Note

Low Risk
Changes are limited to CI workflow steps for test tooling; application code and test selection are untouched.

Overview
The test job in .github/workflows/ci.yml now caches the macOS go install of vexctl so PR runs can skip a ~110s compile and its sum.golang.org flake surface.

VEXCTL_VERSION moves to job-level env, and setup-go gets an id so the cache key includes the resolved Go version. Restore vexctl (macOS) restores $RUNNER_TEMP/vexctl-bin; on a hit, Install vexctl skips go install but still runs vexctl version and puts that directory on PATH. On a miss, macOS builds with GOBIN=$RUNNER_TEMP/vexctl-bin (same 5-attempt retry as before). Save vexctl (macOS) writes the cache only on main after a miss, matching the existing rust-cache policy. Linux/Windows still download the pinned release binaries unchanged.

Reviewed by Cursor Bugbot for commit c4924e6. Configure here.


Generated by Claude Code

The macOS `test` leg compiles vexctl (and all of sigstore/cosign) with
`go install` on every run, because the v0.3.0 darwin release binaries
lack LC_UUID. Over the last 39 macOS test jobs that step took a median
110s (max 168s), and it is the step's only network flake source: the
sum.golang.org tile reads that the 5-attempt retry loop exists for.

The binary depends only on the vexctl version, the Go toolchain that
setup-go resolved and the runner OS/arch, so restore it from an
actions/cache entry keyed on exactly those and skip the compile on a
hit. Like the cargo cache, the entry is saved from main only. Linux
and Windows keep downloading the sha256-pinned release binary; all
three OSes now put the same $RUNNER_TEMP/vexctl-bin on PATH, and
`vexctl version` runs on every path so a bad restore fails the step.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DBLy7nL4ekAfds2AURvgc
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 5, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit c4924e6. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 5, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: labeled Ready for review at c4924e6 (c4924e64d85259a39a91aa1aea894737d5d2bd30).

  • CI: 339/339 green (6 skipped by matrix rule) on the head commit.
  • Bugbot: reviewed c4924e6; no findings.
  • Mergeable against main (checked with git merge-tree after today's main merge wave).

Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) merged commit 9c43dfc into main Oct 5, 2026
70 of 74 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/cache-macos-vexctl branch October 5, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants