feat(storage): count state cache hits and misses

[MegaRedHand]

Stacked on #658: has_state only consults the state cache from that PR on.

Motivation

Nothing reports whether a state lookup was served from Store's 32-entry state LRU. Two slow paths seen on the followers can only be blamed on cache misses by inference today:

• the epoch-boundary head recompute (0.5-4 s), where checkpoint_state(E-1, R) likely misses and rebuilds the state on the chain actor
• import guards, where a has_state that misses the cache falls through to RocksDB

Change

One counter, lean_state_cache_lookups_total{method, kind, result}:

label	values	why
method	get_state, has_state, cached_state	gossip validation calls cached_state once per attestation and aggregate; summed, it would bury the import path's misses
kind	block, checkpoint	a checkpoint-state miss is the one the epoch-tick head recompute pays
result	hit, miss

Where each lookup is counted:

site	method	LRU op
read_state (behind get_state, and the writer thread's lean parent read)	get_state	get
Store::cached_state (fork choice's checkpoint_state, gossip validation, head state)	cached_state	get
Store::has_state	has_state	peek

read_state and cached_state share a new state_writer::cache_get helper, so the counting lives in one place. rebase_onto_resident's peek is not counted: it picks a base to share memory with, not the state being asked for.

Behavior change: has_state checks the cache before pending_states

This is the same order read_state uses. A state in flight is in both, so no answer changes. With pending_states first, a just-imported parent was answered before the cache check and its lookup went uncounted.

Testing

• cargo test -p ethlambda-storage --profile release-fast --lib: 146 passed
• cargo clippy -p ethlambda-storage --all-targets -- -D warnings: clean
• One-off local check (not committed) that each of cached_state hit (block), cached_state miss (checkpoint), has_state hit/miss and get_state hit/miss increments exactly its own series once. Not committed because the counter lives in the global registry, which every test in the crate shares, so an exact-count assertion would be racy under parallel tests.
• Spec suites not run locally.

[github-actions]

🤖 Kimi Code Review

I'll review this PR which adds metrics for state cache lookups in the storage layer. Let me analyze each file.

crates/storage/src/metrics.rs

Line 47-101: The new metrics code looks well-structured. Good documentation on why method is a label rather than separate series. The StateCacheMethod enum with its label() method is clean and avoids string allocation at the call site.

One minor suggestion: Consider making StateCacheMethod derive strum::Display if you already depend on strum, but the hand-written label() is perfectly fine and more explicit.

crates/storage/src/state_writer.rs

Line 131-142: The new cache_get function correctly promotes entries to most-recently-used via get (not peek), and properly instruments the lookup.

Line 171: The replacement of inline cache access with cache_get is correct and maintains the same LRU promotion behavior.

Potential issue at line 136: cache.lock().unwrap() — this will panic if the mutex is poisoned. This is consistent with the existing codebase pattern (I see the same at line 174 in the original), but be aware that in a storage layer, poisoned mutex recovery might be preferable. However, since this matches existing conventions, it's fine.

crates/storage/src/store.rs

Line 2654-2655: cached_state now uses cache_get with StateCacheMethod::Cached. Correct — this is a memoization lookup where a miss is the caller's responsibility.

Line 2686-2716: The has_state reordering requires careful attention.

Critical correctness issue in has_state (lines 2700-2716):

The comment at line 2697-2699 states: "Cache first, the same order as read_state: a state in flight is in both, so the order changes no answer."

This is incorrect reasoning. The read_state function (in state_writer.rs, lines 158-178) checks pending_states after the cache miss:

// read_state path:
if let Some(state) = cache_get(...) { return Ok(Some(state)); }  // cache first
if let Some(state) = pending.get(root) { return Ok(Some(state)); }  // pending second
// then backend

But has_state now checks cache first, then pending, then backend. The comment claims this is "the same order as read_state" — but read_state was already cache-first. The original has_state was pending-first:

// ORIGINAL has_state:
if self.pending_states.get(root).is_some() { return Ok(true); }  // pending FIRST
if self.state_cache.lock().unwrap().peek(...).is_some() { return Ok(true); }  // cache SECOND

The PR changed has_state from pending-first to cache-first, and the justification is flawed. Let me re-verify...

Actually, re-reading: the PR comment says "Cache first, the same order as read_state". The read_state function is cache-first. The original has_state was pending-first for a reason stated in the original comment: "Same pending-before-backend order as read_state; see its doc for why the backend never has to consult pending_states on its own."

Wait — let me re-read read_state more carefully. In state_writer.rs:

if let Some(state) = cache_get(cache, key, StateCacheMethod::Get) {
    return Ok(Some(state));
}
if let Some(state) = pending.get(root) {
    return Ok(Some(state));
}

So read_state is cache-first, pending-second. The original has_state was pending-first, cache-second. The PR changes has_state to cache-first, pending-second.

Why was has_state pending-first originally?

Looking at the original comment: "Same pending-before-backend order as read_state; see its doc for why the backend never has to consult pending_states on its own."

This is about the backend not needing to check pending_states, not about cache vs. pending ordering. The read_state function checks cache, then pending, then backend. The original has_state checked pending, then cache, then backend.

The PR's change to cache-first in has_state is actually correct for consistency with read_state, but the comment's justification is slightly confused. The key insight is: a state in flight is in both cache and pending (as the comment notes), so order doesn't change the answer.

However, there's a performance subtlety: if a state is in pending_states but not yet in cache, the original code would find it faster (pending is a DashMap, typically faster than Mutex<LruCache>). The new code pays for the cache lock first. But the PR's goal is metrics coverage, and this ordering ensures every call is instrumented.

More importantly: The peek vs get distinction is preserved correctly — has_state uses peek (no LRU promotion), while cache_get uses get (promotes). This is semantically correct for existence checks.

docs/metrics.md

Documentation is clear and includes important guidance about reading the metric per-method rather than summed. Good operational context about what different miss patterns indicate.

---

Summary

Item	Severity	Description
1	Minor	has_state comment at line 2697-2699 slightly mischaracterizes the ordering change — it says "same order as read_state" but read_state was already cache-first; the change is making has_state match read_state, not noting they were already the same. The comment could be clearer: "Now cache-first, matching read_state's order, so every lookup is instrumented."
2	Info	Consider whether cache_get should return Option<Arc<BeaconState>> or be split into cache_get and cache_peek variants to avoid the cloned() overhead when the caller only needs existence (but has_state correctly uses direct peek for this).

No security vulnerabilities or consensus-critical bugs found. The metrics addition is purely additive (no behavior changes except LRU ordering in has_state, which is harmless due to the "in both" property). The peek vs get semantics are correctly maintained.

The code is well-structured, properly documented, and follows Rust best practices. The metrics design with per-method labeling is appropriate for the differing call rates and operational significance.

---

Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

[github-actions]

🤖 Claude Code Review

Review of PR 659: count state cache hits and misses

The PR looks good and I found nothing blocking. It adds one IntCounterVec, lean_state_cache_lookups_total{method,kind,result}. A shared cache_get helper counts lookups for get_state, has_state and cached_state. The method label keeps gossip validation's high-rate cached_state calls from drowning out the import path's misses. The doc note telling readers to read the counter per method is useful for the same reason.

I reviewed the diff only and did not build it or run tests.

Points worth a look:

1. has_state reorder (store.rs, around line 2700).

   • Checking the cache before pending_states returns the same answers as before, as long as the comment's claim holds: any cached block state is also in pending_states or the backend.
   • The cost is that a state found only in pending_states now counts as a has_state miss. The metric then reads as "not resident in cache" and not as "had to hit storage". The new docs already say "came from the writer's buffer or RocksDB", so this is accurate, just worth knowing when reading dashboards.

2. CacheKey taken by value in cache_get. This is fine since CacheKey is cheap and Copy-like. The key is still used afterwards for the kind label, and the code compiles on that basis.

3. Per-call with_label_values (metrics.rs). Each call hashes three label strings. cached_state is hot, since it runs once per attestation and aggregate. It is a small cost next to the mutex lock and Arc clone. If it ever shows up in a profile, pre-resolve the 12 label combinations.

4. Dependency direction. metrics.rs now imports state_writer::CacheKey, and state_writer.rs imports StateCacheMethod. Rust allows the cycle, but the dependency now runs both ways. Taking a kind: &'static str or a small enum would avoid it. This is minor.

5. Has label comment. The StateCacheMethod::Has doc says a miss falls back to the write buffer and then to storage. That matches the new code order, so it is consistent.

Docs and metric naming follow the project conventions (lean_ prefix, LazyLock). The change is approved from my side.

---

Automated review by Claude (Anthropic) · sonnet · custom prompt

[github-actions]

🤖 Codex Code Review

Looks good overall. This PR is observability-only: it adds cache-hit/miss metrics without changing fork choice, attestation processing, state transition, XMSS, or SSZ logic, and I don’t see a consensus-safety regression in the touched paths.

• No correctness bug found in the metric wiring: read_state now counts get_state lookups via cache_get in crates/storage/src/state_writer.rs:136, cached_state does the same in crates/storage/src/store.rs:2656, and has_state still uses peek so it does not perturb LRU ordering in crates/storage/src/store.rs:2705.
• The has_state reordering is behavior-preserving for existence checks. The comment is accurate that a resident state can be in both cache and pending_states, so checking cache first changes accounting, not the returned value, in crates/storage/src/store.rs:2700.
• Security-wise, nothing here affects validation or persistence semantics; it only increments Prometheus counters after in-memory cache probes. I don’t see a new attack surface beyond the existing mutex contention on the shared state cache.

One minor performance/readability note:

• crates/storage/src/store.rs:2705 / crates/storage/src/store.rs:2706: has_state duplicates the “lookup + metric” pattern instead of sharing a helper for non-promoting cache probes. A tiny cache_peek(...) helper alongside cache_get(...) in crates/storage/src/state_writer.rs:136 would reduce drift risk if labels or counting semantics change later. Not blocking, just a maintainability cleanup.

So: brief ack from my side — the patch looks sound.

---

Automated review by OpenAI Codex · gpt-5.4 · custom prompt