Skip to content

fix(cli): return 404 when adding a session to a missing eval set - #7481

Open
ankit2235 wants to merge 2 commits into
google:mainfrom
ankit2235:fix/cli-add-session-404
Open

ankit2235 wants to merge 2 commits into
google:mainfrom
ankit2235:fix/cli-add-session-404

Conversation

@ankit2235

Copy link
Copy Markdown

Link to Issue or Description of Change

2. Or, if no issue exists, describe the change:

Problem:
POST /dev/apps/{app_name}/eval-sets/{eval_set_id}/add-session returns a 500 when the eval set doesn't exist. The endpoint only catches ValueError around add_eval_case, but every eval sets manager (local, GCS, in-memory) raises NotFoundError for a missing eval set, as documented on EvalSetsManager.add_eval_case. NotFoundError isn't a ValueError, so it goes unhandled.

This is already tracked by test_add_session_to_eval_set_unknown_eval_set_is_a_client_error, which was added as a strict xfail in 4912a08 with the reason "add-session maps ValueError, but the managers raise NotFoundError".

Solution:
Catch NotFoundError and return a 404, the same way the update and delete eval case endpoints already do. The existing ValueError -> 400 handling for a duplicate eval case is unchanged. The xfail marker is removed since the test now passes (it's strict, so leaving it would fail the run).

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

With the xfail removed and without the fix, the test fails with:

NotFoundError: EvalSet missing_eval_set not found for app test_app.

With the fix:

pytest tests/unittests/cli/test_fast_api.py::test_add_session_to_eval_set_unknown_eval_set_is_a_client_error
1 passed

pytest tests/unittests/cli -n 4
1442 passed, 5 skipped, 2 xfailed

Manual End-to-End (E2E) Tests:

Not run against a live adk web server. The regression test drives the real FastAPI app through TestClient, which covers the endpoint's error mapping.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

The add-session endpoint only mapped ValueError to an HTTP error, but
every eval sets manager raises NotFoundError when the eval set does not
exist, so the request failed with a 500. Map NotFoundError to 404, the
same way the update and delete eval case endpoints already do.

@yang0228 yang0228 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 579d501eed34823e99b6c0c6af415421639cc388. No blocking correctness findings. Catching NotFoundError matches the local, GCS and in-memory managers' contracts and the adjacent update/delete handlers; the existing duplicate-case ValueError handling remains intact.

Independent validation on Python 3.12.13:

  • Parent commit: the existing regression fails with uncaught NotFoundError when run with --runxfail; PR head: both add-session tests pass.
  • Four additional TestClient probes pass: canonical and legacy routes return exactly 404 for a missing eval set, while successful insertion returns 200 and duplicate insertion remains 400.
  • CLI + eval-set-manager suites: 1,548 passed, 7 skipped, 2 xfailed, 5 setup errors. All five errors also reproduce on the parent commit (agentplatform.frameworks is unavailable in my environment), so this is not an all-green suite claim.
  • Pre-commit passes for both changed files, with no changes to the checkout.

I did not run the full Python-version matrix or a live adk web server. One non-blocking test-strengthening suggestion is attached.

Comment thread tests/unittests/cli/test_fast_api.py
@ankit2235
ankit2235 requested a review from yang0228 October 9, 2026 16:21

@yang0228 yang0228 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed ce679d03762df5315ad44c0470d26029cd829a44 following your update. The regression now asserts exactly 404, addressing my earlier suggestion. The production fix is unchanged from my previous review; no blocking findings.

Fresh validation on Python 3.12.13:

  • Both add-session tests pass (2 passed).
  • Four additional TestClient contract probes pass: canonical and legacy routes return exactly 404 for a missing eval set; successful insertion remains 200 and duplicate insertion remains 400.
  • Pre-commit passes for both changed files, with no changes made by the hooks.

This is a focused follow-up review. I did not rerun the full suite, the Python-version matrix, or a live adk web server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants