Skip to content

feat: per-channel SparseTerms sum and phase-centre shifts in sparse_terms_from_chunks (streaming P5) - #601

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/streaming-p5-cubes-phase-centre
Oct 1, 2026
Merged

Jammy2211 merged 2 commits into
mainfrom
feature/streaming-p5-cubes-phase-centre

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

Streaming visibilities phase 5, the last phase of the streaming-visibilities epic (Discussion https://github.com/orgs/PyAutoLabs/discussions/13; issue #600). Two asks from the discussion land here:

  • Per-channel cubes. MFS (multi-frequency synthesis) SparseTerms are the sum of the per-channel terms. SparseTerms.__add__ already summed field-wise; this PR adds __radd__ so sum(list_of_terms) works and proves the identity: three channels accumulated separately and summed equal one accumulation over all chunks and the in-memory MFS apply_sparse_operator operator and scalars at rel 1e-12, in numpy and use_jax=True; an array-free dataset built from the summed terms reproduces the in-memory MFS inversion and log_evidence at 1e-10.
  • Phase-centre shifts, chunk by chunk. sparse_terms_from_chunks(..., phase_centre=(y0, x0)) (arcsec, autoarray (y, x) order) multiplies each chunk's visibilities by exp(+2πi(u l0 + v m0)) (l0 = x0, m0 = y0 in radians) before the dirty image is formed, so a source at (y0, x0) lands at the image origin. Threaded through Interferometer.from_stream. Only dirty_image_native changes; W̃, the dirty beam, sum_weights, data_term and noise_normalization are bit-identical to the unshifted run (asserted). The sign and the (y, x) order are pinned by a DFT point-source test: a source at native pixel (3, 2) of an 11×11 mask peaks at the mask centre (5, 5) after phase_centre=(1.0, −1.5), and at (8, 0) with the opposite sign.
  • phase_centre is recorded as SparseTerms provenance and checked in __add__, so shifted and unshifted channel terms cannot be summed silently. Unshifted accumulations record (0.0, 0.0); None means "not recorded" (hand-built terms) and still adds.

The companion autolens_workspace PR adds the array-free datacube example (scripts/interferometer/features/datacube/modeling_array_free.py, smoke-listed).

API Changes

Additive: SparseTerms.phase_centre (provenance), SparseTerms.__radd__, sparse_terms_from_chunks(phase_centre=), Interferometer.from_stream(phase_centre=). Behaviour change: SparseTerms.__add__ raises on differing recorded phase centres, and accumulations without a shift now record phase_centre=(0.0, 0.0).
See full details below.

Review (Codex gpt-6-astra, 2026-10-01)

Five findings, each reproduced before editing. Fixed in the second commit: F1 (high) the PyAutoGalaxy dataset.fits round trip dropped phase_centre (reloaded None summed with anything) → persisted losslessly in the companion PyAutoGalaxy PR; F2 (high) apply_sparse_operator_from_chunks(phase_centre=) desynchronised the operator from the retained data (sparse χ² 0 vs residual χ² 2 on a one-visibility case) → typed raise; F4 (medium) data_term came from unshifted data while the dirty image used shifted data (Codex case: 1e8 vs 99998200.02, Δχ² 1800 with σr≈σi inside the 1e-5 tolerance) → shift applied before every data-dependent term; plus the jax leg of the MFS test now exercises the JAX kernel builder (spy-asserted) and __add__ checks transformer_class_name. F3/F5 are pre-existing (NUFFT ignores a nonzero mask origin; provenance cannot see masks with the same geometry but different masked pixels) → Mind draft draft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md.

Test Plan

  • New (14): sum of channels == MFS (numpy, jax); __radd__ accepts 0 only; phase-centre accumulation == in-memory on pre-shifted data (NUFFT, DFT) with unshifted terms bit-identical; DFT point-source re-centring (sign + order); __add__ phase-centre mismatch raises / unrecorded-left keeps right; from_stream(phase_centre=) matches the pre-shifted in-memory dataset (operator 1e-12, inversion + log_evidence 1e-10); from_sparse_terms(sum(per-channel terms)) matches the in-memory MFS dataset.
  • test_autoarray: 1917 passed (main: 1903).
  • CI green on all legs.
Full API Changes (for automation & release notes)

Added

  • aa.SparseTerms.phase_centre: Optional[Tuple[float, float]] = None — provenance, arcsec, (y, x); (0.0, 0.0) for unshifted accumulations, None = not recorded.
  • aa.SparseTerms.__radd__(other) — returns self for other == 0 so sum(list_of_terms) works; NotImplemented otherwise.
  • aa.util.inversion_interferometer.sparse_terms_from_chunks(..., phase_centre=None) — per-chunk data *= exp(+2πi(u·l0 + v·m0)), l0 = x0, m0 = y0 in radians (arcsec input); dirty image re-centred on (y0, x0); all other terms unchanged.
  • aa.Interferometer.from_stream(..., phase_centre=None) — forwards to the accumulator.

Changed Behaviour

  • aa.SparseTerms.__add__ — raises InversionException when phase_centre is recorded on both operands and differs (as for origin / pixel_scales / shape_native / eps).

  • sparse_terms_from_chunks — records phase_centre=(0.0, 0.0) when no shift is given (was unrecorded).

  • Interferometer.apply_sparse_operator_from_chunks(..., phase_centre=…) — now raises DatasetException (it would shift the operator's dirty image but not the retained data); use from_stream(phase_centre=…).

  • sparse_terms_from_chunks(phase_centre=…) — data_term is formed from the shifted visibilities (equal to the unshifted value to rounding when σr = σi exactly), so every data-dependent term describes the same visibilities.

  • SparseTerms.__add__ — also refuses terms whose recorded transformer_class_name differs.

Heart RED development override (human-authorized, 2026-10-01)

Shipped under the AUTONOMY.md "Human override for Heart RED (development only)". Live human authorization in the CLI session (Fable 5.1), quoted: "Yes: library now, workspace when its run passes" for task streaming-p5-cubes-phase-centre (#600). Exact RED reasons at ship time (pyauto-heart readiness):

  • release validation FAILED (stage integrate)
  • workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py)
  • manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml

None relate to this branch. Branch gates passed: test_autoarray 1917, test_autogalaxy 1307; Codex review 5 findings (3 introduced → fixed + red-checked, 2 pre-existing → filed). Scope: push + PR-open only. Merge requires a separate human /prm with every required check green; not a release; no claim that Heart is healthy.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

Jammy2211 and others added 2 commits October 1, 2026 10:20
…erms_from_chunks (streaming P5)

- SparseTerms gains `phase_centre` provenance ((y, x) arcsec), checked in
  __add__ like origin/eps, and __radd__ for 0 so sum(list_of_terms) works
  (MFS terms = sum of per-channel terms).
- sparse_terms_from_chunks(phase_centre=...) multiplies each chunk's
  visibilities by exp(+2 pi i (u l0 + v m0)) before forming the dirty image,
  re-centring a source at (y0, x0) onto the origin; every other term is built
  from the unshifted chunk and is bit-identical. Unshifted accumulations
  record (0.0, 0.0), so shifted and unshifted terms refuse to be summed.
- Interferometer.from_stream forwards phase_centre.
- Tests: channel sum == one accumulation == in-memory MFS (numpy + JAX),
  phase centre == in-memory on pre-shifted data (NUFFT + DFT), DFT
  point-source sign/order pin, provenance checks, array-free inversion parity.

Refs #600

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…a path, check transformer provenance (streaming P5 review)

- sparse_terms_from_chunks applies the phase-centre shift to the data before
  every data-dependent term, so the dirty image and data_term describe the
  same shifted visibilities (with sigma_r ~= sigma_i within the check's
  rtol, the unshifted data_term was off: 1e8 vs 99998200.02 on a
  quarter-turn baseline). data_term is phase-invariant only for exactly
  equal sigmas; the P5 test now compares it at rel 1e-12.
- Interferometer.apply_sparse_operator_from_chunks rejects phase_centre with
  a DatasetException pointing at from_stream: it shifted the operator but
  not the retained data (sparse chi2 0 vs 2 against the retained data).
- SparseTerms.__add__ checks transformer_class_name with the other recorded
  provenance (mismatch raises, unrecorded skips).
- The MFS channel-sum test's JAX leg now runs the JAX brute force
  (method="numpy" + use_jax=True, spied); method="nufft" ignores use_jax.

Refs #600

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211

Copy link
Copy Markdown
Collaborator Author

Workspace PR: PyAutoLabs/autolens_workspace#582 (array-free datacube example; merges after this PR under the library-first gate).

@Jammy2211
Jammy2211 merged commit ffd13ba into main Oct 1, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the feature/streaming-p5-cubes-phase-centre branch October 1, 2026 10:26
@Jammy2211 Jammy2211 removed the pending-release PR queued for the next release build label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant