Skip to content

fix: audit and repair broken doc URLs across PyAuto repos #508

Closed
Closed
fix: audit and repair broken doc URLs across PyAuto repos#508

Description

@Jammy2211
Jammy2211
opened on May 14, 2026

Overview

Documentation URLs across the PyAuto repos have rotted. The Google Colab links in the PyAutoLens new-user guide include a literal hhttps:// typo; dozens of Jammy2211/<workspace> GitHub paths now 301-redirect to PyAutoLabs/<workspace>; and they reference a release branch that no longer exists on either owner. There is also no automated check to catch this rot before users hit it. This task adds a reusable URL-audit tool to admin_jammy/software/ and uses it to clean up every broken doc URL across the 12 PyAuto repos.

Plan

  • Build a Python URL-checker in admin_jammy/software/url_check/ that scans repo paths, dedupes URLs, and HEAD/GET-validates them in parallel. For colab.research.google.com/github/<owner>/<repo>/blob/<ref>/<path> links, validate the underlying raw.githubusercontent.com/<owner>/<repo>/<ref>/<path> instead — Colab returns HTTP 200 even for dead refs, so the colab URL itself can't be trusted.
  • Run the audit across all PyAuto repos and produce a structured Markdown + JSON report listing each broken URL with repo:file:line locations.
  • Apply scripted fixes for the three known broken patterns: hhttps:// → https://; Jammy2211/{autolens,autogalaxy,autofit}_workspace → PyAutoLabs/...; /blob/release/ → /blob/main/.
  • Investigate and fix any remaining broken URLs the audit flags (arxiv DOIs, readthedocs paths, dead external refs).
  • Ship per-repo PRs. Doc-only changes have no API impact, so library-first gating doesn't apply.
Detailed implementation plan

Affected Repositories

  • PyAutoLens (primary — where the bug was reported)
  • admin_jammy (tool lives here)
  • PyAutoConf, PyAutoFit, PyAutoArray, PyAutoGalaxy
  • HowToFit, HowToGalaxy, HowToLens
  • autofit_workspace, autogalaxy_workspace, autolens_workspace

Work Classification

Both (library docs + workspace tutorial scripts). Library-first gate not required: doc-only, no API surface.

Branch Survey

Repository Current Branch Dirty?
admin_jammy main 2 unrelated (papers.md, week.md)
PyAutoConf main clean
PyAutoFit main clean (feature/priors-jax-native lives in another worktree)
PyAutoArray main clean
PyAutoGalaxy main clean (feature/fit-ellipse-jax lives in another worktree)
PyAutoLens main clean
HowToFit main clean
HowToGalaxy main clean
HowToLens main 2 regenerated dataset files (ignored)
autofit_workspace main clean
autogalaxy_workspace main 8 regenerated dataset files (ignored)
autolens_workspace main 17 regenerated dataset files (ignored)

Conflict status: worktree_check_conflict flags PyAutoFit and PyAutoGalaxy as claimed by other tasks, but those tasks edit source code and this task only touches docs/ and tutorial scripts — explicitly cleared by the user to proceed in parallel.

Suggested branch: feature/url-check
Worktree root: ~/Code/PyAutoLabs-wt/url-check/ (created later by /start_library)

Implementation Steps

  1. Build the audit tool — admin_jammy/software/url_check/url_check.py:

    • Regex-extract URLs from *.md, *.rst, *.py, *.ipynb, *.txt, *.yml, README* (strip trailing ).,;!)
    • Dedupe; keep {url: [(repo, file, line), ...]} index
    • Parallel HEAD requests (~16-thread pool); fall back to GET on 405/403
    • Colab handler: rewrite colab.research.google.com/github/<owner>/<repo>/blob/<ref>/<path> → raw.githubusercontent.com/<owner>/<repo>/<ref>/<path> and HEAD-check that
    • Categories: ok (2xx), redirect (3xx — record final URL), broken (4xx/5xx/timeout/DNS)
    • Output: admin_jammy/software/url_check/report.md + report.json
    • --fix-known-patterns flag runs the scripted rewrites listed below
    • run.sh wrapper invokes the audit against all PyAuto repos by default

  2. Scripted fixes for the known patterns:

    • hhttps:// → https:// (2 known sites: PyAutoLens/docs/overview/overview_2_new_user_guide.md:31, PyAutoGalaxy/docs/overview/overview_2_new_user_guide.md:17)
    • github.com/Jammy2211/{autolens,autogalaxy,autofit}_workspace → github.com/PyAutoLabs/... (~113 sites)
    • /blob/release/notebooks/ → /blob/main/notebooks/
    • Verify each rewrite returns 200 before committing

  3. Case-by-case fixes for whatever the audit flags beyond known patterns (arxiv, readthedocs, ADS, JOSS, missing PNG image URLs from raw.githubusercontent.com/...).

  4. Colab verification pass: confirm every Colab URL's underlying notebook exists. Tag-pinned URLs (blob/<version-tag>/...) are left as-is if the tag resolves; flagged if not.

  5. Ship per repo as feature/url-check PRs via /ship_library or /ship_workspace.

Key Files

  • admin_jammy/software/url_check/url_check.py — new audit tool
  • admin_jammy/software/url_check/run.sh — wrapper
  • admin_jammy/software/url_check/report.md + report.json — generated artifacts
  • PyAutoLens/docs/overview/overview_2_new_user_guide.md — typo + pattern fixes
  • PyAutoGalaxy/docs/overview/overview_2_new_user_guide.md — same
  • Other docs/*.{md,rst} across libraries — pattern rewrites
  • Workspace *.py tutorial scripts — pattern rewrites
  • README files across repos — pattern rewrites

Testing

  • Re-run the audit after fixes; expect zero broken, only acceptable redirects.
  • Manually open 3-5 fixed Colab links to confirm load.
  • No code/test changes needed (doc-only).

Original Prompt

Click to expand starting prompt 
URL links to Google colabs are not correct, linking to broken colabs which dont run, for example
in the PyAutoLens docs:

https://pyautolens.readthedocs.io/en/latest/overview/overview_2_new_user_guide.html

Here:

CDD Imaging: For image data from telescopes like Hubble and James Webb, go to imaging/start_here.ipynb.


Can you scan all repos for all URLs and check they work, in particular making sure the google colab pages
that come up actrually run?

Activity

  1. Jammy2211 commented on May 15, 2026

    @Jammy2211
    Jammy2211
    on May 15, 2026
    CollaboratorAuthor

    Library phase shipped — 6 PRs open across the libraries + tooling.

    The audit tool now lives at admin_jammy/software/url_check/ and is reusable. We scanned all 12 PyAuto repos (~3000 unique URLs, ~125k locations), then applied ~20 classes of scripted rewrites driven by the patterns the audit surfaced.

    What the audit found before fixes: 304 broken URLs, including the hhttps:// typo you spotted plus a long tail of stale Jammy2211/<workspace> paths, references to a release branch that no longer exists on the workspaces, the joshspeagle/nautilus sampler having moved to johannesulf/nautilus, bokeh's CoC moving into docs/, sphinx-doc renaming /en/main → /en/master, and a chunk of workspace notebooks that were reorganised (e.g. modeling/imaging/features/<x>.ipynb is now imaging/features/<x>/modeling.ipynb).

    What's fixed in this wave (library-side):

    Repo Files PR
    admin_jammy (audit tool + stale refs) 5 Jammy2211/admin_jammy#21
    PyAutoConf 2 PyAutoLabs/PyAutoNerves#105
    PyAutoFit 40 PyAutoLabs/PyAutoFit#1265
    PyAutoArray 2 PyAutoLabs/PyAutoArray#309
    PyAutoGalaxy 23 PyAutoLabs/PyAutoGalaxy#413
    PyAutoLens 24 #509

    Your reported bug — the hhttps:// typo on overview_2_new_user_guide line 31 — is in #509 along with the same typo in PyAutoGalaxy. All library unit tests pass (3236 tests total across the 5 libraries). Line endings are preserved, so the diffs are pure URL changes.

    Still to come (workspace phase): the audit also flagged URLs inside the workspace repos themselves (HowToFit, HowToGalaxy, HowToLens, autofit_workspace, autogalaxy_workspace, autolens_workspace). Those weren't touched in this wave — they'll get the same rewrites in a workspace-side PR after the library PRs merge. After both waves, broken count should drop from 304 → ~50 (the remainder are external paywalled / dead links we can't auto-fix).

  2. Jammy2211
    added 11 commits that reference this issue on May 15, 2026

    docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…

    abab1ae

    docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…

    85bba5a

    docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…

    f14097d

    docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…

    5b59863

    docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…

    86ef8a8

    docs: audit-driven URL fixes (workspace phase) (#7)

    25dc836

    docs: audit-driven URL fixes (workspace phase) (#7)

    ac79df8

    docs: audit-driven URL fixes (workspace phase) (#10)

    09b63e3

    docs: audit-driven URL fixes (workspace phase) (#57)

    7ea92ac

    docs: audit-driven URL fixes (workspace phase) (#70)

    4ef5b23

    docs: audit-driven URL fixes (workspace phase) (#152)

    c62fd5e

  3. Jammy2211 commented on May 15, 2026

    @Jammy2211
    Jammy2211
    on May 15, 2026
    CollaboratorAuthor

    Workspace phase shipped + merged — all 12 PRs across libraries, tooling, and workspaces are now on main. ✅

    Phase Repo PR
    Tool admin_jammy #21
    Library PyAutoConf #105
    Library PyAutoFit #1265
    Library PyAutoArray #309
    Library PyAutoGalaxy #413
    Library PyAutoLens #509
    Workspace HowToFit #7
    Workspace HowToGalaxy #7
    Workspace HowToLens #10
    Workspace autofit_workspace #57
    Workspace autogalaxy_workspace #70
    Workspace autolens_workspace #152

    Audit deltas:

    • Initial: 304 broken URLs across 12 repos
    • After library wave: 124 broken
    • After workspace wave: 104 broken — a drop of 200 URLs fixed end-to-end

    What the remaining 104 are: mostly external paywalled / dead links we can't auto-fix (academic.oup.com journal articles, the shut-down geekfeminism.wikia.com CoC link, the bdiemer/colossus bitbucket default-branch quirk, a couple of departmental academic homepages), plus a small set of internal readthedocs page renames that need editorial decisions rather than mechanical rewrites (pyautolens.readthedocs.io/en/latest/installation/numba.html, pyautolens.readthedocs.io/en/latest/advanced/slam.html, etc. — the docs that the link target was pointing to no longer exist; those URLs should either be removed from the docs or repointed at the new canonical page).

    The audit tool now lives at admin_jammy/software/url_check/. It can be re-run any time with python3 admin_jammy/software/url_check/url_check.py --repos <repo1> [repo2 ...] (or via the convenience run.sh), and the report drops in alongside the script.

    Your originally-reported bug — the hhttps:// typo in overview_2_new_user_guide.md — is now fixed on main in both PyAutoLens and PyAutoGalaxy. The whole new-user-guide page's URLs have been refreshed (Jammy2211 → PyAutoLabs, /release/ → /main/, and the broken notebooks/point_source/start_here.ipynb link on the PyAutoGalaxy page is now correctly notebooks/multi/start_here.ipynb).

    Closing this issue.

  4. Jammy2211
    closed this as completedon May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions