[experiments] Daily Experiment Report — 2026-09-30

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-30

52 experiments analysed across 49 workflows. 26 are 🟢 READY on sample counts, but none has a PROMOTE or REJECT decision: outcome observations, guardrail metrics, or multi-variant support are still missing.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis (samples)	26
Decisions with sufficient evidence (PROMOTE/REJECT)	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no PROMOTE decisions; multi-experiment workflows: smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra)

Reason-code breakdown

• insufficient_observations: 26
• unsupported_multi_variant: 14
• insufficient_samples: 9
• guardrail_unsupported: 3

📊 All experiments (core decisions, verbatim from `gh aw experiments analyze`)

Experiment	Workflow	Readiness	Usable samples	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	🟢 READY	caveman:72, verbose:59 (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	🟢 READY	batch:68, per_scenario:63 (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	🟡 COLLECTING	single_agent:17, sub_agents:21 (min 20)	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	🟡 COLLECTING	phased_sub_agents:48, single_agent:38 (min 264)	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	🟢 READY	assertive:165, clinical:161, narrative:164 (min 20)	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	🟡 COLLECTING	concise:4, detailed:7 (min 20)	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	🟢 READY	neutral:28, urgent:25 (min 20)	EXTEND	insufficient_observations
prompt_style	cicoach	🟢 READY	concise:39, detailed:59 (min 20)	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	🟡 COLLECTING	prose:34, ste:3, structured:43 (min 30)	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	🟡 COLLECTING	concise:0, structured:0 (min 30)	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	🟢 READY	single_agent:44, sub_agents:43 (min 20)	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	🟡 COLLECTING	brief:12, comprehensive:7 (min 20)	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	🟢 READY	concise:77, detailed:72 (min 20)	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	🟡 COLLECTING	gpt-5.3-codex:13, gpt-5.3-codex-spark:11 (min 20)	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	🟡 COLLECTING	agent:1, claude-haiku-4.5:72, claude-sonnet-4.6:34, claude-sonnet-5:9, small-agent:2 (min 20)	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	🟡 COLLECTING	executive_summary:54, full_detail:58, ste:16 (min 20)	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	🟢 READY	concise:70, verbose:70 (min 20)	EXTEND	insufficient_observations
output_format	dailycompilerquality	🟡 COLLECTING	concise:55, detailed:64, ste:17 (min 20)	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	🟡 COLLECTING	agent:2, claude-haiku-4.5:61, claude-sonnet-4.6:43, claude-sonnet-5:11, small-agent:1 (min 20)	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	🟡 COLLECTING	agent:1, claude-haiku-4.5:49, claude-sonnet-4.6:47, claude-sonnet-5:14, small-agent:2 (min 20)	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	🟢 READY	multi_candidate:41, single_pass:57 (min 30)	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer	🟡 COLLECTING	n/a (min 20)	EXTEND	insufficient_observations
output_format	dailyissuesreport	🟢 READY	collapsible:61, inline:63, ste:22 (min 20)	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	🟢 READY	concise:63, detailed:51 (min 20)	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier	🟡 COLLECTING	n/a (min 20)	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	🟡 COLLECTING	n/a (min 20)	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	🟢 READY	iterative:81, single_pass:59 (min 30)	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	🟡 COLLECTING	bullet_list:28, prose:27, structured_sections:22 (min 30)	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	🟡 COLLECTING	default:2, relaxed:1, tight:1 (min 20)	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	🟡 COLLECTING	no:6, yes:10 (min 20)	EXTEND	insufficient_samples
output_format	deepreport	🟢 READY	annotated_brief:64, executive_brief:84, full_briefing:59, ste:42 (min 20)	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	🟡 COLLECTING	brief:6, detailed:6 (min 20)	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	🟡 COLLECTING	concise:19, detailed:18, step_by_step:16 (min 20)	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	🟢 READY	full_bash:60, minimal_toolset:68 (min 20)	EXTEND	insufficient_observations
prompt_style	issuearborist	🟢 READY	concise:48, detailed:39 (min 20)	EXTEND	insufficient_observations
reasoning_depth	plan	🟡 COLLECTING	baseline:4, deep:2, shallow:1 (min 20)	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef	🟡 COLLECTING	n/a (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	🟢 READY	single_agent:99, sub_agents:109 (min 20)	EXTEND	insufficient_observations
caveman	smokecopilot	🟢 READY	no:186, yes:186 (min 20)	EXTEND	insufficient_observations
subagent_model	smokecopilot	🟢 READY	large:176, small:176 (min 20)	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	🟢 READY	no:95, yes:95 (min 20)	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	🟢 READY	large:95, small:95 (min 20)	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	🟢 READY	no:81, yes:82 (min 20)	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	🟢 READY	large:82, small:81 (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	🟡 COLLECTING	delegated_sequential:17, inline_strict:14, single_agent_control:19 (min 30)	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	🟢 READY	single_agent:129, sub_agents:144 (min 20)	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	🟢 READY	parallel_sub_agents:42, single_agent:34 (min 20)	EXTEND	insufficient_observations
prompt_style_test	smokeproject	🟡 COLLECTING	n/a (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	🟡 COLLECTING	n/a (min 20)	EXTEND	insufficient_observations
model_size	testqualitysentinel	🟢 READY	claude-haiku-4.5:52, claude-sonnet-5:53 (min 20)	EXTEND	insufficient_observations
tone_style	typist	🟢 READY	conversational:49, formal:42 (min 20)	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	🟡 COLLECTING	eager:6, lazy:10 (min 20)	EXTEND	insufficient_samples

Notes on scope

• Charts and per-run success/duration statistics were not produced this run: the sandbox has no authenticated workflow-run listing and analyze returned no per-variant outcome observations (observations empty), so descriptive stats would not be meaningful.
• No workflow declares a tracking issue: field, so no issue comments or labels were emitted.
• Report action equals the core decision for every experiment.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Fix observation pipeline for the ~24 insufficient_observations experiments (e.g. smokecopilot, smokegemini, gpclean, typist) so READY ones can reach PROMOTE/REJECT.
2. Back run_success_rate / empty_output_rate guardrails with supported metrics (cicoach, dailyfact, dailysecurityredteam) to clear guardrail_unsupported.
3. Reduce 3+ variant experiments (awfailureinvestigator, deepreport, dailyissuesreport, model_size family) to pairwise contrasts or add core multi-variant support; ste arms are under-sampled (3–42 runs).

Top eval actions

1. Add graders emitting output_length_chars/output_token_count for output_format experiments (deepreport, dailyissuesreport, dailycodemetrics, dailycompilerquality).
2. Add a success/quality grader for prompt_style workflows (dailynews, issuearborist, dailyastrostylelitemarkdownspellcheck).
3. Add a false-positive grader mapped to dailysecurityredteam prompts.

Decision-pipeline gaps

• Unresolved native metrics: run_success_rate, empty_output_rate, token_count_per_run, run_duration_ms (dailysafeoutputoptimizer has no variant data).
• experiments list needs the experiments/* branches fetched locally; --repo on analyze did not work from the sandbox.
• Promote factorial interaction diagnostics (smoke caveman × subagent_model) into core analysis.
• Empty-variant experiments: dailyfunctionnamer, dailyrenderingscriptsverifier, prsouschef, smokeproject, smoketemporaryid, dailysafeoutputoptimizer.

Run: 36686689672

Generated by 🧪 daily-experiment-report · copilot · auto · 31.5 AIC · ⌖ 19.7 AIC · ⊞ 16.3K · ◷

• expires on Oct 3, 2026, 12:04 AM UTC-08:00