Surfaced during PR review of rossoctl/aiac#214 (part of the Digested access-control policy epic #2537).
Problem
CI runs the offline unit suite only (.github/workflows/ci.yaml: pytest with the default addopts, which deselects integration/system/llm/eval). As a result no prompt behavior is guarded by CI — the live-LLM tests that verify the PRB's proposer/auditor prompts actually elicit the correct grants and denies (-m llm, e.g. test/unit/agent/policy_rules_builder/test_graph_live_llm.py) run only when a human opts in with LLM_* env.
The concrete risk: a prompt edit (prompts.py _DENY_RULES / _MAPPING_RULES) that stops the real LLM emitting a user-role prohibition would still pass a bare pytest, and the safety-critical failure mode (access-broadening via a dropped deny) would reach main uncaught. The offline unit tests only pin the deterministic precheck/build plumbing, not elicitation.
Scope
This is general to the whole live-LLM suite, not specific to any one test — hence a standalone infra issue rather than a code change in a feature PR.
Options to consider
- A CI lane (likely nightly / on-label, not per-push, given cost + non-determinism) that runs
-m llm with LLM_* secrets, so prompt regressions are caught.
- Or a documented decision to leave prompt-elicitation verification as a manual pre-merge gate, with the offline suite guarding plumbing only.
Acceptance criteria
Surfaced during PR review of rossoctl/aiac#214 (part of the Digested access-control policy epic #2537).
Problem
CI runs the offline unit suite only (
.github/workflows/ci.yaml:pytestwith the defaultaddopts, which deselectsintegration/system/llm/eval). As a result no prompt behavior is guarded by CI — the live-LLM tests that verify the PRB's proposer/auditor prompts actually elicit the correct grants and denies (-m llm, e.g.test/unit/agent/policy_rules_builder/test_graph_live_llm.py) run only when a human opts in withLLM_*env.The concrete risk: a prompt edit (
prompts.py_DENY_RULES/_MAPPING_RULES) that stops the real LLM emitting a user-role prohibition would still pass a barepytest, and the safety-critical failure mode (access-broadening via a dropped deny) would reach main uncaught. The offline unit tests only pin the deterministic precheck/build plumbing, not elicitation.Scope
This is general to the whole live-LLM suite, not specific to any one test — hence a standalone infra issue rather than a code change in a feature PR.
Options to consider
-m llmwithLLM_*secrets, so prompt regressions are caught.Acceptance criteria
llmlane runs in CI (and on what trigger) or stays a manual gate.LLM_*secrets wired, a lane added that runs-m llmand skips cleanly when unset, flakiness/cost bounded (e.g. nightly).