Act IV · What we learned
AI Factory · Lessons learned
Every incident became a rule
Light green is not mergeable
What happened
A CI subscription reported "all 18 checks success" while the backend Integration suite and Playwright were still pending.
Rule now
Merge only when check-heavy-ci.sh exits 0: Unit, Integration, Coverage Gate, Bruno and Playwright are terminal success.
`Issue: #N` does not close anything
What happened
Fleet PRs merged and their issues stayed open, because commits said Issue: #N.
Rule now
Closing commits and PR bodies use Closes #N; intermediate slices use Refs #N, and a guard checks them.
Bot commits with [skip ci] stall the merge
What happened
PR Validation committed wiki reports onto PR heads with [skip ci]; the head SHA had an empty check suite, and merge-when-green waited forever.
Rule now
Report workflows never commit onto a PR head; harnesses judge CI on the last commit without [skip ci].
A stale tip is not a product bug
What happened
Integration and Playwright failures (Budget/person, undefined, undefined) on branches behind main were already fixed on main.
Rule now
Merge origin/main first and re-run heavy CI before touching product code. Never rebase a pushed branch.
An invented Issue number passed Gate 1
What happened
Gate 1 only checked the format #[0-9]+, so a proposal could cite #999999.
Rule now
validate-sdlc-plan.sh resolves the Issue live via gh; exploration findings go through openspec-triage into real Issues first.
Closed issues left active changes that blocked every PR
What happened
After an issue closed, its unarchived OpenSpec change made plan validation fail on every open PR, and Main Health went red.
Rule now
Archive the change in the PR that implements it; multi_pr changes are archived by their last PR. Main Health fails on leftovers.
Models report success they have not earned
What happened
Local workers wrote "1,940 tests pass" and "PR created" into tasks.md, copied SHAs from main, and claimed changes their commits never made.
Rule now
The harness owns the ledger and the pushes; gates run real commands; Gate 4 compares every commit-body line with git show --stat.
Too much always-loaded context
What happened
Bulk rule imports and alwaysApply skills crowded the context window of every session.
Rule now
The always-loaded set stays ≤8,000 tokens: slim AGENTS.md, a 3 KB Constitution card, and on-demand rules.
Heavy suites on docs-only PRs
What happened
Docs and OpenSpec PRs queued full Java and Playwright runs and starved shared runners.
Rule now
Path-scoped CI: docs paths skip leaf jobs, while aggregator checks keep their required names green; main always runs everything.
Parallel heavy PRs are slower
What happened
Several heavy PRs at once enqueued Playwright behind each other; every PR waited longer and cost more.
Rule now
Serialize Playwright-heavy PRs; one domain-sized slice per PR; draft Dependabot PRs when they starve the queue.
Board, labels and state drift apart
What happened
Closed issues stayed In Progress on the board; open ones carried stale WIP labels.
Rule now
One coherent state: CLOSED ⇒ Done without WIP labels; OPEN WIP ⇒ In Progress + status:in-progress; checked by a script.
Work that lives only on one disk
What happened
Agents stopped mid-task with commits that were never pushed, and nobody could resume.
Rule now
Continuous publication: push the branch at once and after every group of commits, with an Issue comment on stage and what remains.
CodeQL default setup beside the advanced workflow
What happened
With both enabled, the SARIF upload was rejected and Analyze failed.
Rule now
Advanced XOR default: keep codeql.yml, and disable default setup with security/enable-gh-secure.sh --apply.
Toolchain majors from a bot
What happened
Dependabot proposed TypeScript 7; openapi-typescript needs ^5 and TS7 has no compiler API, so the PR could never go green.
Rule now
Toolchain majors that break the build are planned changes, not bot PRs: an ignore rule with its removal condition.
Anonymous image pulls hit rate limits
What happened
Docker Hub answered 429 on shared runners and service containers failed to start.
Rule now
Pull CI images through mirror.gcr.io and retry browser installs.
Evidence uploads must not fail a passing scan
What happened
The weekly k6 run passed, but the job went red because the artifact storage quota was hit. ZAP failed on an informational alert about non-storable 404 answers.
Rule now
Report uploads are continue-on-error; the ZAP allowlist is a reviewed set with reasons, pinned by a test.
A deploy cannot fix a setting
What happened
Pages deploys failed with 404 because GitHub Pages was disabled in repository settings; no workflow change could fix it.
Rule now
Owner-only steps (settings, quotas, tokens) are written down as exact steps for the Owner, never worked around by an agent.
Bots without write access still need issue hygiene
What happened
Agent tokens could open PRs but got 403 on issue labels or comments, so issue state silently drifted.
Rule now
Keep status in the PR body with Closes/Refs as the sync channel, and hand the Owner a paste-ready template; never invent a parallel tracker.