Fifth in the 6-part CCAR-F scenario series — full source for every scenario: labs.
The scenario, as written in the exam guide
Claude Code integrated into the CI/CD pipeline: automated code review, generation of test cases, and feedback on pull requests, minimizing false positives.
| Domain | Weight | Where it shows up in this scenario |
|---|---|---|
| Claude Code Configuration & Workflows | 20% | The GitHub Actions workflow — headless invocation, permissions, output handoff |
| Prompt Engineering & Structured Output | 20% | A prompt asking for a confidence score and a concrete failure scenario, plus a schema that validates what comes back |
The exam guide names the hard part explicitly, in its own text: minimizing false positives. Everything else in this scenario — running Claude Code headless, posting to a PR — is comparatively mechanical.
A prompt instruction is not a threshold
The intuitive fix is prompt-only: tell the model to be conservative, to only flag things it's sure about. labs's prompts/review-prompt.md does exactly that — it asks for a confidence score (0.0–1.0) and a concrete failure scenario ("this breaks under these specific conditions," not "this could be cleaner") instead of a vague concern. That's real, and it matters.
But it's still just a request the model can miscalibrate on any given run. A false positive doesn't have to happen often to train a team to stop reading the bot's comments entirely — and once that happens, the tool is worse than not running at all, because it still burns CI minutes and PR-page real estate.
src/filter-findings.ts is the actual gate: MIN_CONFIDENCE = 0.7, and severity "nitpick" is dropped unconditionally regardless of confidence. That's a second, independent check — code, not another prompt asking the model to "really be sure this time." It's the same shape as Scenario 1's process_refund: ask nicely in the prompt, then enforce for real downstream.
Degrade, don't crash
post-review-comments.ts validates every finding against the same schema the prompt asks Claude Code to follow. If one entry is malformed — a missing field, a confidence outside 0–1 — that single entry gets logged and dropped, not treated as a reason to fail the whole CI check. A review step that crashes because the model produced one slightly malformed JSON object is a worse failure mode than one that posts three valid findings and silently skips a fourth. Same instinct as Scenario 3's coordinator handling a partial multi-agent result: a structurally-invalid piece is an expected case to route around, not an exception to propagate.
Run it yourself
git clone https://github.com/aicertstudy/labs
cd labs/ccar-f/scenarios/scenario-5-claude-code-ci
npm install
npm test # 5 passing tests on the filter/format logic
npm run post -- fixtures/sample-findings.json
The fixture has 4 findings; watch exactly 2 clear the bar (a real bug, and a risk right at 74% confidence) while a nitpick and a 35%-confidence risk get suppressed and logged. No GITHUB_TOKEN needed — without it, the script prints what it would have posted instead of calling the GitHub API, which is what makes the filtering logic checkable without a real PR.
Test yourself
1. Why isn't asking the model for a confidence score in the prompt enough to minimize false positives on its own?
Answer: the confidence score is still something the model can miscalibrate on any given run — it's a signal, not an enforcement mechanism.
A prompt instruction (even a good one asking for calibrated confidence) only holds if the model correctly self-assesses every time. A fixed threshold checked in code doesn't depend on that self-assessment being reliable on the specific run it's protecting.
2. Why does filterFindings drop every "nitpick" severity finding unconditionally, rather than applying the same confidence threshold to it as "bug" and "risk"?
Answer: nitpicks are a different kind of claim entirely — a style preference isn't made more "true" by higher model confidence, so a confidence threshold doesn't apply to that category at all.
Confidence measures certainty about a factual claim (this breaks under these conditions). A style preference isn't a factual claim the model can be more or less certain about in the same sense — so gating it on confidence would be a category error, not a stricter filter.
3. What happens in this pipeline if Claude Code writes one finding missing the confidence field?
Answer: that one finding is logged as skipped and excluded from the review; the rest of the run proceeds normally.
The zod validation in post-review-comments.ts checks each finding independently, so one malformed entry doesn't invalidate the batch — it's treated as a per-item parsing failure, not a run-level one.
4. In the sample fixture, a "risk" finding about percentOff going negative has 35% confidence and gets suppressed. What would have to change for that same finding to be worth surfacing?
Answer: either its confidence would need to rise to 70%+ (i.e., the model would need real evidence, not a hypothetical), or the team would need to deliberately lower MIN_CONFIDENCE — which trades fewer suppressed real risks for more false positives.
The threshold is a single tunable trade-off point, not a correctness boundary — moving it doesn't make findings above it "right" and below it "wrong," it changes how much noise the team is willing to tolerate in exchange for catching more borderline risks.
5. The workflow posts one summary comment instead of inline per-line review comments. What GitHub API detail is the reason this scenario doesn't do that by default?
Answer: inline review comments require a diff position (an offset into the unified diff), not just a file and line number — computing that correctly is meaningfully more involved than posting a single issue comment.
A file/line pair identifies a location in the final version of a file; a diff position identifies a location within the diff itself, which depends on the exact diff hunks GitHub generated for that PR. Getting this wrong silently misplaces or fails to post inline comments, which is why the reference implementation ships the simpler, always-correct summary-comment version first.
Free CCA-F practice exam, covering all 5 domains: aicert.study/certifications/ccaf.