An AI Agent That Investigates Broken Builds

AI coding tools have made writing code faster than ever. Cursor, Copilot, ChatGPT — you describe what you want, the code appears, you ship it. Velocity is up. PR volume is up. And CI failures?…

AI coding tools have made writing code faster than ever. Cursor, Copilot, ChatGPT — you describe what you want, the code appears, you ship it. Velocity is up. PR volume is up. And CI failures? Also up — except debugging them hasn’t gotten any faster.

But when an AI writes a chunk of code, you don’t have the same mental model you’d have if you’d written it yourself. When CI goes red, you’re not just doing log archaeology — you’re doing it on code you didn’t fully author, in files you may have only skimmed. The gap between “8 tests failed” and “here’s what broke and why” has never been wider.

CI Sherlock is an AI agent that closes that gap. It runs as a GitHub Action after your test suite, correlates failing tests with the files changed in the PR, asks the model to reason about the root cause, and posts a plain-English summary directly on the PR — so you understand what broke before you’ve even opened the logs.

The problem with CI failures today

AI writes the code. CI tells you it’s broken. Nothing in between explains the connection.

Test runners tell you what failed. GitHub shows you what changed. But neither connects those two things — and that connection is exactly what you need when you’re reviewing a 40-file PR that an AI assistant generated in 10 minutes.

The result is the same manual process it’s always been: scan the failure, scan the diff, form a hypothesis, dig into logs. Except now you’re doing it on more PRs, with more changed files, and less personal familiarity with the code. It scales badly.

Flaky tests make this worse. If a test has a 15% failure rate across the last 20 runs, that context is invisible to anyone looking at a single CI run. You might spend time debugging something that isn’t actually broken — or asking your AI assistant to fix something that was never the problem.

What CI Sherlock does

CI Sherlock is a Python agent that plugs into any GitHub Actions workflow. After your tests run, it:

Parses the test report (Playwright JSON for now, with a pluggable parser interface for other frameworks)
Fetches the PR diff from GitHub — every file changed, every line
Correlates failing tests with changed files using path matching and heuristics
Asks Model to reason about the root cause given the failures, errors, and correlation signals
Stores the results in SQLite for historical trend analysis
Posts a structured summary comment on the PR

It also surfaces CI optimization signals — slow tests, missing caches, tests that could be parallelized — so the agent is useful even when your build is green.

Architecture

CI Sherlock runs entirely inside GitHub Actions. There’s no external service to maintain, no database server, no API to deploy. The only external calls are to the GitHub API and the Claude API.

flowchart TB
  GA["GitHub Actions: test job"] -- produces --> PR[playwright-report.json]
  PR -- uploaded as artifact --> AR[GitHub Artifacts]
  AR -- downloaded by --> SH["CI Sherlock: sherlock job"]
  SH --> PP[PlaywrightParser]
  SH --> LE[LLMEngine]
  SH --> AN[Analyzer]
  SH --> GC[GitHubClient]
  SH --> PC[PRCommenter]
  SH --> DB[(SQLite database)]
  LE -- structured output --> CL[Claude Opus 4.6]
  GC -- fetches PR diff --> API[GitHub API]
  PC -- posts comment --> API
  DB -- history --> SD[Streamlit dashboard]
  classDef accent stroke-width:1.5px
  class SH accent

The sherlock job depends on the test job — it only runs after tests complete, whether they passed or failed. This means the agent always has access to the full test output.

Data flow

Here’s what happens step by step when a PR is opened and CI runs:

sequenceDiagram
  participant D as Developer
  participant GA as Actions
  participant PW as Playwright
  participant CS as CI Sherlock
  participant GH as GitHub API
  participant C as Claude
  participant DB as SQLite
  D->>GA: opens PR / pushes commit
  GA->>PW: runs test suite
  PW->>GA: playwright-report.json (artifact)
  GA->>CS: triggers sherlock job
  CS->>GA: downloads playwright-report.json
  CS->>CS: PlaywrightParser → list[TestResult]
  CS->>GH: GET /repos/{owner}/{repo}/pulls/{pr}/files
  GH->>CS: list[ChangedFile] with patch text
  CS->>CS: Analyzer.correlate() → AnalysisResult
  CS->>DB: write run + test_results + correlations
  CS->>C: AnalysisResult → structured prompt
  C->>CS: LLMInsight (root_cause, confidence, recommendation)
  CS->>DB: write insight
  CS->>GH: POST /repos/{owner}/{repo}/issues/{pr}/comments
  GH->>D: PR comment notification

The whole flow typically adds 30–60 seconds to a CI pipeline, most of which is the LLM call.

The correlation engine

The most important piece — and the one that requires no LLM — is the correlation between failing tests and changed files.

flowchart TB
  FT[Failed tests] --> A[Analyzer]
  CF[Changed files] --> A
  A --> M{Match type}
  M -- test file = changed file --> DM["direct_match · score 1.0"]
  M -- import detected --> IM["imported_module · score 0.8"]
  M -- same directory --> SD["same_directory · score 0.6"]
  M -- no match --> UN["unmatched · score 0.0"]
  DM & IM & SD & UN --> R[AnalysisResult]
  classDef accent stroke-width:1.5px
  classDef muted stroke-width:1.5px
  class DM accent
  class UN muted

A few examples of how this plays out in practice:

Changed file	Failing test	Correlation	Score
`src/components/Button.tsx`	`tests/components/Button.spec.ts`	direct match	1.0
`src/auth/login.ts`	`tests/auth/login.spec.ts`	direct match	1.0
`src/utils/format.ts`	`tests/checkout/payment.spec.ts`	same directory	0.6
`src/api/users.ts`	`tests/users/list.spec.ts`	same directory	0.6

High-score correlations are passed to the LLM as strong evidence. Low-score or unmatched failures are flagged as potentially environmental or flaky.

The LLM layer

The analyzer produces a structured AnalysisResult. The LLM engine turns that into a human-readable explanation.

flowchart TB
  AR[AnalysisResult] --> PB[Prompt builder]
  PB --> CF[Changed files with patch diffs]
  PB --> FT[Failed tests with error messages]
  PB --> CS[Correlation signals]
  CF & FT & CS --> C[Claude Opus 4.6]
  C --> I["instructor · guaranteed JSON"]
  I --> L[LLMInsight]
  L --> R1["root_cause: string"]
  L --> R2["confidence: 0.0–1.0"]
  L --> R3["recommendation: string"]
  L --> R4["flaky_tests: list of test names"]
  classDef accent stroke-width:1.5px
  class C,L accent

We use instructor to patch the OpenAI client, which means GPT-4o’s response is always validated against a Pydantic model. No JSON parsing errors, no partial outputs, no prompt engineering needed for output format.

class LLMInsight(BaseModel):
    root_cause: str
    confidence: float = Field(ge=0.0, le=1.0)
    recommendation: str
    flaky_tests: list[str]

insight: LLMInsight = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    response_model=LLMInsight,
)

The LLM layer is optional. If OPENAI_API_KEY is not set, the agent posts raw correlation analysis without the AI section.

Flaky test detection

Flaky tests are detected at two levels:

flowchart TB
  T[Test results] --> Q1{"retry_count > 0 AND status = passed?"}
  Q1 -- yes --> F1["Flag as flaky · current-run signal"]
  Q1 -- no --> H[Query run history in SQLite]
  H --> Q2{"failure_rate > 10% over last 20 runs?"}
  Q2 -- yes --> F2["Flag as flaky · historical signal"]
  Q2 -- no --> N[Not flaky]
  F1 --> S[FlakySignal]
  F2 --> S
  classDef warn stroke-width:1.5px
  classDef muted stroke-width:1.5px
  class F1,F2,S warn
  class N muted

The current-run signal catches tests that retried their way to a pass. The historical signal catches tests with intermittent failures that don’t always retry — you need multiple runs in the database to see them. This is why the SQLite store matters from day one.

Persistence and the dashboard

Every run writes to SQLite. The schema is simple:

erDiagram
  runs ||--o{ test_results : has
  runs ||--o{ correlations : has
  runs ||--o| insights : has
  runs ||--o{ optimization_suggestions : has
  runs {
    text id PK
    text repo
    integer pr_number
    text commit_sha
    text status
    integer total_tests
    integer failed_tests
    real release_readiness_score
    text created_at
  }
  test_results {
    integer id PK
    text run_id FK
    text test_name
    text test_file
    text status
    integer duration_ms
    integer retry_count
    text error_message
  }
  correlations {
    integer id PK
    text run_id FK
    text test_name
    text changed_file
    real score
    text reason
  }
  insights {
    integer id PK
    text run_id FK
    text root_cause
    real confidence
    text recommendation
  }
  optimization_suggestions {
    integer id PK
    text run_id FK
    text type
    text description
    text impact
  }

The Streamlit dashboard reads from this file and renders:

Run timeline — pass/fail status over time
Test failure heatmap — test × run grid, coloured by status
Flaky leaderboard — tests ranked by failure rate
Slowest tests — average duration chart
Release readiness score — composite 0–100 score with factor breakdown

Run it locally after downloading the DB artifact from a CI run:

pip install git+https://github.com/slavkurochkin/ci-sherlock-agent.git
ci-sherlock dashboard --db ./history.db

The PR comment

Here’s what lands on your PR after a failed run:

## CI Sherlock Analysis

### Failures  8 tests failed

**Root cause** (82% confidence)
Recent changes to `Button.tsx` renamed the `.btn-primary` class to `.btn-default`.
Six failing tests use a selector that targets `.btn-primary` and can no longer find the element.

**Recommendation**
Update selectors in `tests/components/Button.spec.ts` and `tests/auth/login.spec.ts`
to use `.btn-default`, or add a backward-compatible alias in the component.

### Correlation
| Test | Changed file | Signal |
|---|---|---|
| `login should redirect after auth` | `src/components/Button.tsx` | direct match |
| `checkout should submit form` | `src/components/Button.tsx` | direct match |
| `signup renders submit button` | `src/components/Button.tsx` | direct match |

### Flaky tests detected
- `payment.spec.ts > should process card` — retried 2x, passed on retry (historical rate: 18%)

### CI optimization
- **Slow tests**: `checkout.spec.ts` avg 18s, `search.spec.ts` avg 15s — consider splitting
- **No test sharding detected** — pipeline could be ~40% faster with `--shard` flag

How to add it to your project

1. Configure Playwright to emit JSON

// playwright.config.ts
export default {
  retries: 2,
  reporter: [
    ["list"],
    ["json", { outputFile: "playwright-report.json" }]
  ],
  use: {
    trace: "on-first-retry",
  }
}

2. Add the workflow

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report.json
        if: always()

  sherlock:
    needs: test
    runs-on: ubuntu-latest
    if: always()
    permissions:
      pull-requests: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: playwright-report
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install git+https://github.com/slavkurochkin/ci-sherlock-agent.git --quiet
      - name: Run CI Sherlock
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GITHUB_REPOSITORY: ${{ github.repository }}
          GITHUB_SHA: ${{ github.sha }}
          SHERLOCK_PR_NUMBER: ${{ github.event.pull_request.number }}
        run: ci-sherlock analyze

That’s it. No config file needed to get started.

What’s shipped

CI Sherlock was built in four phases, all now complete:

Phase	What shipped
1 — Skeleton	Parser + correlation + SQLite + PR comment (no LLM)
2 — Intelligence	LLM root cause with actual diff patches, confidence scores, structured insights
3 — Flaky + optimization	Historical flake detection, slow-test and parallelization analysis
4 — Dashboard + scoring	Streamlit UI, release readiness score, trend charts

The parser is designed to be extended. Playwright is the MVP target, but the BaseParser interface means Jest, Vitest, Cypress, and pytest support can be added without touching the rest of the codebase.

What could come next:

Additional parser targets (Jest, Vitest, pytest)
Cache detection optimization signal
Persistent history across CI runs via git branch storage (see docs/HISTORY_DB_PERSISTENCE.md)
PyPI release for simpler installation

The full implementation plan, data model, and architecture docs are in the repo.

CI Sherlock is open source. The goal is a tool that makes CI failures faster to understand and easier to fix — so developers spend less time on archaeology and more time building.