Doubling Claude's success at diagnosing failed CI runs

The situation

Ask Claude Code why a CI run failed and it has two bad options: read the whole log, or read part of it.

A real React CI log runs to 13,734 lines, about 422,000 tokens, most of it setup noise. Pasting it fills the context window. Truncating it keeps the start and the end and often drops the one line that explains the failure. Either way the model guesses, and a wrong guess sends a developer down the wrong path.

What I did

• Built a summarising engine that folds stack traces into the line that started them, groups repeated lines by template, detects log levels, test-runner marks and compiler errors, and keeps a window of context around every distinct failure.
• Rendered each summary under a fixed budget, with line ids that let Claude drill back into the raw log when it needs more.
• Packaged it as a Claude Code plugin: a hook spots pasted CI links and giant pastes, a skill teaches Claude when to use it, and a paste guard stops a 50,000-line log reaching the context by accident.
• Redacted secrets before anything is shown, from cloud keys and API tokens to connection strings and JWTs, and kept raw logs in a private, self-expiring cache.
• Designed a benchmark to test it honestly: blind Claude agents, one input each, graded by a separate judge agent against labelled root causes.

How it was built

pith is a single Python package using only the standard library, so it installs anywhere with pipx or uv and runs entirely locally with no API key. The Claude Code plugin adds hooks, a skill and a ! pith shell command; the same CLI works in any terminal or with any AI tool.

An optional mode asks a fast classification model to rank the most likely root cause first, sending only redacted, masked templates and capping the cost at about a cent and a half per run.

• Python
• Claude Code plugins
• Hooks and skills
• GitHub Actions
• gh CLI
• LLM evaluation
• Secret redaction

The result

Half the size, twice the right answers.

Given only a pith summary, Claude found the right root cause in 43 of 49 runs and never named a wrong one. Given a 30,000-character excerpt of the same logs, it got 21 right and 15 wrong, at more than twice the size.

Every input, prompt, answer and verdict is published in the repository, so anyone can check the result or rerun it.