Greenwash
A GitHub App that catches AI coding agents cheating to make CI green, with a probability on every finding.
- Status
- Open source (Apache-2.0)
- Role
- Solo: design, build, evaluation
- Timeline
- Sep 2026
- Links
- GitHub
A 1:51 walkthrough. Greenwash is my own project, built on TypeSafe's API.
The problem
Green builds that shouldn't be green
When a coding agent can't fix a failing build, it sometimes makes the build pass anyway. It weakens an assertion, skips the test, hard-codes the answer the test expects, or marks the CI step as non-blocking. The build turns green and the reviewer moves on.
These changes are small and easy to miss in a large diff. A general AI code review tends to produce an essay, not a clear signal about which line to look at.
What it catches
Ten ways to fake a green build
- Weakened test assertion
- Test skipped or disabled
- Expected value rewritten without a stated behavior change
- Hard-coded answer for specific inputs
- Logic replaced with a placeholder
- Error silently swallowed
- Type or lint check suppressed
- CI step removed or made non-blocking
- Test file deleted
- Snapshot updated without any source change
Architecture
Narrow questions, not an essay
Greenwash reads every changed hunk of a pull request and asks narrow yes/no questions about it, such as "does this delete a test assertion or replace it with a weaker one?". TypeSafe's structured-judgment models answer each question with a probability. Eight checks are answered by a model; two are plain code rules.
The model handles judgment. Code handles policy: thresholds, severity, and what gets reported. Findings appear as a check run with line annotations and a single pull request comment that updates in place on every push.
Decisions
What I chose, and what it cost
| Decision | Why | Trade-off |
|---|---|---|
| Probabilities, not verdicts | A flagged change may be intentional; reviewers see how confident each finding is | Needs thresholds: flagged at 0.8 by default, worth a second look from 0.5 |
| Comment mode by default | Never blocks a merge unless a repository opts in | Enforcement is opt-in |
| Config read from the base commit | A pull request cannot switch off its own checks | Config changes apply only after merge |
| Fail neutral | If Greenwash itself errors, the check completes as neutral | An outage means no findings for that run |
| One request per hunk | Small, focused inputs; only the diff, path and pull request text are sent | Each hunk is judged without the rest of the file |
For each hunk, Greenwash sends only the diff, the file path, and the pull request's title and description. It stores nothing and has no database; findings live only on GitHub.
Results
Evaluated like a regression suite
- checks
- 10
- model-answered
- 8
- labelled cases
- 32
For every model-answered check, two real examples and two legitimate look-alike changes. All 32 classify correctly at default thresholds. The set is small and hand-written, so it is a regression suite, not a measure of real-world precision.
Known limits
What it does not do yet
- GitHub only, no GitLab or Bitbucket yet
- Each hunk is judged on its own, without the rest of the file or repository
- Very large pull requests are capped at 300 hunks, and each hunk at 8,000 characters
- Every finding is a probability: intentional changes can be flagged and subtle cheating can be missed
Related: AI reliability & evaluation