# Coder Eval

> Six skills that let your agent set up, write, run and analyze Coder Eval suites, including whether your Claude Code skills actually trigger.

UiPath's Claude Code plugin for Coder Eval, its open-source framework for evaluating and benchmarking AI coding agents and their skills. Coder Eval runs a real agent in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. This plugin puts writing, running and analyzing those suites behind six skills.

What its skills do
- init: scans your repository for what's worth evaluating (Claude Code skills, an MCP server, a CLI) and scaffolds a task directory with one real task
- check-skill: builds and runs an activation suite for one of your skills, with requests that should trigger it and near misses that shouldn't, then reports recall, precision and F1
- task: turns a plain-language description into task YAML with weighted criteria, validated with coder-eval plan
- lint-tasks: reviews existing tasks for criteria that can't fail, prompts that give away the answer, fixtures with no cleanup and near-duplicates, each with a fix (read-only)
- analyze: reads a finished run and writes analysis.md with failure patterns, per-task findings and fixes
- ci: writes a GitHub Actions workflow that runs your suite as a CI gate or on a schedule, to catch skills that quietly stop triggering

How to install it
In Claude Code, run /plugin marketplace add UiPath/coder_eval, then /plugin install coder-eval@coder-eval. The skills drive the coder-eval command-line tool, which isn't bundled: install it with uv tool install coder-eval (or pip install coder-eval). If it's missing, init, task and check-skill offer to install it and ask first.

Good to know
- Running a suite uses the evaluated agent's own credentials, such as ANTHROPIC_API_KEY for the default Claude Code agent. check-skill tells you how many rows it will run and what that implies for cost, and asks before running.
- Suites can evaluate Claude Code, OpenAI Codex, Google Antigravity, OpenCode or Pi.
- init and ci run only when you call them; the other four can also be picked up by the agent when a request calls for them.
- lint-tasks needs neither the CLI nor credentials.

- Type: Plugin
- Made by: UiPath
- Category: AI & agents
- Formats: Claude plugins
- Version: latest
- Website: https://coder-eval.com/
- Source: https://github.com/uipath/coder_eval/tree/HEAD/plugins/coder-eval
- Page: https://pluginboard.ai/plugins/coder-eval

## Works with

- Claude
- GitHub Copilot
- Claude Code
- OpenClaw

## Install

### Claude

In Claude (web or desktop): Customize → Plugins → Add → Add marketplace, enter UiPath/coder_eval, then add coder-eval

### Claude Code

In Claude Code:

```text
/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval
```

---
From Pluginboard (https://pluginboard.ai).