Coder Eval

UiPath's Claude Code plugin for Coder Eval, its open-source framework for evaluating and benchmarking AI coding agents and their skills. Coder Eval runs a real agent in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. This plugin puts writing, running and analyzing those suites behind six skills. What its skills do - init: scans your repository for what's worth evaluating (Claude Code skills, an MCP server, a CLI) and scaffolds a task directory with one real task - check-skill: builds and runs an activation suite for one of your skills, with requests that should trigger it and near misses that shouldn't, then reports recall, precision and F1 - task: turns a plain-language description into task YAML with weighted criteria, validated with coder-eval plan - lint-tasks: reviews existing tasks for criteria that can't fail, prompts that give away the answer, fixtures with no cleanup and near-duplicates, each with a fix (read-only) - analyze: reads a finished run and writes analysis.md with failure patterns, per-task findings and fixes - ci: writes a GitHub Actions workflow that runs your suite as a CI gate or on a schedule, to catch skills that quietly stop triggering How to install it In Claude Code, run /plugin marketplace add UiPath/coder_eval, then /plugin install coder-eval@coder-eval. The skills drive the coder-eval command-line tool, which isn't bundled: install it with uv tool install coder-eval (or pip install coder-eval). If it's missing, init, task and check-skill offer to install it and ask first. Good to know - Running a suite uses the evaluated agent's own credentials, such as ANTHROPIC_API_KEY for the default Claude Code agent. check-skill tells you how many rows it will run and what that implies for cost, and asks before running. - Suites can evaluate Claude Code, OpenAI Codex, Google Antigravity, OpenCode or Pi. - init and ci run only when you call them; the other four can also be picked up by the agent when a request calls for them. - lint-tasks needs neither the CLI nor credentials.

• “Check whether my pdf-forms skill actually triggers when it should.”

  Copy

• “Set up coder-eval in this repository and scaffold a first task.”

  Copy

• “Write an eval task that checks the agent adds a --json flag to our CLI.”

  Copy

• “Review our existing eval tasks for criteria that can't fail.”

  Copy

• “Why did the last eval run fail, and what regressed since the previous one?”

  Copy

• “Add a GitHub Actions workflow that runs our eval suite on a schedule.”

  Copy