Loading listing…
OpenJudge
Security scan passedEvaluate AI apps with your agent: design test sets and graders, check judges against people, test RAG and prompts, and red-team them.
Pack · 21 skillsby agentscope-ai· ResearchUnclaimed · Yours? Claim it
Preview
4 images
What OpenJudge does
About
OpenJudge is AgentScope's open-source framework for evaluating AI applications such as agents and chatbots. This skill pack teaches your agent its evaluation workflows, from a first test set to a full report, plus workflows for academic papers and for comparing models. What you can do with it - Start an evaluation from nothing, or from the traces, prompts or RAG pipeline you already have - Design evaluation datasets: test cases, stratified samples and adversarial examples - Pick graders and metrics, write LLM-as-judge prompts and combine scores into one - Measure how well a judge agrees with human labels and spot its biases - Evaluate a RAG system, separating retrieval problems from generation problems - Compare a changed prompt with the baseline, and red-team an app for jailbreaks, prompt injection and PII leaks - Compare models or agents arena-style on your task, with test queries and rubrics generated for you - Review academic papers, and check a BibTeX file for made-up references against CrossRef, arXiv and DBLP - Build graders and RL reward signals with the OpenJudge Python library How it connects - Most skills use the OpenJudge Python library (pip install py-openjudge) and an LLM you choose as the judge, through an OpenAI-compatible endpoint. - You can also try its graders in the browser at openjudge.me without installing anything. Good to know - The pack also has three stand-alone skills: one that checks whether an API endpoint is really backed by Claude, one for MiniMax's mmx command-line tool, and one that suggests combinations of skills for a multi-part task. - It's open source under the Apache 2.0 licence.
More from Research
Skills people install alongside OpenJudge.
