skill

Google Agents CLI Eval

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (...

google581,775+ installsVetted

About

# Agent Evaluation Guide

> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https://docs.astral.sh/uv/getting-started/installation/index.md) first if needed.

> **Scaffolded project?** If you used `/google-agents-cli-scaffold`, dataset and a custom metric are already scaffolded in `tests/eval/` (Python projects) or `eval/` (Go projects). For simplicity, this skill and its references use the Python directory layout; adjust accordingly if you've scaffolded a Go agent. > You already have `agents-cli eval run` (chains `generate` + `grade`), `tests/eval/datasets/`, and `tests/eval/eval_config.yaml`. Start with executing `eval run` and iterate from there.

## Reference Files

| File | Contents | |------|----------| | `references/dataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn / multi-turn / multi-agent, common mistakes | | `references/metrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config | | `references/user-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics | | `references/builtin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility | | `references/advanced-commands.md` | Opt-in commands: `eval analyze`, `eval optimize`, `eval submit` / `eval results` | | `references/multimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern | | `references/live-eval.md` | Live and voice agents — `--mode adk_live`, what gets graded, user-only turn authoring, the Live-model and region traps |

---

## The Quality Flywheel

Improving agent quality is iterative. The 4 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Eval Service in Gemini platform for better quality and scale.

### 1. Prepare Data

**Default:** Use or edit the scaffolded `tests/eval/datasets/basic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.

**Opt-in (ADK projects):** `agents-cli eval dataset synthesize`: user-simulate multi-turn datasets when you lack data; its output already includes traces, so Stage 2 collapses to `agents-cli eval grade` alone. See *Eval Commands* and `references/user-simulation.md`.

### 2. Run the Eval (always run)

**Default:** `agents-cli eval run` runs the agent over the dataset and grades the traces, writing `results_<ts>.{json,html}` to `artifacts/grade_results/`. Its last line is the verdict, e.g. `RESULT: FAIL (custom_response_quality 3.20 < 4.00; custom_response_quality: 1 case errored)`, and it exits 1 when a metric misses its threshold or a case errored. Agree the thresholds with the user and record them in the eval config (see *Evaluation Configuration Schema*).

**Decoupled form:** `eval generate` then `eval grade`, for a custom traces location, re-grading without re-running the agent, or traces from `synthesize` (`eval grade` alone).

### 3. Analyze Failures

**Default:** Start from the `Failures:` list above the `RESULT:` line, which names each metric that missed its threshold, the cases below it, and the cases that errored. Then open the latest `artifacts/grade_results/results_<ts>.html` (or `.json`) for those cases' scores and judge rationales — see *What to fix when scores fail* below for the fix table.

**Opt-in:** `agents-cli eval analyze`, LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See `references/advanced-commands.md`.

### 4. Optimize & Code Fix

**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.

**Opt-in (ADK projects):** `agents-cli eval optimize` runs ADK GEPA prompt optimization against a target metric (see `references/advanced-commands.md`). Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.

> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.

### Running the loop

Iterate stages 2 → 3 → 4 → 2 (with `synthesize`, re-run Stage 1 each pass, then `eval grade`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes, which is normal. Only after a case passes should you expand coverage

Install

Run this command

npx skills add google/agents-cli

Works with

claude appclaude codeclaude apicursorcodexwindsurfclinezed

Manual steps

Install with `npx skills add google/agents-cli`, or clone the repository and copy the `skills/google-agents-cli-eval` folder into your Claude skills directory.

View source

Frequently asked questions

What is the Google Agents CLI Eval skill?

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatev…

How do I install Google Agents CLI Eval?

Run this in your terminal:

npx skills add google/agents-cli
Which AI tools does Google Agents CLI Eval work with?

It works with claude_app, claude_code, claude_api, cursor, codex, windsurf, cline, zed.

Who made Google Agents CLI Eval?

google.

Is Google Agents CLI Eval free?

Yes, it is free to use.

Related assets

More curated picks in Development & Code.

All Google Agents CLI Eval alternatives →
skillclaude_appclaude_codeclaude_api
git clone https://github.com/anthropics/skills && cp -r skills/skills/claude-api ~/.claude/skills/
Claude Api
Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.…★ 4.8 · 2,500+
skillclaude_appclaude_codeclaude_api
npx skills add mattpocock/skills
Tdd
Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.1,046,962+
skillclaude_appclaude_codeclaude_api
npx skills add mattpocock/skills
Grilling
Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrase…1,041,607+
skillclaude_appclaude_codeclaude_api
npx skills add mattpocock/skills
Code Review
Review the changes since a fixed point (commit, branch, tag, or merge-base) along two axes: Standards (does the code follow this repo's documented cod…840,484+
skillclaude_appclaude_codeclaude_api
npx skills add mattpocock/skills
Diagnosing Bugs
Diagnosis loop for hard bugs and performance regressions. Use when the user says "diagnose"/"debug this", or reports something broken/throwing/failing…746,030+
skillclaude_appclaude_codeclaude_api
npx skills add microsoft/azure-skills
Azure AI
Build and troubleshoot Azure AI Search, Speech, OpenAI, and Document Intelligence apps. WHEN: AI Search, vector search, hybrid search, semantic search…727,368+

Audit before you install

Run any source through our checks - AI visibility, security, performance, and stack detection.

More in Development & Code