Kipt

Your prompts and skills, one keystroke away.

Explore

google-agents-cli-eval skill: evaluate agents with agents-cli

The agents-cli eval skill from google/agents-cli teaches coding agents to evaluate agents: datasets, metrics, LLM-as-judge, failure fixes and eval compare.

By the Kipt team · Updated 26 October 2026

Source: google/agents-cli · Maintainer: Google · License: Apache-2.0 · Checked: 26 October 2026

What is the google-agents-cli-eval skill?

google-agents-cli-eval is an Agent Skill that teaches a coding agent how to evaluate an AI agent with the agents-cli command. It covers eval datasets, built-in and custom metrics, LLM-as-judge scoring, reading failures and the fix loop Google calls the Quality Flywheel.

It is one of seven skills in google/agents-cli, Google's "CLI and skills for building agents on Gemini platform." The SKILL.md description says it "Applies to any agents-cli project, whatever framework the agent is written in," and points agent code, deployment and scaffolding to their own skills.

As of 26 October 2026, its skills.sh page listed about 511,900 installs (snapshot, 26 October 2026). The SKILL.md metadata lists Google as author and version 1.11.0.

What does it do?

The SKILL.md assumes a scaffolded project, where a dataset and a custom metric already sit in tests/eval/ (Python) or eval/ (Go). Its advice is to run agents-cli eval run first and iterate from there.

The Quality Flywheel

Four stages, each with a default path the coding agent does itself and an opt-in command:

Stage Default Opt-in command
1. Prepare data Edit tests/eval/datasets/basic-dataset.json, start with 1 to 2 cases agents-cli eval dataset synthesize (ADK)
2. Run the eval agents-cli eval run eval generate then eval grade
3. Analyze failures Read the latest results_<ts>.html or .json agents-cli eval analyze
4. Optimize and fix Edit prompts, tool descriptions or instructions agents-cli eval optimize (ADK)

After each fix, the agent runs agents-cli eval compare <prev>.json <new>.json to confirm the target metric improved without regressing others. The skill expects 5 to 10 or more iterations per case and says to keep a slice of cases out of the loop: "Hold cases back."

Choosing metrics

The skill maps goals to built-in metrics. multi_turn_task_success checks whether the agent achieved the user's goal, multi_turn_trajectory_quality judges the reasoning path, and multi_turn_tool_use_quality judges tool calls. Others include final_response_quality, hallucination, grounding, safety and final_response_match. Only the three multi-turn metrics accept multi-turn traces. agents-cli eval metric list shows them all, and Google's metric docs hold the definitions.

When no built-in fits, a YAML config file declares metrics_to_run plus a pool of custom_metrics: a prompt_template makes an LLM-judge metric, and a custom_function makes a Python metric. A metric defined in the pool runs only if it is also listed in metrics_to_run or passed with --metrics.

Fixing what fails

A table maps each low score to a change. A low multi_turn_tool_use_quality means fixing tool descriptions, parameter docstrings or instructions. A low hallucination score means tightening instructions to stay grounded in tool output. For tool checks, it recommends rubric-based metrics over hardcoded call sequences, because agents call helper tools in different orders.

Gotchas it covers

  • The ADK App name must match the agent's directory, or you get "Session not found" errors.
  • eval run, eval grade and eval submit default to the global endpoint and don't inherit the project region; pass --region for data residency.
  • Models with thinking enabled may skip tool calls; force tool use with tool_config set to mode="ANY".
  • Use a before_agent_callback to initialize session state and avoid a KeyError on the first turn.

Seven reference files go deeper: dataset_schema.md, metrics-guide.md, user-simulation.md, builtin-tools-eval.md, advanced-commands.md, multimodal-eval.md and live-eval.md.

How do you install it?

The Getting Started guide lists Python 3.11+, uv and Node.js as prerequisites. The recommended install adds the CLI and all seven skills:

uvx google-agents-cli setup

Or install skills only, either the whole set or just this one (the command shown on skills.sh):

npx skills add google/agents-cli
npx skills add https://github.com/google/agents-cli --skill google-agents-cli-eval

The skill's metadata lists the agents-cli binary as a requirement, installed with uv tool install google-agents-cli. In Claude Code, run /skills to confirm the agents-cli skills are listed. Codex and Cursor read the same SKILL.md format; see Codex skills and Cursor skills.

How do you use it?

The skill loads when you ask to "run an evaluation", "evaluate my agent", "write an eval dataset", "analyze eval failures", "compare eval results" or "optimize agent", per its description. You can also call /google-agents-cli-eval in Claude Code. The workflow skill marks it mandatory before any eval run.

Example prompts (suggestions, not quotes from the repo):

  • Run the evals and paste the scores table. Don't change anything yet.
  • Add three eval cases for refund requests, including one where the order ID is missing.
  • multi_turn_tool_use_quality dropped after my last change. Compare the two results files and tell me why.
  • Write a custom metric that fails any response longer than 120 words.
  • Synthesize five multi-turn scenarios for an angry customer and grade them.

Google's Evaluation Guide covers the same commands for running evals by hand.

Tips and limits

From the SKILL.md, the repo docs and skills.sh, not from testing:

  • Show the numbers. The skill's "Proving Your Work" section tells the agent to paste the scores table after each run and before deploy, since the exit code is always 0.
  • Don't move the bar. Its shortcuts table rejects lowering the bar, skipping flaky cases or only editing expected outputs.
  • Some commands are ADK only. eval dataset synthesize and eval optimize load the agent through ADK; the rest of the loop is framework-agnostic.
  • Optimization costs. GEPA optimization "makes many LLM calls and can take a long time." The agent should run it once, and only when asked.
  • Local code metrics run with your privileges. Python metrics run in-process by default; set execution: "remote" to run them in the Gemini platform sandbox instead.
  • Security audits. On skills.sh, Gen Agent Trust Hub rated it Pass with risk level Safe (22 October 2026); its report notes that local custom metrics need trusted code and that traces can carry indirect prompt injection. Socket (20 October 2026) and Snyk (risk level Low, 20 October 2026) rated it Pass.

Under Apache 2.0 you can copy and modify the skill as long as you keep the license and notices and mark files you change.

Save it in your Kipt library

In Kipt, open Skills and paste the skill's GitHub folder link to import it with its seven reference files, or paste the repo link and pick from the list of all seven agents-cli skills. You can also drop the folder or a .zip. Your skills live in your own private GitHub repo and install into Claude Code or Codex with one command:

npx skills add https://github.com/<you>/<repo>.git --all -g

Kipt is not affiliated with Google.

Similar skills

Questions

What is the google-agents-cli-eval skill?

It is one of seven Agent Skills in Google's google/agents-cli repo. It teaches a coding agent how to evaluate an AI agent with agents-cli: write an eval dataset, pick built-in or custom metrics, run eval run, read the results, fix failures and confirm the fix with eval compare. It applies to any agents-cli project, whatever framework the agent uses.

What does agents-cli eval run do?

It runs the agent over the eval dataset and grades the traces in one command, chaining eval generate and eval grade. Results land in artifacts/grade_results/ as a timestamped JSON file and an HTML file. The skill warns that eval run exits 0 whatever the scores are, so you have to read the scores.

Which metrics does the agents-cli eval skill recommend?

It maps goals to built-in metrics: multi_turn_task_success for whether the agent achieved the user's goal, multi_turn_trajectory_quality for the reasoning path, multi_turn_tool_use_quality for tool calls, final_response_quality, hallucination or grounding, safety, and final_response_match against a golden answer. When no built-in fits, you write a custom LLM-judge or Python metric.

How do I install the agents-cli eval skill?

Run uvx google-agents-cli setup to install the agents-cli command and all seven skills, or add only this one with npx skills add https://github.com/google/agents-cli --skill google-agents-cli-eval. The skill needs the agents-cli binary to run evals.

Do I need Google Cloud to run agents-cli evals?

Partly. Managed built-in metrics and commands such as eval analyze call the Eval Service in Gemini platform, which defaults to the global endpoint. The skill says local custom metrics with a custom_function grade in-process with no Google Cloud project or region required.

Should I use eval optimize to fix failing cases?

Only when you ask for prompt optimization. The skill says GEPA optimization is long-running and expensive, so the agent should iterate with manual fixes first and then run a single final eval optimize, never a loop. It is for prompt-only failures in ADK Python projects.

More in Skill directory

All guides

Keep your prompts and skills in one place, in your own GitHub. Install every skill with one command.

Get started

Kipt is an independent product, not affiliated with or endorsed by Anthropic, OpenAI, Cursor or GitHub. Claude, ChatGPT, Codex, Cursor and GitHub are trademarks of their owners.