DABYTE DATA DESK · standards

Writing MCP tool definitions a model calls correctly

Answer. Most tool descriptions say what a call returns and never say when to reach for a different tool. Across 362,226 scored MCP tool definitions, the two lowest-scoring dimensions are Behavioral Transparency (mean 2.90, 46.1% below the passing bar) and Usage Guidelines (mean 3.03, 44.5% below), while Purpose Clarity sits at 4.47 with only 4.0% failing — the corpus knows what its tools do and not when to use them, and those two weak dimensions carry 40 percent of the scoring weight. Four changes address it: name the alternative tool explicitly in the description, put behavioral annotations on every tool, keep parameter meaning inside the input schema, and declare an outputSchema with structured content. Two cautions govern all of them. An annotation that contradicts the prose is both a scoring failure and a false statement. And the one controlled study on rewriting descriptions found that augmenting every component raised task success by a median of 5.85 percentage points while raising execution steps by 67.46 percent — so the target is density, not volume.

Published

Who reported it, and when we saw it

10 sources in our monitored set. First seen by us at 2026-08-09 14:59 UTC; our polling resolution is one hour, so this is when we observed it, not when it was published.
SourceLink
https://github.com/glama-ai/tool-definition-quality-score
https://modelcontextprotocol.io/specification/2026-07-28/server/tools
https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main
https://blog.modelcontextprotocol.io/posts/2026-03-16-tool-annotations
https://arxiv.org/abs/2602.14878
https://arxiv.org/abs/2602.18914
https://www.anthropic.com/engineering/writing-tools-for-agents
https://glama.ai/mcp/connectors/ai.dabyte/dabyteai
https://github.com/creanlab/ai-visibility-index/commit/acbea9aa1925960
https://registry.modelcontextprotocol.io/v0/servers?search=dabyte

Key facts

Where tool definitions actually fail

Glama's Tool Definition Quality Score (TDQS) is an open rubric for how well an MCP tool definition communicates to an agent. As of June 2026 — the figure published in its README when we read it on 2026-08-09 — it had scored 228,369 current tools across 15,036 MCP servers, and 362,226 tool definitions in total counting superseded versions retained from earlier captures. Six dimensions, weighted: Purpose Clarity 25 percent, Usage Guidelines 20, Behavioral Transparency 20, Parameter Semantics 15, Conciseness and Structure 10, Contextual Completeness 10. Tier A starts at 3.5, tier B at 3.0. The README states that the tool-level figures aggregate all 362,226 scored definitions rather than only the current tools. Broken out that way, the corpus is strong on what a tool does and weak on when to reach for it. Purpose Clarity has a mean of 4.47 with 4.0 percent of tools below the passing 3, and Conciseness and Structure matches it at 4.47. Usage Guidelines means 3.03 with 44.5 percent below. Behavioral Transparency means 2.90 — under the passing bar on average — with 46.1 percent below. Those two dimensions carry 40 percent of the weight between them, and 56.4 percent of tools carry at least one smell, the rubric's term for any dimension scoring under 3. That shape matches what a description looks like when someone writes one in a hurry: a sentence about the return value. It answers a question the agent was not stuck on. The question it was stuck on is whether this call or a different one is the right move.

Say when not to use it, and name the tool that is

The rubric's anchors for Usage Guidelines are explicit about where the points are: 5 requires explicit when, when-not and named alternatives; 4 is clear context with no exclusions; 3 is implied usage; 2 is no guidance; 1 is misleading. The distance between 4 and 5 is one sentence — the one that names the other tool. Two independent audits found the same hole before TDQS did. A study of 856 tools across 103 MCP servers (arXiv 2602.14878, submitted 2026-02-16 and revised 2026-05-31) found that 97.1 percent of descriptions carry at least one smell and 56 percent fail to state their purpose clearly; the TDQS README, summarising that same audit, puts the share that never say when a tool should or should not be used at 89 percent, a figure we could not locate in the paper's own abstract. A second study of 10,831 MCP servers (arXiv 2602.18914, submitted 2026-02-21) ran a controlled mutation experiment and found that among functionally equivalent servers, standard-compliant descriptions reached 72 percent selection probability against a 20 percent baseline. What those studies measure is tool selection. The specific failure we describe next is not measured, and we want to be clear about the difference. An agent that can look up one record and can also fetch the whole table will reach for the per-record call, because that call maps onto the nouns in the question. If the table tool does not say to prefer it over calling the per-record tool repeatedly, and the per-record tool does not point at the table tool for whole-field questions, nothing in either definition rules out twenty calls where one would do. That is the reasoning we followed when rewriting our own definitions, and it is recorded in the commit message — it is not a result we A/B tested. The cost would land on context window and latency and would show up in no log as a failure. Anthropic's tool-writing guidance points the same direction (published 2025-09-11): "More tools don't always lead to better outcomes", with a recommendation to build a few tools targeting high-impact workflows and to consolidate multi-step operations rather than wrap every endpoint. Name the alternative by its exact tool name. Telling a model to use another tool if this one is not right is not actionable; a literal tool name is.

What fuller descriptions bought, and what they cost

The same 856-tool study also measured what happens when you act on this advice, and the result is a caution worth reading before you rewrite anything. When the authors augmented descriptions across all of their components, task success improved by a median of 5.85 percentage points and partial goal completion by 15.12 percent — but the number of execution steps rose by 67.46 percent, and performance regressed in 16.67 percent of cases. That is close to the opposite of the intuition that a fuller description makes an agent do less work. It does not retire the when-not-to-use sentence, which is one line rather than an augmentation across every component, but it does rule out the general claim that more description reliably means fewer calls. We have not measured our own rewrite against that baseline, so we cannot say which side of it we landed on. The same paper's ablations point at the usable version: compact combinations of components often preserved behavioral reliability while cutting token overhead. TDQS encodes the same instinct in a Conciseness and Structure dimension worth 10 percent, where the corpus already scores 4.47. The budget you spend restating the schema in prose is taken from somewhere. Density, not volume.

Annotations belong on every tool, and the defaults bite

MCP's ToolAnnotations carries four behavioral fields, and their defaults are not neutral. From schema.ts at spec revision 2026-07-28: readOnlyHint defaults to false, destructiveHint to true, idempotentHint to false, openWorldHint to true. Omitting the block therefore does not mean unspecified. A client reading the defaults sees a tool that may modify its environment, may do so destructively, is not safe to repeat, and reaches an open world of external services. For a read-only lookup that is four wrong statements, and a client that gates on hints will interrupt the user for confirmation before a read. That is the reason to emit all four. The reason is narrower than it first looks, and the schema says so itself. Both destructiveHint and idempotentHint are annotated as meaningful only when readOnlyHint is false. On a read-only tool you declare those two to overwrite the defaults a client would otherwise assume, not because the schema treats them as substantive claims about your tool. Emitting them is still correct; describing them as guarantees about a read-only tool is not. The hints are not enforcement, and it is worth being precise about that before selling them as safety. schema.ts notes that all ToolAnnotations properties are hints, not guaranteed to provide a faithful description of tool behavior, and that clients should never make tool use decisions based on annotations received from untrusted servers. The spec page states an adjacent requirement as a MUST: clients must consider tool annotations to be untrusted unless they come from trusted servers. The MCP blog post of 2026-03-16 sets out what they are good for instead — driving confirmation prompts, enabling graduated trust across deployment contexts, helping users understand what a tool does without running it, and feeding a policy engine. Hard guarantees need sandboxing or network controls, and an untrusted server can lie.

The honesty check that catches otherwise good servers

TDQS grades Behavioral Transparency relative to the annotations, and it has one hard gate: a description that contradicts its annotations scores 1 on that dimension and raises a public Annotation Contradiction flag. In the corpus 0.55 percent of tools trip it, alongside 1.26 percent with no description at all and 0.06 percent that are tautological. The rubric's worked example is a description claiming a tool creates something while its annotations declare readOnlyHint true. The variant that catches servers written in good faith is quieter. Write in prose that the server queries engines live, then set openWorldHint false because the code actually reads a file at call time. Both halves cannot be true. It is a rubric failure and, separately, a false statement to every human and machine that reads the definition. Our own case is public because the fix is. The measurement runs as a weekly pipeline, and the server reads the published JSON at call time rather than reaching anything external, so openWorldHint false is accurate — and the prose is written to match rather than claiming live engine queries. readOnlyHint true is the other claim that has to hold, and it does. The two middle fields are where we nearly talked ourselves into an error, so it is worth naming. Because the schema makes destructiveHint and idempotentHint meaningful only when readOnlyHint is false, neither carries a substantive claim on a read-only tool. And idempotence in MCP is defined as repeated calls with the same arguments having no additional effect on the environment — a statement about side effects. It is not a statement about the return value being stable, so the fact that our figures do not change until the next weekly release is a different property entirely and is not evidence for that annotation. There is no tooling that checks any of this. Read each factual claim in the description against each annotation, by hand, before you ship.

Put the meaning in the schema: parameters and output

TDQS scores Parameter Semantics on what the description adds beyond the input schema, and the baseline moves with schema coverage: above 80 percent of properties carrying a description, the baseline is 3 even when the prose says nothing about parameters; below 50 percent, the description has to compensate; a tool with no parameters starts at 4. Note what that means for our own evidence before you take it as proof. Only one of our five tools takes a parameter at all, so the other four collect the no-parameter baseline of 4 on this dimension rather than earning it, and the practice described here is exercised on exactly one property. That property, a brand identifier, carries the pattern ^[a-z0-9-]{1,80}$, three concrete example values, and one exclusion — not the display name. The prose says only what a schema cannot: that an unknown identifier returns an error naming every valid one, so a second attempt can succeed, and that a brand absent from the index has not been measured at all, which is a different thing from a measured zero. Prose that restates the parameter list duplicates the schema and drifts from it at the next change. Output is the other half, and here the spec is directive. A tool may declare an optional outputSchema; once it does, servers MUST provide structured results that conform to it and clients SHOULD validate against it. Results then carry a structuredContent field alongside the usual content array. This is an addition, not a replacement — the spec says a tool returning structured content SHOULD also return the serialized JSON in a TextContent block for backwards compatibility. The spec also notes that structuredContent is server-produced result data and is unrelated to LLM structured outputs, meaning schema-constrained model generation; same words, different mechanism. For a data server the argument is blunt: a server whose purpose is measurements should not hand them over as a string for the model to parse back out. Declaring the schema also buys description budget, since TDQS scores Contextual Completeness with the note that a description need not explain return values when an output schema exists.

How many tools, and how to verify what you shipped

The count guidance is narrower than keep it small, and the popular version of it is wrong at the bottom end. TDQS anchors Tool Count Appropriateness at 5 for each tool earning its place, typically 3 to 15; at 3 for borderline, where 1 to 2 feels thin and 16 to 25 feels heavy; and at 1 for extreme mismatch, meaning 50 or more, or a single trivial tool. Fewer is not automatically better. Tool count is the lowest of the server-coherence dimensions in the corpus at 3.87, against 4.57 for naming consistency and 4.50 for disambiguation, so the common failure is a surface badly scoped in either direction. The second half is that the score is computed from the published endpoint, not from your source tree. Glama re-introspects a hosted connector's tools in place and re-scores a definition when its inputHash no longer matches the recorded score, with a sweep job every five minutes in batches of 100. Editing descriptions locally changes nothing until you redeploy. Verify with a tools/list call, not with the diff. One scope limit is worth knowing before you plan around the rubric at all: it grades callable tools, so a server exposing only resources gets little from it. Our change is one public commit: acbea9a in creanlab/ai-visibility-index, dated 2026-08-05, touching a single file, mcp-server/server.py, with +176 and −14 lines. Its message records the state it replaced — five descriptions of one sentence each, annotations absent entirely, and measurements handed over as a string. The connector scores A 4.9/5.0, last tested 2026-08-09 13:34, across five tools that each declare an outputSchema and all four annotations. One disclosure belongs next to that number: Glama publishes no score history, so there is no recorded score from before the commit. The commit message is the only record of the prior state, and the 4.9 is the current score rather than the measured other end of a delta.

Questions this answers

Do annotations actually stop a client from calling a destructive tool?

No. MCP's schema.ts states that all ToolAnnotations properties are hints, not guaranteed to describe behavior faithfully, and that clients should never make tool use decisions based on annotations from untrusted servers; the spec page separately requires clients to treat tool annotations as untrusted unless they come from trusted servers. Per the MCP blog post of 2026-03-16, annotations drive confirmation prompts, graduated trust, helping users understand a tool without running it, and policy-engine inputs. Hard guarantees need sandboxing or network controls. Set them because omitting them makes a client assume the worst defaults, not because they enforce anything.

Should I set destructiveHint and idempotentHint on a read-only tool?

Emit them, but do not present them as claims about your tool. schema.ts marks both as meaningful only when readOnlyHint is false, so on a read-only tool they carry no substantive meaning. The reason to include them anyway is that omitting the annotations block leaves the client with the defaults, which are destructiveHint true and idempotentHint false. Keep the definition straight too: idempotence in MCP means repeated calls with the same arguments have no additional effect on the environment. It is not a claim that a call returns the same data twice — cached or frozen output is a different property and is not evidence for that field.

If I declare an outputSchema, can I drop the text content block?

No. The spec says a tool returning structured content SHOULD also return the serialized JSON in a TextContent block for backwards compatibility, so send both. Declaring the schema does bind you: servers MUST provide structured results that conform to it, and clients SHOULD validate them, so a schema that drifts from what the server actually returns is worse than none.

I rewrote my descriptions and the score did not move. Why?

Because the score is computed from the published endpoint. Glama re-introspects a hosted connector's tools in place and re-scores only when a definition's inputHash no longer matches the recorded score, with a sweep job every five minutes in batches of 100. Redeploy first, then confirm with a tools/list call against the live URL before you look at the score again.

Machine access