DEV Community

Cover image for A tool call that came back as text
Tom Jones
Tom Jones

Posted on Edited on

A tool call that came back as text

One benchmark gave us a quiet failure. The model wrote a tool call, and our gateway handed it back as chat text.

The call that got lost

Picture 1. The prompt and the API disagree about one character. The model followed the prompt.

The setup is small. The ToolACE prompt (from the ToolACE paper) lists tools by their original names, like SleepOptimizer.configureRoom, and asks the model to write calls as text: [SleepOptimizer.configureRoom(...)]. Many tool APIs reject a dot in a function name, so we offered the tool as SleepOptimizer_configureRoom.

The model then did something reasonable. It wrote the call as text, using the original name from the prompt. Our recovery step turns a written call into a real tool call, and it did not recover the dotted name, so the call text went to the user as the answer.

One narrow rule

Picture 2. Only a dot or a space may stand in for an underscore.

The fix maps a written name to the one offered tool, and only where a dot or a space stands in for an underscore. A hyphen or a slash gets no such pass, because a-b(x=1) and a/b(x=1) read as arithmetic, and turning arithmetic into a tool call is a worse bug than the one we started with.

Four parser traps

Picture 3. Four parser traps, each with a test.

The parser had more traps than I expected. Review turned up four scanner traps, listed here.

  • a triple quoted string that the rewrite could change
  • a # comment that threw the scanner off by one
  • a lone carriage return, which ends a Python comment
  • a trailing comma that blocked recovery

Each one got a test, and each test was checked by breaking the code and watching it go red.

What changed

Picture 4. One internal replay on our Base lane: 337 text calls, 0 recovered before, 328 after.

The numbers, from one internal replay of a ToolACE sample on our Base lane (our own run, which also included other tasks, so treat it as a lab number): of 1,998 answers, 337 were text calls. The old gateway recovered none of them. The new one recovered 328, and no answer outside ToolACE changed. The other 9 stay as text, because the written name contains a slash, which is also division, so recovery refuses them.

What I would take from it

What I would take from it: a model that follows your prompt and a gateway that follows your schema can both be right and still lose the call between them. If you map names, make the map as narrow as you can defend, then try to break it with inputs that look like math.

Has anyone else seen a prompt and a tool schema disagree about a name?

Top comments (0)