Tool Selection

Tool selection is the model deciding which tool fits the step in front of it.

With three tools it is nearly always right.

With fifty it is unreliable, and accuracy falls as the list grows.

Ten to fifteen well-named tools is a practical upper bound.

Overlap causes more trouble than absence.

Two tools that both sort of search will be chosen between at random.

Merge them, or make the descriptions state clearly when each applies.

Names carry real weight, so search_internal_docs beats search2 by a wide margin.

The description should include when not to use the tool, which people rarely write.

For large tool sets, retrieve a relevant subset before showing them to the model.

That is retrieval over tool descriptions, and it works well.

Grouping tools by task and exposing one group at a time is another approach.

Watch for the model inventing a tool that does not exist, which happens under pressure.

Reject those cleanly with a message listing the real options.

Log which tool was chosen for which request, since the pattern of errors shows what to rename.