Cost, latency & quality

Optimize cost per successful task, not merely cost per model call. Cheap repeated failures can be expensive.

The denominator changes the decision

A small model may cost less per request but need more retries or escalations. A larger context may reduce one retrieval miss while raising every request's cost. Include model usage, tool services, retries, and human review in the accounting boundary you choose, and disclose what is excluded.

How it works

Measure quality and latency on the same representative cases before choosing a cheaper configuration. Caching can help repeated queries, but cache keys must incorporate permissions, relevant versioning, and freshness. Routing easy cases to a smaller model requires a tested routing rule and a fallback for uncertain cases.

A concrete example

In a synthetic batch, configuration A costs $1 for 80 successes; B costs $1.20 for 96. Both cost $0.0125 per success under this simplified accounting. B resolves more cases, but the choice also depends on critical failures and latency. These numbers are invented for arithmetic, not vendor prices.

Apply it to your assistant

Lower B successes to 60 and compare cost per success. Before running the exercise, predict the result. Afterward, explain which assumption changed and add one case where the system should refuse, ask for clarification, or escalate.

All exercise inputs and outputs are deterministic teaching examples. No language model is called. Run the same idea against a versioned dataset before making a production claim.

Key takeaway

Compare feasible configurations using end-to-end success, cost, and latency together.

JavaScript exercise: Cost, latency & quality · code experiment

Lower B successes to 60 and compare cost per success.

const configs = [{ name: 'A', batchCost: 1, successes: 80 }, { name: 'B', batchCost: 1.2, successes: 96 }];
for (const c of configs) console.log({ name: c.name, costPerSuccess: c.successes ? c.batchCost / c.successes : null });
console.log('Synthetic dollars for a fixed toy batch; not current model prices.');

Knowledge check

A cheaper model doubles retries and reduces resolved requests. Which metric helps reveal the tradeoff?

1. Parameter count alone
2. Cost per successful task, with quality and latency constraints
3. The smallest per-token price

Sources

• Holistic Evaluation of Language Models — Liang et al., 2022. A framework for evaluating multiple dimensions of model behavior, beyond one headline score.

Read this lesson as Markdown

Continue learning

• Multi-agent systems — Multiple agents introduce coordination, not automatic correctness. Use them where specialization or independent work has a measurable benefit.
• Scaling & production — Production agents need durable state, concurrency limits, and safe recovery when processes or dependencies fail.
• Observability & monitoring — Observability connects a user-visible outcome to the steps that produced it, without collecting unnecessary sensitive data.
• Cost, latency & quality — Optimize cost per successful task, not merely cost per model call. Cheap repeated failures can be expensive.