Why your AI bill is growing, and what actually reduces it
AI usage tends to creep. A new feature launches, adoption climbs, and the invoice surprises you. The goal of this playbook is simple: spend less without losing quality. You will learn how to pick the right model for each job, keep token counts under control, cache repeated work, and set up useful metrics that reflect customer value, not just server costs.
We will avoid vague advice. Instead, we’ll focus on specific changes you can ship in days, not months. These techniques work for support chatbots, internal copilots, research assistants, and content tools. Most are vendor‑neutral and apply whether you use hosted APIs or run your own models.
Start with utility, not dollars: measure the right thing
Cost matters, but the only number that really counts is cost per unit of value. Define value clearly for your product. Here are common choices:
- Support resolution: cost per issue resolved without human handoff.
- Docs retrieval: cost per correct answer with reference citation.
- Content drafting: cost per accepted draft above an internal quality bar.
- Coding assistant: cost per accepted change or test passed.
Build a small “golden set” of tasks that represent real work. Log the model, context length, and outcome for each run. Your cost decisions should follow these utility metrics, not raw tokens or GPU hours alone.
Right-size the model, right-size the context
Model selection that adapts in real time
The biggest savings come from routing easy tasks to small, cheap models and reserving premium models for hard problems. Use a two-stage policy:
- Stage 1: lightweight attempt. Try a small, fast model on the task. Ask the model to estimate its own confidence on a narrow scale (for example, “Sure/Unsure”).
- Stage 2: escalate when needed. If confidence is low or the user pushes back, escalate to a larger model or add retrieval. Keep the transcript, but truncate or summarize earlier turns to control context size.
This simple gate can cut spend by 30–70% while keeping outcomes steady. Add a timeout rule: if latency is high or the queue is long, prefer smaller models to avoid slow, expensive escalations that erode user satisfaction.
Context window hygiene
Large context windows are convenient, but they invite waste. Focus on precision retrieval and lean instructions:
- Instruction IDs: Store your system prompt as short, stable tags (e.g., “Policy A: tone=helpful, citations=true”). Expand them server-side so the LLM sees the full text, but only when necessary.
- Chunk right: Use content chunks near 300–800 tokens with overlap tailored to your domain (more overlap for code, less for plain text). Chunk too small and you pay more in extra hits; too big and you pay for irrelevant tokens.
- Selective history: Don’t send the whole chat back. Keep the last user message, the last model reply, and a compressed recap of earlier turns. Summaries reduce drift and cost.
Token diet: trim inputs and tame outputs
Cut boilerplate without losing safety
Most prompts carry repeated policy text. Streamline it:
- Policy macros: Create server-side macros for compliance, style, and formatting. Insert only relevant macros per request. Avoid “one mega prompt to rule them all.”
- Few-shot alternatives: Replace long few-shot examples with short pattern descriptions and a single canonical example. Use JSON schemas and explicit keys to reduce ambiguity.
- Stop sequences and max tokens: Set strict output limits and stop tokens. When possible, use providers’ “structured output” or “JSON mode” to avoid verbose prose.
Structure beats prose
Ask for structured output preferentially. Even a simple format like key-value pairs can reduce output bulk and post-processing. If the model must write longer text, require a short outline first and then a second call to expand only the accepted sections. Editing beats regenerating.
Caching that actually helps
Two layers of reuse
Think about caching as two separate wins:
- Input cache: Fingerprint the full request payload (prompt + parameters + tools + retrieval targets). If the hash matches a recent request, reuse the response for a short time (minutes to hours). For consumer chat, a few minutes is plenty; for documentation bots, you can often cache for days.
- Retrieval cache: Cache embedding vectors and nearest-neighbor results per document version. Tie the cache key to a content hash so updates bust the cache automatically.
Near-miss matching
Exact match caching misses common rephrasings. Add a small approximate input matching step:
- Before hitting the LLM, embed the user query and compare to recent queries. If similarity is above a threshold, reuse or lightly edit the prior answer.
- Log cache reuse to keep an eye on drift. If users often correct reused answers, lower the threshold or shorten the cache TTL.
Retrieval that’s cheaper and sharper
Get selective with vectors
Embedding size matters. Smaller embeddings with a good index (like HNSW or IVF-Flat) can be far cheaper than large embeddings with brute-force search. Start with medium dimensions and measure accuracy on your golden set before upgrading. Include cheap metadata filters (language, product area, date) to reduce candidate sets before scoring.
Rerank only when needed
Save expensive cross-encoders for tough cases. First, fetch top-k with a fast vector search. Then, rerank for the top slice only (e.g., top 20 down to top 5). If the query is simple or the user has a narrow filter set, skip reranking entirely.
Distill knowledge to reduce context
When fine-tuning pays off
Fine-tuning is not just an accuracy play; it’s a context compression tool. If you keep pasting the same policies, style notes, or domain examples, consider a lightweight finetune or parameter-efficient approach (like LoRA or adapters). The goal is to move repeated instructions into the model’s parameters so your prompts shrink.
Rules of thumb:
- At least a few thousand high-quality examples before you try fine-tuning.
- Use counterexamples to prevent overfitting to the happy path.
- Keep a “shadow” evaluation set that never enters the tune.
Even a modest finetune can halve token usage while improving consistency, which lowers rework and back-and-forth turns.
Structured tools reduce chatter
Prefer functions over open-ended dialogue
Where possible, give the model tools with clean inputs and outputs (think “search_products(query)”, “book_slot(time)”). You will see fewer speculative paragraphs and more precise calls. This cuts tokens and shrinks failure modes. Add tight parameter validation so bad calls fail fast and cheaply. Consider logit bias or tool selection hints to discourage irrelevant tools.
Batching, streaming, and smart retries
Batch work that users don’t wait for
Many workloads tolerate delay. Batch them:
- Precompute embeddings for new documents on ingest, not at query time.
- Nightly cleanups: re-summarize long chats, refresh stale caches, and rebuild outlines.
- Bulk operations: run dozens of small prompts in one API batch when the provider supports it. This reduces per-call overhead.
Stream results when users are watching
For interactive flows, stream partial results. Users start reading sooner, so you can adopt smaller models or reduce retries. If the first tokens look wrong, cancel early to avoid paying for a full generation.
Self-hosting without headaches
When to bring models in-house
Self-hosting pays off when your traffic is steady, latency matters, or data constraints force you on-prem. But savings only appear if you hit good utilization. Key levers:
- vLLM or similar runtimes for high throughput and KV cache reuse.
- Quantization (e.g., 4-bit or 8-bit) to fit larger models on smaller GPUs with small accuracy tradeoffs.
- Autoscaling and request shaping so you don’t pay for idle capacity. Gate long-context or low-utility tasks during peak hours.
Watch for hidden costs: storage for models, observability, on-call burden, and security reviews. If peaks are spiky, a hybrid model makes sense: baseline on your cluster, burst to a provider when queues grow.
Contracts, pricing, and guardrails
Know how your provider bills you
Many providers bill on input and output tokens, sometimes at different rates. Long contexts are a silent budget killer. Negotiate volume discounts, and check for committed use options if your traffic is predictable. If you use multiple vendors, build a small abstraction layer so you can route to the best price-performance model per task without a rewrite.
Set hard limits and graceful degradation
Enforce guardrails so no single user triggers outsized cost:
- Daily and monthly token caps per user and per tenant.
- Maximum context and generation size per call.
- Forced compression or summarization when a session grows beyond a threshold.
- Fallback to smaller models or answer short stubs when budgets are tight. Communicate clearly with users to maintain trust.
Observability that drives decisions
From logs to learning
Instrument every request with a correlation ID, token counts, latency, model, and success flags from your golden set. Roll these up into Utility per Dollar dashboards. Look for:
- Prompts that produce frequent retries or user corrections.
- Large contexts with low citation usage in answers (waste signal).
- High-churn retrieval indices that thrash caches.
Run weekly experiments. If a smaller model matches utility within a confidence band, switch and redeploy. Make it routine.
Anti-patterns to avoid
- One mega-prompt for everything. It bloats context and hides problems.
- Unbounded chat history. Old turns become token drag and confuse the model.
- Reranking by default. Save it for tough queries.
- Retrieval everywhere. Many tasks need none; others need only a small reference.
- Ignoring output limits. Always cap generations and set stop sequences.
- No cache busting. Tie caches to document versions so facts stay fresh.
Three quick wins you can ship this week
1) Add an input and retrieval cache
Compute a stable hash of the prompt plus parameters. On a hit, reuse the answer for a few minutes. For retrieval, cache top-k results for each document version. Expect immediate savings for popular questions and shared assets.
2) Replace long few-shots with a schema and one example
Ship a JSON schema and validate outputs. Less prompting, less error, fewer retries. Ask for an outline first for long drafts to avoid full rewrites.
3) Route easy tasks to a small model
Define “easy” by pattern: short answers, FAQ matches, simple lookups. Escalate only on low confidence or negative user feedback. Keep a daily report to verify quality stays level.
Case snapshots
Internal knowledge bot
Before: single large model, long prompts, full chat history sent every turn. After: small model first, escalate on low confidence; history summarized; top-k retrieval cached with document hashes. Result: 55% cost reduction, slightly faster answers, higher citation rates.
Support triage
Before: default to max context; freeform output; frequent retries. After: intent classifier on a tiny model, structured tool calls for ticket routing, strict max tokens. Result: 40% fewer tokens per ticket, lower handoff time to agents.
Marketing drafts
Before: generate full blog drafts in one shot, then regenerate after feedback. After: outline-first flow, section-by-section expansion, cached brand rules, constrained tone settings. Result: 60% fewer tokens per accepted draft, faster approvals.
A 30/60/90 plan for LLM FinOps
Day 1–30: baselines and basic controls
- Define utility metrics and build a golden set.
- Instrument tokens, latency, and outcomes per request.
- Add input and retrieval caching with versioned keys.
- Enforce max tokens and stop sequences across endpoints.
Day 31–60: routing and retrieval tuning
- Introduce small-first model routing with confidence gating.
- Re-chunk content and tune top-k to reduce irrelevant text.
- Add structured outputs and schema validation.
- Batch non-urgent jobs; stream interactive ones.
Day 61–90: compression and contracts
- Test a small finetune to absorb repeated instructions.
- Quantize and evaluate a self-hosted model if traffic is steady.
- Review provider pricing; negotiate volume or commit discounts.
- Set budget caps and graceful fallbacks per tenant.
Checklists you can lift directly
Prompt hygiene checklist
- Compact system prompts with macros.
- One canonical example, not ten near-duplicates.
- JSON or key-value outputs wherever possible.
- Outline-first for long pieces.
- Explicit stop tokens and tight max output.
Retrieval checklist
- Chunk to 300–800 tokens with task-aware overlap.
- Filter by metadata before vector search.
- Rerank only top candidates when needed.
- Cache results by document version.
Routing checklist
- Small model first for patterns you trust.
- Confidence gating and user feedback triggers.
- Escalate with summarized history, not the full chat.
When quality dips, diagnose fast
If users complain or acceptance rates slip, look at three places first:
- Context bloat: Are you sending irrelevant history or too many documents?
- Bad retrieval: Are chunk sizes or filters off, causing noisy candidates?
- Overrouting to small models: Did a new task type sneak into the easy bucket?
Roll back one change at a time. Your golden set and A/B toggles should make this safe and quick.
The mindset that keeps costs down
Cost control isn’t a one-time refactor. It’s a product habit. Treat prompts like code. Treat retrieval like a database. Treat models like dependencies that change over time. With the right LLMFinOps habits—clear utility metrics, routing, token discipline, and pragmatic caching—you can keep bills predictable as usage grows. Most teams can get 30–60% savings in a month without sacrificing user outcomes. The rest comes from steady iteration.
Summary:
- Optimize for cost per unit of real value, not just token counts.
- Route easy tasks to small models; escalate only when confidence is low.
- Trim prompts with macros, schemas, and outline-first flows.
- Cache inputs and retrieval results; bust caches with document versions.
- Use selective retrieval and minimal reranking to avoid token noise.
- Consider fine-tuning to compress repeated instructions into the model.
- Prefer structured tool calls over open-ended chat to reduce tokens.
- Batch background work; stream interactive results; cancel early on errors.
- If self-hosting, use vLLM, quantization, and autoscaling to hit good utilization.
- Set budget caps and graceful fallbacks; watch Utility per Dollar dashboards.
