The app works.
Every new user makes it lose money.
Traditional software assumed the next user was almost free. An AI feature does not work that way. Every interaction can hit a frontier model, so inference is a real, variable cost of goods sold, and if nobody designed it deliberately it is usually far higher than it needs to be. We measure where the money actually goes, then engineer it down without cutting what the user can do.
Per-seat pricing assumed every seat cost the same to serve.
That was approximately true for traditional SaaS, where the marginal compute cost of one more user rounds to nothing. It is not true for AI products, where one heavy user's inference bill can be many times a light user's. The result is a structural loss hiding inside a healthy-looking revenue line: a flat-rate plan where a small number of power users are subsidised by everyone else, and nobody noticed because the bill is one number in a dashboard.
The same survey that puts the AI median gross margin at 50 percent found 75 percent of companies changed their pricing or packaging in the past year, and hybrid seat-plus-usage pricing grew from 25 percent to 37 percent adoption in twelve months. The market is repricing around a cost structure that is no longer flat per seat. The engineering has to move with it.
- Every turn resends the whole conversation. The Messages API is stateless. Turn 50 pays full input price for turns 1 through 49 unless someone managed it.
- Caching was switched on and never verified. A one-byte change in the prefix silently bills at full price. Nothing errors. The only reliable check is reading the cache-hit token count back off the response.
- One model handles everything. Routine and hard requests both go to the most expensive model available, because that was the fastest way to make the demo good.
- max_tokens is set to a worst-case guess. It costs nothing on runs that finish early, and burns the entire cap every time a generation loops.
- Nobody knows the cost per active user. There is one blended infrastructure number, not a breakdown by category, which means there is nothing actionable to point at.
$10,312 a month, or $1,400 a month, for the same product.
Below is a full cost model for a chat-based AI product at 10,000 monthly active users, with every assumption written down so you can change one and rerun it. The arithmetic is shown rather than asserted, because the whole point of this service is that we measure instead of guessing.
This is a modeled example with stated assumptions. It is not a client result, and it is not a promise. Rescue Engineers has not run this specific product. The numbers below are arithmetic applied to published model prices, using assumptions we have labeled so you can challenge them. What your own system would save depends entirely on its starting architecture.
- Monthly active users10,000
- Chat sessions per user per month5
- Turns per session10
- Average user message50 tokens
- System prompt, sent every request1,000 tokens
- Naive average assistant response300 tokens
- Naive history handlingFull resend every turn
- Naive modelOne frontier model, no caching
- Frontier rate, Claude Opus 5$5 / $25 per MTok
- Cheaper rate, Claude Haiku 4.5$1 / $5 per MTok
Model prices are first-party API rates for input and output per million tokens, taken from the providers' published pricing pages as accessed 12 September 2026. Model pricing moves. We re-check it against the live page at audit time rather than quoting a number off a marketing site.
The naive bill, in full
Within one 10-turn session, turn k sends the system prompt, every prior turn's user and assistant tokens, and the current message: 1,000 + 350 x (k-1) + 50 tokens, where 350 is one prior turn's combined user and assistant length. Summed across 10 turns, one session is 26,250 input tokens and 3,000 output tokens.
Five sessions per user is 131,250 input and 15,000 output tokens per user per month. Across 10,000 users that is 1,312,500,000 input tokens and 150,000,000 output tokens a month.
1,312.5 MTok at $5 is $6,562.50 of input. 150 MTok at $25 is $3,750.00 of output. Total $10,312.50 a month, or roughly $1.03 per user per month, before a single dollar of infrastructure.
The three levers applied
Prompt caching on the growing session prefix. Every turn after the first reads the already-seen prefix at 0.10x the input rate and writes only the newly appended content at 1.25x. This does not reduce what is processed. It reduces what is billed for the repeated portion.
Output length discipline. A tighter system prompt and an honest max-tokens setting cut the average response from 300 to 150 tokens. That 50 percent figure is a labeled assumption, not a measured average, and it has to be checked against a real response-length distribution before anyone quotes it.
Session-level model routing. Prompt caches are model-scoped, so switching mid-session throws away the cache. Routing therefore happens once per session: 70 percent of sessions run entirely on Haiku 4.5, 30 percent entirely on Opus 5. That split is also a labeled assumption, representative of an assistant-style product where most requests are routine.
With 150-token responses, one session becomes 2,850 cache-write tokens, 16,650 cache-read tokens and 1,500 output tokens. The write-plus-read total is lower than the naive 26,250 purely because shorter responses shrink what gets appended to history each turn.
| Per session, optimized | Haiku 4.5 | Opus 5 |
|---|---|---|
| Cache write, 2,850 tokens at 1.25x input | $0.00356 | $0.01781 |
| Cache read, 16,650 tokens at 0.10x input | $0.00167 | $0.00833 |
| Output, 1,500 tokens at full rate | $0.00750 | $0.03750 |
| Total per session | $0.01273 | $0.06364 |
50,000 sessions at $0.20625 each. Matches the top-down calculation exactly, which is the internal consistency check.
Blended at the 70/30 split, one session costs $0.02800. Same interaction volume, same model family, nothing removed from what the user can do.
$8,912.35 a month, in this model, under these assumptions. Change the routing split or the response length and the number changes with it.
This is a model, not a measurement. It assumes the cache stays warm across a session, which holds for an actively used chat inside the provider's cache window and not for sessions with long idle gaps. It assumes the 70/30 routing split and the 50 percent output reduction hold, and both have to be measured per product rather than inherited from this page. It does not model batch pricing, which does not apply to synchronous chat, semantic caching, which only fits high-repeat-query products, or self-hosting, which only pays past a volume threshold most products never reach.
This example sits toward the high end of plausible savings, specifically because the naive baseline modeled here, no caching, no routing, no output discipline, everything on a frontier model, is a realistic description of a large share of AI features shipped without anyone checking the bill. A system that already does these things well will save far less, and we will say so.
In impact order, with what each one costs you.
Every cost fix trades something. Some trade latency, some trade correctness risk, some trade engineering complexity. A recommendation that does not name its tradeoff is a sales pitch, not an engineering judgment, so each of these carries its own.
Session-level model routing
Send the request to the cheapest model that can plausibly handle it, and escalate only when the cheap model is uncertain or the task classifies as hard. A 2026 arXiv paper on cascaded serving reports that a two-stage cluster-then-escalate cascade retains 97 to 99 percent of the strongest model's accuracy while reducing time per output token.
What it costs you. A second round trip on every escalated request, and a new failure surface: a misclassified request gets a confident wrong answer, not an error. Routing pays off on high-volume, well-understood task distributions and pays off least on open-ended work where most requests escalate anyway. Before building a cascade, we check whether the flagship model at a lower reasoning setting already clears the bar, because one model keeps one cache namespace.Prompt caching on the conversation prefix
The provider stores the exact byte sequence of a prompt prefix and charges a fraction of the input rate to reuse it. Anthropic reads at 0.10x the input rate across Opus, Sonnet and Haiku, and writes at 1.25x for a five-minute lifetime or 2x for an hour. OpenAI caches automatically at roughly 10 percent of the input rate once a prefix passes a minimum length. Google charges a separate cached-token rate plus an hourly fee to keep the cache warm.
What it costs you. Discipline. It is a prefix match, so one byte of drift, an injected timestamp, a reordered tool list, per-user text spliced into the system prompt instead of appended after it, silently bills the rest at full price with no error. The first write is a premium, not a discount. We verify it by reading cache-hit token counts off live responses, not by trusting the parameter.Output length discipline
Unbounded generation is a cost bug, not a feature. Two patterns do most of the damage: a max-tokens cap set far above real need, which is free until a repetition loop burns the whole cap, and free text where the answer is actually structured data, which buys restated context, preamble and markdown the caller then strips out. Schema-constrained structured output fixes the second directly and removes a class of parsing bugs with it.
What it costs you. Tuning work per route. The cap has to match the measured distribution of response length for that specific endpoint, not a worst-case guess, which means instrumenting the distribution first.Context management instead of full-history resend
The most common and most expensive mistake. Because the API is stateless, an unmanaged conversation pays full input price for everything that came before, on every turn. The fixes, ordered by saving against effort: cache the stable prefix, keep a sliding window of recent turns, summarize older turns into decisions and facts, retrieve rather than resend, and for agent loops specifically, prune stale tool results out of the resent history rather than compacting the conversation.
What it costs you. Fidelity, unless you spend the engineering. A window is cheap and forgets. A summary preserves more and costs more to build. Retrieval preserves the most and turns this into a retrieval-quality problem instead of a cost problem.RAG versus long context
Long context puts everything possibly relevant in the window and bills every token on every call unless it is cached, and caching only amortizes well when the same large context is reused across many callers. A per-tenant context has no shared cache to amortize against. RAG indexes once, then retrieves the top-K chunks per query, so query-time input stays roughly constant however large the corpus grows. RAG is meaningfully cheaper per query, typically by one to two orders of magnitude, because the mechanism is a genuine difference in tokens sent.
What it costs you. Retrieval infrastructure and retrieval quality. The exact multiplier depends entirely on how large the long context being compared against is, so we present the mechanism, not a borrowed number, until we know your corpus. Long context plus caching still wins for a fixed document shared across all users.Semantic caching
Different from prompt caching. This matches queries that mean approximately the same thing using embedding similarity, and reuses a prior response for a differently worded question. It fits FAQ-shaped support bots and search-like interfaces where many people ask close variants of one thing.
What it costs you. A wrong-answer risk that looks nothing like an outage. "Cancel my subscription" and "pause my subscription" sit close in embedding space and need different answers. The similarity threshold has no universally correct value: too loose leaks wrong answers, too tight barely hits. This is a heavier decision than a cost-savings framing suggests, and we will usually put it last.Batch APIs
Anthropic, OpenAI and Google all run an asynchronous batch endpoint at roughly 50 percent off standard synchronous pricing, with results returning inside a provider-defined window rather than immediately. OpenAI also publishes a flex tier that matches batch-level pricing for synchronous requests in exchange for slower, best-effort latency, which is a useful middle option.
What it costs you. Latency, so this is a throughput lever and never a user-facing one. It fits nightly summarization, bulk classification, embedding backfills, report generation and enrichment jobs. It does not fit a chat response or a live agent action, which is exactly why it is last on this list rather than first: the discount is the largest and the applicable surface is the smallest.Inference is the fastest-growing line. It is rarely the only one.
Cost to serve is compute plus database plus storage plus egress plus third-party APIs plus the observability tooling that quietly scales with traffic and gets left out of the calculation. A single blended per-user number tells you nothing actionable. The category breakdown tells you where to look.
The egress trap
Bandwidth out of a cloud provider is one of the most consistently underestimated lines in a hosting bill, because it does not appear until traffic is already flowing. The hyperscalers charge tiered per-GB egress on object storage; Cloudflare built R2's core differentiation around charging no egress fee at all. For anything serving real media volume, large API responses or cross-region traffic, egress is worth pricing out explicitly before you commit to a storage provider, not discovered on the first big invoice. We price it against the provider's live pricing page during the audit, because these rates move.
Serverless connection exhaustion
Every serverless or edge invocation can open its own database connection. A Postgres instance maxes out somewhere between roughly 100 and 500 total connections depending on size. Serverless concurrency has no such ceiling, so a traffic spike creates hundreds of simultaneous invocations against a pool that cannot serve them, and the database starts refusing connections while the functions themselves look healthy. The fix is a connection pooler in transaction-pooling mode, which is exactly why every major managed-Postgres provider ships one.
The convenience premium crossover
A managed platform is worth paying for exactly as long as the engineering hours it saves, times your fully loaded cost per hour, exceeds its premium over commodity infrastructure. At low volume that inequality overwhelmingly favours the platform. As volume grows the premium scales roughly linearly with usage while the effort to run commodity infrastructure does not scale with traffic once it is built. So the inequality flips at a specific, calculable volume. Finding that crossover is an exercise, done on a schedule, not a vibe triggered by an uncomfortable invoice.
Migrating providers later is genuinely expensive: re-platforming a database, rewriting infrastructure-coupled code, and carrying the cutover risk of a data or DNS migration. But over-engineering for a scale that never arrives is also a real cost, and in practice it is the more common one. Teams that adopt a distributed, self-managed stack before product-market fit pay an ongoing engineering tax for flexibility they do not yet need. Build on the convenient platform until the premium is verifiably larger than a migration, using the arithmetic above, then migrate deliberately on a planned schedule rather than in a panic.
Six checks that turn a blended number into a ranked list.
Three months of billing, broken out
Cloud and infrastructure billing exports split by service: compute, database, storage, egress, third-party APIs. Not the invoice total, the line items.
LLM usage by model and cache state
Provider usage dashboards broken out by model tier and by cached versus uncached tokens. This single check frequently reveals whether prompt caching is even functioning.
Queries by total time, not peak time
A moderately slow query run 100,000 times usually costs more in aggregate than a very slow query run ten times. We rank by total consumed time.
Egress against what is necessary
Is media served through a CDN with correct cache headers, or hitting origin storage on every single request? The difference is a line item.
The LLM call sites, read directly
We grep the code for max-token settings and for how conversation history is assembled, looking specifically for unbounded generation and full-history resend.
Cost segmented by cohort
Free versus paid, or by plan tier, to find whether a specific tier is being served at a structural loss. That is the pattern that ends in an emergency repricing.
A fixed fee to find it. A base fee plus a capped bonus to fix it.
Two engagements, deliberately separate. The audit replaces a guess with a measured, ranked list and stands on its own. Implementation is scoped from that list, after you have seen it.
Tiered by system size, using infrastructure spend or monthly active users as the band, rather than billed hourly. Hourly billing penalizes us for being efficient at exactly the moment we are asking you to trust our efficiency judgment. One to two weeks. Read-only access. No production changes.
A base fee covering the engineering, plus a smaller, capped bonus tied to measured savings against an agreed baseline over a fixed post-launch window, typically 60 to 90 days. Instrumentation goes in before any change ships, so the before-and-after is not arguable afterwards.
Because the incentives are wrong and the measurement is contestable. Duckbill Group, the best-known AWS cost-consulting firm, explicitly does not price on percentage of savings and publicly criticises contingency billing for cost work as creating misaligned incentives. Three specific problems, independent of that example:
Measurement disputes. The percentage is meaningless without an agreed baseline, and baselines are negotiable after the fact. Traffic growth, an unrelated feature shipping, and ordinary seasonality all make "did the audit cause this" genuinely contestable.
Incentive misalignment. Savings are typically self-reported by the vendor's own tooling. The model structurally rewards finding an inefficient client late rather than keeping a client efficient continuously, because a lean client with little waste to find pays almost nothing for comparable work.
Revenue lumpiness. Zero revenue until measurement completes exposes us to your usage volatility and to any slow-walking of implementation. A firm under that pressure starts optimising for the measurement rather than for your system.
See the full pricing breakdown for every service line.
Exactly what is in scope, and exactly what is not.
The same spec block sits on every service page, so you can compare them line for line. What is excluded matters as much as what is included, because that is where scope disputes come from.
- Current cost per active user, broken down by category: compute, database, storage, egress, third-party APIs and LLM inference, each as its own line rather than a blended average.
- The top five to ten cost drivers ranked by dollar impact, each backed by evidence: billing exports, query logs, APM traces or provider usage dashboards. Never a guess.
- For each driver, the specific fix, an estimated impact range, the engineering effort as small, medium or large, and the tradeoff the fix introduces, named plainly rather than buried.
- A prioritized roadmap ordering the fixes by impact against effort, so you can stop at any point and still have taken the largest wins.
- A direct check on whether prompt caching, batching and routing are actually functioning, which is separate from whether they were configured.
- A gross-margin read: what your current cost to serve implies about margin at today's volume and at 10x today's volume.
- Implementation. The audit diagnoses. Building the fixes is the separate Optimization Engagement, quoted after you have read the report.
- Any production change during the audit. Access is read-only by design.
- A promised savings percentage. We will not quote one before measuring, on this page or on a call.
- Renegotiating your contracts with cloud or model vendors on your behalf.
- A security audit. Cost and security overlap in places, and we will flag anything alarming we trip over, but a real security review is a separate engagement.
- Product or pricing strategy. We will tell you which cohort is served at a loss. What you charge them is your decision.
A written report, roughly 10 to 20 pages, plus the working spreadsheet behind it so you can change an assumption and rerun the arithmetic yourself. A one-page summary written for a board or an investor, and a walkthrough call with the engineer who did the work, not an account manager.
Audit: one to two weeks from the point access is granted, depending on system size. Access delays are the single most common cause of a slower audit.
Optimization engagement: scoped per engagement, typically four to eight weeks for a first wave of fixes, with measurement instrumentation live before any change ships.
Senior engineers only, with production experience that predates the current generation of AI tooling. The engineer who reads your billing data is the engineer who writes the report and the engineer who joins the walkthrough. No junior pool, no offshore handoff, no account manager in between.
- System size, measured by infrastructure spend or monthly active users, which is what sets the audit band.
- Number of distinct services and providers in the bill. One managed platform is a shorter read than nine accounts across three clouds.
- Whether usable instrumentation already exists. A system with no APM, no query logging and no per-model usage breakdown has to be instrumented before it can be measured.
- How many LLM call sites there are, and whether they run through one shared client or are scattered through the codebase.
- For implementation: how many roadmap items you take, and whether any of them touch a data migration.
You send a short description of the system and roughly what you spend a month. We come back with the audit band and the exact access list. Access granted, audit starts, report lands inside two weeks with a walkthrough booked. If you want the fixes built, that gets scoped from the roadmap you now hold, with a fixed price agreed before anyone writes code.
What founders ask about cost to serve.
How much will you save me?
We will not answer that before we have measured your system, and you should be suspicious of anyone who does. Savings depend entirely on the starting architecture. A codebase that already caches prompts, has correct indexes and runs right-sized infrastructure might yield single-digit percentage savings. A codebase built fast with no caching, no batching, full-history context stuffing and every call hitting a frontier model can plausibly see a 60 to 90 percent reduction in the AI inference line specifically, though rarely that percentage off the total bill. The audit is what turns that range into your number.
Is the 86 percent figure on this page a result you achieved for a client?
No. It is a modeled example with every assumption labeled, applied to published model prices. It is not a client result and it is not a promise. We show the arithmetic specifically so you can change an assumption and watch the number move, which is the opposite of a case study.
Do you need production access?
Not for the audit. We need read-only access to billing dashboards, your LLM provider usage console, slow-query logs or APM data, and either read access to the repository or a codebase walkthrough. No production changes happen during the audit. That is diagnosis, not surgery.
Will this make the product worse for users?
That is the right question, and it is why every lever on this page carries its stated tradeoff. Caching and routing are invisible to the user when done properly. Output length discipline is usually an improvement, because verbose preamble was never the useful part. Semantic caching is the one technique that trades a correctness risk for a cost saving, which is why we rank it last and will often recommend against it.
Why not just charge a percentage of what you save me?
Because it creates a measurement dispute and an incentive problem. The percentage is meaningless without an agreed baseline, and traffic growth, an unrelated feature shipping, or ordinary seasonality all make the causation contestable after the fact. It also rewards a firm for finding an inefficient client late rather than keeping a client efficient. We charge a fixed audit fee and a base implementation fee, with a capped bonus as an alignment mechanic rather than the primary revenue model.
My app is not AI-powered. Is this still relevant?
Yes, though the shape differs. Inference is the fastest-growing line in most modern bills, but egress, an unpooled database under serverless traffic, an unnecessary managed-platform premium and an observability pipeline that scales with traffic are all ordinary software problems. The method is the same: break out the categories, rank by dollar impact, fix in that order.
How current are the model prices you use?
The figures on this page were taken from the providers' published pricing pages on 12 September 2026 and are date-stamped for that reason. Model pricing moves, and some published rates are explicitly time-boxed promotions. We re-check every rate against the live page at audit time rather than quoting a number off a marketing site.
- Model pricing and prompt caching mechanics: claude.com/pricing, Anthropic prompt caching documentation, OpenAI pricing, Google Gemini API pricing. All accessed 12 September 2026.
- Model cascading result: "Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving," arXiv:2606.27457, posted 25 June 2026.
- AI-native gross margin: Bessemer Venture Partners, State of AI 2025; ICONIQ Capital, 2026 State of AI.
- Pricing-model shift and the 50 percent median target margin: Kyle Poyar, Growth Unhinged, "The 2026 State of B2B SaaS and AI Monetization Report," 13 May 2026, 230 companies surveyed.
- Conventional SaaS gross margin benchmarks: KeyBanc Capital Markets and Sapphire Ventures SaaS Survey; Benchmarkit 2025 SaaS Performance Metrics.
- Cost-consulting pricing models: GeekWire coverage of Duckbill Group, 2026.
Send us the bill. We will tell you where it is going.
A senior engineer reads your real billing data and your real call sites, then hands you a ranked list with the evidence attached. No promised percentage, no guessing.