To The Moon

In the early days of the cloud era, when dissecting costs most people just looked at the basic price of compute and storage (for services like EC2 and S3). Overtime, the real focus shifted to egress.

In today’s AI world, the initial focus is on the input/output pricing per 1M tokens, but the real focus is on the cache read rate. What is the cache read rate? The cache read rate is what a model provider charges to essentially re-reat input tokens it processed / stored from a prior call. If you think about an agent loop with the same system prompt, same tool definitions, etc, all of this history / context doesn’t need to be resent and reprocessed every time (and this turns out to be a majority of input tokens).

Here’s some data on the headline prices everyone looks at (just a blended input/ouput of the vanilla pricing). I use 80/20 input / output ratio because that’s a reasonable approximation of a normal chat workload.

As you can see above, Astra / Fable (those are thought of as the true frontier) both price at $10 / 1m input tokens and $50 / 1m output tokens (blended to $18). Another takeaway is a “mid tier” has emerged in the $2 / $10 range (~$4 blended). And again, this chart is just the vanilla pricing.

In the next chart below I try to look at a more reasonable workload - a 40 turn agent session with a 85% cache hit. The order doesn’t really change, but a couple call outs. Fable comes in ~16% under the Astra pricing (when vanilla list is the same). $SpaceX(SPCX)$ Grok 4.7 is the cheapest flagship model on vanilla pricing, but is more expensive than Sol and Terra (xAI charges $0.50 per cached token while OpenAI charges $0.10)

How I interpret these two charts - all the model providers are running a price war on the acquisition price, when in reality they’re actually competing on the retention price (and gives us a peak at what the models may be more optimized for). The vanilla list price will attract users, top of funnel, etc. Cache hit price is what actually gent’s spend. Astra, for instance, is $1 per million cached tokens. That feels more priced for chat. Fable at $0.25 is more for agents. Luna at $0.01 is priced to run in a loop.

So bringing this all back to the egress analogy. Egress really mattered because it created real switching costs. Your data lived in one cloud, and moving it out was EXPENSIVE. A cache is kind of similar. Swapping out models mid session means building that cache over again. Now, the natural push back is “well who’s switching models mid session…”Fair, but an entire category was built on just this! Model routers have the pitch of “send every call to the cheapest model that can handle the task appropriately.”

Just looking at the vanilla pricing - that’s smart! But under cache pricing this can become a trap - because the cache is keyed to one model / one provider. So every model switch is a “full vanilla price re-read” of the whole context. Looking at the 40-turn agent session I talked about earlier - All on Fable at a 95% cache hit rate: ~$4.90.

But if you keep alternating turns between Fable and Sol (which btw is 1/5 the vanilla price), and you get to $18! Routing to the cheaper model ends up costing ~4x than not routing and hitting the cache. Per-call multiplexing is probably dead for agents, as routing only really makes sense at the session level. So there is some switching costs built in here…Take it one step further…

…I also deeply believe the future is very long running agents. Or agents that are perpetually running in a loop (think AI employees just continuously doing their job running 24x7). Then the context and the cost and all the cached info starts to really matter! This may be too far-fetched of a conclusion to draw, but I think there is a world where lock in emerges from the cache-hit pricing as agents start running longer and longer.