45 billion tokens a day: how AT&T tries to keep its AI bill flat

AT&T says its AI use grew from about 8 billion tokens a day to over 45 billion while costs stayed flat. Here is the method, and what it leaves open.

AT&T says it now runs an average of 45 billion AI tokens per day. According to Fierce Network, the figure was about 8 billion a day last year. The same report says costs have stayed relatively flat.

If you run a business in India and have started paying for AI tools, that second sentence is the one to sit with. Most of us have seen the opposite. A pilot works, more teams start using it, and the monthly invoice climbs faster than the value anyone can point to.

What a token is, and why the bill follows it

A token is a small chunk of text, roughly a word fragment, that a language model reads or writes. Providers charge by the token. Every question you send, every document you paste in, every reply you receive is counted.

Think of a tea stall that charges by the cup. A few customers, and the bill is small. Hand every customer in the building a free cup and the stall's cost is no longer a footnote.

AT&T's own numbers show how fast this can happen. A diginomica report quotes its chief technology officer, Jeremy Legg, saying the company is burning a trillion-plus tokens a month, with consumption moving up by double digits. The same report says AT&T has over 100 generative AI models in production and handles over 300,000 calls a day, which it transcribes and analyses.

The method: send each task to the cheapest model that can do it

According to AT&T's own write-up by its chief data and AI officer, Andy Markus, the company built an AI gateway that routes each prompt to a cost-effective model. At every turn it weighs speed, cost and expected output quality. It can even switch models partway through a conversation.

Fierce Network adds the detail that makes this interesting. The router is aware of the cache. When a model has already read a long conversation, repeating that text costs about one-tenth of the normal price. Switch to a different model and you lose that discount. So for a long coding session, staying on the current model can be cheaper than moving to a "cheaper" one. Fierce reports that off-the-shelf routers generally do not account for this.

The second lever is open models, which a company can run itself. AT&T has built telecom-specific models, called OTel 1.0 and OTel 2.0, fine-tuned from Google's Gemma 4 according to Fierce. It aims to serve 65 to 70 percent of its tokens through open models eventually, and expects 25 to 40 percent this year.

The third lever is the plainest. AT&T is reviewing about 150 AI use cases in production. Seven of them, mostly classification, turned out to be suitable for traditional machine learning instead of a generative model at all.

Read together, the layers are these: the gateway decides, the open models absorb the routine work, and the premium models are kept for the hard cases. Sometimes the cheapest token is the one you never send.

What the sources do not tell us

The 45 billion figure and the flat-cost claim come from AT&T and its executives. No source gives a dollar amount or a percentage. AT&T's own post says it is "reducing AI costs as much as 90%", but offers no baseline, time period or method, so treat that as an upper bound and nothing more.

Fierce also notes that open models are estimated to run six to ten months behind the leading closed models, and that this is an estimate. The 25 percent of tokens that could move over applies only to closed-model use cases more than a year old, not to everything AT&T does.

And AT&T is a very large company with its own engineers. Whether a business with ten staff and one chatbot gains much from this machinery is not something these articles answer. The honest reading is that the principle travels better than the tooling.

Three things to try this week

  1. List every AI tool your team pays for and write down what each costs per month. If the usage is not visible, that is your first finding.
  2. Pick the single highest-spend use case. Ask whether it truly needs the most capable model, or whether a smaller one gives an answer your team would accept. Test twenty real examples side by side and keep the results.
  3. Look for one task that is really classification, such as tagging support tickets or sorting invoices. Ask whether a simple rule or a basic machine learning model could do it. AT&T's advice to other operators, as Fierce reports it, is to start with the most obvious, highest-cost use cases ranked by token spend.

None of these needs new software. They need a spreadsheet and a few hours of curiosity.

Which of your AI tasks would you still pay for if you had to see its cost on every single request?

By Nikhil Teltumbade, Director, Godrix AI Solutions.
Working on something similar? Godrix AI Solutions builds AI automation and apps for business workflows. Book a free audit.

← All posts