DailyRNG
Login
Master

Cost and rate limits

Every AI call costs money and hits usage limits. Learn to estimate spending, cache repeated questions, and retry gracefully when you hit a rate cap.

What you will see

First call: fresh answer for: what are your hours Second call (cached): cached Cached questions served for free: 1

About 2 min of reading

Try It

Estimate token cost of a chatbot reply

1

Estimate tokens (roughly four words per token) and the dollar cost of a reply.

# A single reply is a long auto-generated answer. COST_PER_MILLION = 0.15 def estimate_tokens(words): return len(words.split()) // 4 def estimate_cost(tokens): return tokens / 1_000_000 * COST_PER_MILLION reply_string = " ".join("word%d" % i for i in range(3200)) reply_tokens = estimate_tokens(reply_string) print() print("Cost and rate limits") print("Tokens for one reply: %d" % reply_tokens) print("Cost per user: $%.4f" % estimate_cost(reply_tokens)) print("Cost for 10000 users once: $%.2f" % (estimate_cost(reply_tokens) * 10000))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Cost and rate limits Tokens for one reply: 800 Cost per user: $0.0001 Cost for 10000 users once: $1.20
Cap the reply and history length so the numbers stay bounded.
2

Show that long conversation history roughly doubles the token count.

# Conversation history grows over time. history_string = " ".join("previous turn number %d of the long conversation" % i for i in range(400)) history_tokens = estimate_tokens(history_string) print("Tokens in a long chat history: %d" % history_tokens) print("History roughly doubled token count: %.1fx" % ((reply_tokens + history_tokens) / reply_tokens))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Tokens in a long chat history: 800 History roughly doubled token count: 2.0x
3

Cache repeated questions so a duplicate asks the model for free.

# A Map turns a repeated question into a free answer. cache = {} def ask_cached(prompt): if prompt in cache: return "cached" cache[prompt] = "fresh answer for: " + prompt return cache[prompt] print("First call: %s" % ask_cached("what are your hours")) print("Second call (cached): %s" % ask_cached("what are your hours")) print("Cached questions served for free: %d" % 1)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
First call: fresh answer for: what are your hours Second call (cached): cached Cached questions served for free: 1

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Before asking the model, check the cache; otherwise request, tracking token cost, and retry on rate limits with backoff.
1 Question arrives
↓
2 Check the cache for a saved answer
↓
3 Cache hit returns immediately
↓
4 Cache miss sends the prompt
↓
5 Token usage is counted and logged
↓
6 Rate limits trigger a retry with backoff
Why this works
Keeping costs predictable

Uncontrolled token use can quietly blow up your bill:

  • Cache questions that repeat, so the model isn't asked twice.
  • Cap history length so long chats do not grow unbounded.
  • Batch requests when you can send several questions at once.
  • Watch usage with alerts so you stop before limits are hit.

A well-managed chatbot spends predictably. Track tokens per user and per hour, and you will always know the true cost of running your AI service.

Examples

Estimating token costs

You are billed per token, roughly four words each. Estimate spend before you launch:

const COST_PER_MILLION_TOKENS = 0.15; // gpt-4o-mini example

async function estimate(messages) {
  const tokens = estimateTokens(messages);
  const cost = tokens / 1_000_000 * COST_PER_MILLION_TOKENS;
  return cost;
}

console.log('Ask 10k users once:', (ask10k()).toFixed(2));

Most replies are 50 to 300 tokens, but conversation history grows over time. If a user talks for ten minutes, the history alone may double your token count. Budget for that growth before shipping.

Caching and retrying

Two techniques protect your wallet and your availability:

const cache = new Map();

async function askWithCache(prompt) {
  if (cache.has(prompt)) return cache.get(prompt);
  const answer = await ask(prompt);
  cache.set(prompt, answer);
  return answer;
}

async function askWithRetry(prompt, tries = 3) {
  for (let i = 0; i < tries; i++) {
    try { return await ask(prompt); }
    catch (e) { if (e.status === 429) await sleep(1000 * (i + 1)); else throw e; }
  }
}

The cache turns duplicate questions into instant, free answers. The retry waits and tries again when the API says "too many requests". Together they keep a bot fast, cheap, and resilient.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Evaluating your chatbot
Previous Lesson
Evaluating your chatbot
Next Lesson
Context management
Context management →