From Prototype to Production: What Breaks When You Scale AI Features

TL;DR

AI features can expose different cost, latency, rate-limit, and reliability constraints as traffic changes. Bounded classification calls are candidates for a compiled model, but there is no universal traffic threshold. Sparkient's four public synthetic domains measure 0.886–0.951 macro F1 and 33–42ms batch-average time per item; the runner did not measure per-request p95.

The traffic, provider, price, cache, and incident figures below are an illustrative scaling scenario, not a customer case study or current provider quote.

The Prototype That Works

You built something great. A content moderation system, a ticket triage tool, a lead scoring API. It uses GPT-4o, and it works beautifully:

async def classify_content(text: str) -> str:
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": text}
        ],
        temperature=0
    )
    return response.choices[0].message.content.strip()

At 100 requests per day during testing, the team in this hypothetical scenario accepts the latency, cost, and provider dependency. Those values must come from the actual prompt, model, traffic, and traces.

Then you launch.

Stage 1: 10K Requests/Day — Latency Creep

What breaks

In this illustrative scenario, the live-model classification owns most of the delay between "submit" and "posted." Replace the table below with production traces from the exact endpoint.

Your frontend team adds loading spinners. Your product manager asks "why is this so slow?"

The numbers

| Metric | At 100 req/day | At 10K req/day | |--------|---------------|----------------| | P50 latency | 620ms | 650ms | | P95 latency | 1,100ms | 1,400ms | | P99 latency | 1,800ms | 2,500ms | | Daily cost | $0.50 | $50 | | User complaints | 0 | "it feels slow" |

The P50 barely changes, but the P95 and P99 get worse. More concurrent requests mean more queuing at the LLM provider. Your tail latency is now 2.5 seconds — long enough for users to abandon actions.

Typical fix at this stage

Most teams add caching and switch to a faster model:

# Switch to GPT-4o-mini and add response caching
import hashlib, redis

cache = redis.Redis()

async def classify_content(text: str) -> str:
    key = f"cls:{hashlib.sha256(text.encode()).hexdigest()}"
    if cached := cache.get(key):
        return cached.decode()

    response = await openai.chat.completions.create(
        model="gpt-4o-mini",  # Candidate model; measure quality, latency, and price
        messages=[...],
        temperature=0
    )
    result = response.choices[0].message.content.strip()
    cache.setex(key, 3600, result)
    return result

In the scenario, the smaller model improves the measured profile and caching removes exact duplicates. Unique inputs still take the live-model path; verify the result with current provider pricing and real traffic.

Stage 2: 100K Requests/Day — Cost Explosion

What breaks

In this illustrative scenario, caching helps with duplicates, but most inputs are unique. An 8% cache hit rate leaves 92% of requests on the live-model path.

Your finance team flags the invoice:

| Metric | At 10K req/day | At 100K req/day | |--------|---------------|----------------| | Daily LLM cost | $50 | $450 | | Monthly LLM cost | $1,500 | $13,500 | | Cache hit rate | 12% | 8% | | Effective cost per decision | [illustrative scenario] | [illustrative scenario] |

The per-decision cost hasn't changed. But volume × cost = a bill that's bigger than your compute infrastructure.

The conversation you have

Engineering: "The feature is working as designed." Finance: "It's costing $13K/month. That's more than our entire AWS bill." Product: "We can't remove it — users love it."

Nobody's wrong. The feature works, it costs what it costs, and users value it. But the unit economics don't scale.

Typical fix at this stage

Teams explore three options:

1. Fine-tune a cheaper model — Reduces per-call cost but still linear pricing
2. Batch processing — Classify in bulk during off-hours (only works for non-real-time needs)
3. Start evaluating alternatives — This is usually when teams discover that classification doesn't require an LLM

Stage 3: 500K Requests/Day — Rate Limits and Reliability

What breaks

In the illustrative scenario, traffic bursts exceed the account's current provider limit even though average volume fits. Provider limits change by model and account, so check the current limit and load-test the real concurrency pattern.

The scenario also assumes a provider incident. If classification is a synchronous dependency, its availability and timeout behaviour become part of the application's own reliability profile.

| Metric | At 100K req/day | At 500K req/day | |--------|----------------|----------------| | Monthly LLM cost | $13,500 | $67,500 | | Rate limit errors/day | 0 | 50-200 | | Provider outage impact | "Slow feature" | "Site is down" | | On-call pages/month | 0 | 3-5 |

What you learn

At this scale, three truths become clear:

1. Usage-priced calls can scale with uncached token volume. Model the actual prompt, discounts, retries, and cache hit rate.
2. Third-party dependencies in the hot path are a reliability risk. Every external call is a potential outage.
3. Bounded classification has alternatives. Compare a live model, a smaller or fine-tuned model, rules, and a task-specific classifier on the same cases.

The Compile, Don't Call Approach

Classification workloads have a specific property that makes them different from generative AI: the output space is bounded. You're choosing from approve/review/reject or spam/not_spam or urgent/normal/low. The LLM isn't creating anything new — it's making a judgment call.

For suitable bounded decisions, you can evaluate an LLM as an offline teacher instead of calling it on every normal-path request:

1. Offline: The LLM generates synthetic training examples covering edge cases
2. Offline: A compiled classifier trains on those examples
3. Offline: The model exports to ONNX format
4. Evaluation: Benchmark the ONNX normal path against the under-100ms compiled-stage target; cloud deployments may enable model-dependent LLM escalation

import httpx

# The LLM taught the decision. The compiled model makes it.
async def classify_content(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()

For edge-critical deployments where even the API call is too much, you can run the model locally:

from sparkient_edge import EdgePredictor

# No cloud dependency. Benchmark on the target hardware.
predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Check out this product!"})
# Inspect result.decision, result.confidence, and result.stage

The Scale Comparison

Compare the approaches using the real request distribution rather than a universal volume threshold:

| Metric | Live LLM call | Sparkient cloud | Sparkient edge | |--------|---------------|-----------------|----------------| | Latency | Model, prompt, and provider dependent | 33–42ms batch-average time per item in four public synthetic runs; measure full-path p95, including escalation | Local hardware dependent | | Billing | Provider token usage | Monthly plan credits plus top-ups | Subscription plus your local infrastructure | | Runtime dependency | LLM provider | Sparkient API; optional LLM escalation | No cloud or LLM dependency | | Quality evidence | Evaluate the actual prompt | Evaluate a trained model on the same held-out cases | Same exported model; verify local runtime |

The compiled option wins only if the measured quality, latency, credit usage, and maintenance profile beat the current path for that project.

When to Make the Switch

Don't over-engineer prematurely, but don't wait until you're on fire either.

Stay with LLM calls if:

• Current volume and latency make the simpler live call the better trade-off
• Your classification needs change weekly (new categories, shifting criteria)
• You're still exploring what the feature should do

Switch to a compiled model if:

• Repeated calls create a measured latency, cost, reliability, or privacy constraint
• Your classification categories are stable
• Latency matters (real-time UX, hot path API)
• You need reliability independent of a third-party API

Use edge deployment if:

• Local measurement shows the cloud round trip misses the latency budget
• You can't tolerate any cloud dependency
• You're running on-device or in environments with unreliable connectivity

The Migration Path

You don't have to rewrite everything at once. Here's a progressive migration:

Week 1: Add timing and cost logging to all LLM calls. Get the baseline numbers.

Week 2: Identify classification workloads (bounded output space). These are your migration candidates.

Week 3: Set up a compiled classifier for your highest-volume classification task. Run it in shadow mode — call both the LLM and the classifier, log both results, compare.

Week 4: Once accuracy is validated, cut over. Keep the LLM as a fallback for low-confidence results.

This is the structure of Sparkient's three-stage pipeline: CEL rules handle deterministic logic, the compiled classifier targets under 100ms, and optional LLM escalation handles configured low-confidence cloud cases. The stage mix and escalation latency must be measured on representative traffic.

FAQ

Q: How long does it take to go from "I have an LLM call" to "I have a compiled classifier"? Sparkient automates example generation, model training, evaluation, and ONNX export. Training duration depends on the dataset and available compute; example generation, review, integration, and production evaluation add project-dependent time. Each completed training run uses 2,000 credits.

Q: What if my classification categories change? Retrain the model. Add the new categories to your decision type definition, trigger a new training run, and deploy the updated model. If your categories change daily, a compiled classifier isn't the right fit — stick with an LLM until your taxonomy stabilizes.

Q: Do I lose accuracy compared to GPT-4o? Across four public domains, compiled models achieve 0.886–0.951 macro F1 and 91–96% accuracy. The benchmarks compare against traditional ML baselines, not GPT-4o. Evaluate the live prompt, classifier, and any optional escalation path separately on the same held-out cases.

Q: What about fine-tuning GPT-4o-mini instead? Fine-tuning is a solid middle ground — you get better accuracy from a cheaper model. But you're still paying per-token and still dependent on the provider's API in production. Fine-tuning makes sense if you need the LLM's flexibility. A compiled classifier makes sense if you need speed, cost predictability, and independence from a third-party API.

Build AI features that scale with your users, not your bill. Start with the free tier — 5,000 credits, no credit card required.