How to Run ML Classifiers Offline Without a Cloud API

TL;DR

Not every application can call a cloud API. Air-gapped environments, IoT devices, mobile apps, and regions with unreliable connectivity all need local inference. Sparkient's edge bundle packages a compiled ONNX model with CEL rules into a self-contained ZIP; install sparkient-edge, load the bundle, and benchmark local decisions without network calls.

The Problem: Cloud APIs Aren't Always an Option

Cloud-based ML is the default, but it assumes something that isn't always true: a reliable, low-latency network connection.

Real scenarios where cloud APIs don't work:

• Air-gapped environments. Defence, healthcare, and financial systems that can't make outbound API calls for security reasons.
• IoT and embedded devices. Industrial sensors, point-of-sale terminals, and smart cameras operating on intermittent cellular connections.
• Mobile apps in the field. Field service tools, agricultural monitoring, and emergency response apps used in areas with no signal.
• Privacy or deployment requirements. A project's legal assessment, contract, threat model, or internal policy may require local processing.
• Latency-sensitive hot paths. Local execution can remove network variance when the cloud path does not meet the measured budget.

The traditional answer is "deploy the model yourself." But that means managing model serving infrastructure, handling updates, dealing with dependency conflicts, and building the inference pipeline from scratch.

Option 1: Build It Yourself with ONNX Runtime

ONNX Runtime is the standard for portable ML inference. If you have an ONNX model, you can run it anywhere Python (or C++, C#, Java, JavaScript) runs.

import onnxruntime as ort
import numpy as np

# Load the model
session = ort.InferenceSession("model.onnx")

# Prepare input (you handle all preprocessing)
input_data = np.array([[0.5, 1.2, -0.3, 0.8]], dtype=np.float32)

# Run inference
result = session.run(None, {"input": input_data})
predictions = result[0]

Pros: Full control, no dependencies beyond onnxruntime, works on ARM and x86.

Cons: You handle everything yourself — text tokenization, feature engineering, preprocessing, label mapping, confidence calibration, and the rules layer. For a text classification model using a semantic text encoder, that means bundling the tokenizer, managing the embedding step, and stitching together the full pipeline.

Option 2: TensorFlow Lite

TFLite is optimized for mobile and embedded devices. It supports quantization out of the box and has strong Android/iOS integration.

import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(input_details[0]["index"], input_data)
interpreter.invoke()
output = interpreter.get_tensor(output_details[0]["index"])

Pros: Excellent mobile support, GPU delegate on Android, small binary size.

Cons: TensorFlow-only ecosystem. If your model was trained in PyTorch or scikit-learn, you need a conversion step that may lose fidelity. Limited server-side support.

Option 3: Core ML (Apple Only)

If you're building exclusively for Apple platforms, Core ML gives you hardware-accelerated inference on the Neural Engine.

let model = try MyClassifier(configuration: MLModelConfiguration())
let prediction = try model.prediction(text: "Check this content")

Pros: Tight Xcode integration and access to Apple hardware acceleration; benchmark the converted model on each target device.

Cons: Apple-only. No cross-platform support. You're locked into the Apple ecosystem.

Option 4: Sparkient Edge

Sparkient Edge packages CEL rules, text assets, and the classifier into a ZIP bundle. The target still needs a compatible Python version and wheels for the package dependencies, so verify the complete runtime on every operating system and architecture you plan to support.

pip install sparkient-edge

Exporting an Edge Bundle

First, train a model through the Sparkient cloud API. Then export it:

import httpx

response = httpx.get(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/export",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)

with open("moderation.zip", "wb") as f:
    f.write(response.content)

The bundle contains everything needed for offline inference:

• The ONNX model (quantized to INT8 for small size)
• The text tokenizer files
• CEL rule definitions
• Input schema and label mappings
• Bundle metadata (version, training date, metrics)

Running Predictions Locally

from sparkient_edge import EdgePredictor

# Load the bundle — one-time initialization
predictor = EdgePredictor.from_bundle("moderation.zip")

# Make decisions — no network calls
result = predictor.predict({
    "text": "Free money! Click here now!!!",
    "account_age_days": 1,
    "previous_violations": 0
})

print(result)
# EdgeDecision includes decision, confidence, stage, reason codes, and class probabilities.

That's it. No API keys or network configuration are needed at runtime. The EdgePredictor loads the ONNX model into memory once; measure latency, memory, and throughput on the target hardware.

Bundle Info and Version Management

from pathlib import Path
from sparkient_edge import load_bundle

bundle = load_bundle(Path("moderation.zip").read_bytes())
meta = bundle["metadata"]
print(f"Decision type: {meta.get('decision_type_name', 'unknown')}")
print(f"Options: {bundle['options']}")
print(f"Trained: {meta.get('trained_at', 'unknown')}")
print(f"Rules: {len(bundle['rules'])} CEL expressions")

When you retrain in the cloud, export a new bundle and swap the file. The predictor API stays the same.

Using Edge with MCP (Local Mode)

The edge package also includes a local MCP server for AI coding assistants:

pip install "sparkient-edge[mcp]"
python -m sparkient_edge

This starts a stdio-based MCP server that your IDE can connect to:

{
  "mcpServers": {
    "sparkient-local": {
      "command": "python",
      "args": ["-m", "sparkient_edge"]
    }
  }
}

After connecting, call load_edge_bundle with the absolute ZIP path. The assistant can then call make_decision locally without cloud connectivity.

Comparison: Edge Inference Options

| Feature | ONNX Runtime (DIY) | TFLite | Core ML | Sparkient Edge | |---------|-------------------|--------|---------|---------------| | Cross-platform | ✅ | ✅ | ❌ Apple only | ✅ | | Text pipeline included | ❌ | ❌ | ❌ | ✅ | | Rules engine | ❌ | ❌ | ❌ | ✅ CEL | | Tokenizer bundled | ❌ | ❌ | ❌ | ✅ | | Quantization | Manual | Built-in | Built-in | Built-in (INT8) | | Latency | Benchmark complete pipeline | Benchmark converted model | Benchmark converted model | Benchmark complete bundle on target hardware | | Setup effort | Own preprocessing and serving | Own conversion and app integration | Apple-specific conversion and app integration | Exported bundle plus local packaging and operations |

Raw ONNX Runtime inference is only one part of the path. Benchmark validation, tokenisation, preprocessing, inference, process boundaries, and concurrency together on the target hardware.

The tradeoff depends on the model and deployment. DIY runtimes maximise control; Sparkient Edge packages the trained decision assets, but you still own target-hardware validation, distribution, updates, and monitoring.

Deployment Patterns

Pattern 1: Air-Gapped Server

# Deploy the bundle to the secure environment via approved transfer
# No outbound network access needed at runtime

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("/secure/models/moderation.zip")

# Run in your application's hot path
for message in message_queue:
    result = predictor.predict({"text": message.content})
    if result.decision == "reject":
        quarantine(message)

Pattern 2: IoT Device with Periodic Sync

import os
from sparkient_edge import EdgePredictor

MODEL_PATH = "/opt/models/current.zip"

# Load whichever bundle is currently deployed
predictor = EdgePredictor.from_bundle(MODEL_PATH)

def on_sensor_event(event_data):
    result = predictor.predict(event_data)
    if result.decision == "alert":
        store_for_sync(result)  # Upload when connectivity returns

# Periodic model update (when connected)
def sync_model():
    if network_available():
        download_latest_bundle(MODEL_PATH)
        global predictor
        predictor = EdgePredictor.from_bundle(MODEL_PATH)

Pattern 3: Embedded in a FastAPI Service

from fastapi import FastAPI
from sparkient_edge import EdgePredictor

app = FastAPI()
predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/moderate")
async def moderate(content: dict):
    result = predictor.predict(content)
    return {
        "decision": result.decision,
        "confidence": result.confidence,
        "stage": result.stage
    }

This gives you a self-contained moderation service with no Sparkient cloud dependency after the bundle and Python dependencies are installed.

FAQ

How large are edge bundles? Bundle size depends on the text model, classifier, tokenizer, rules, and metadata. Inspect the exported ZIP and include transfer, storage, and memory in the target-device test.

Can I run edge bundles on ARM devices? Potentially, if compatible wheels exist for the target Python version, operating system, and architecture. Verify the complete sparkient-edge dependency set and benchmark the actual device before committing to it.

How do I update models in air-gapped environments? Export a new bundle from the Sparkient cloud, transfer it through the approved process, and restart the predictor with the new bundle path. The ZIP contains the model assets, but the Python package and compatible runtime dependencies must already be installed or transferred and managed separately.

What's the minimum Python version? The current package declares Python 3.10 or later. Dependency wheel availability still varies by platform.

Cloud APIs are the easy default, but they are not the only option. An exported ONNX bundle can support local or offline decisions when the target environment meets the package requirements and the measured quality, latency, memory, and throughput thresholds.

Use the trial to evaluate one model, then test an exported edge bundle on the exact target hardware before planning a deployment.