Architecture

Latency is a design decision, not an accident.

Substrate is fast because every layer, from the silicon up, is built for one workload: serving models. Here is how a request becomes a response in under fifteen milliseconds.

The request path

Four steps, four milliseconds of overhead.

Step 01 — Route

Land in the nearest region

An anycast edge accepts the request and routes it to the closest healthy region. No cold start, no DNS round-trip on the hot path.

Step 02 — Batch

Join a continuous batch

The request slots into an in-flight batch without waiting for it to fill. Throughput rises without adding to any single request's latency.

Step 03 — Execute

Run on compiled silicon

The model, compiled ahead of time to Substrate's accelerator, executes as a single fused kernel. No interpreter, no per-op dispatch.

Step 04 — Stream

Return the first token in ~11ms

Tokens stream back as they're produced. The meter starts and stops on tokens served. Nothing in between.

Purpose-built silicon

General-purpose hardware solves a harder problem than yours.

A GPU is a marvel of flexibility, and you pay for all of it. Substrate's accelerator does one thing: run inference. That focus is where the latency and the cost savings come from.

Substrate's custom inference accelerator package.

Specifications

The numbers, without the asterisks.

Performance

p99 latency	< 15 ms at sustained load (11 ms typical, first token)
Throughput	Up to 120k requests/sec per region, elastic
Cold start	None on the hot path, models kept resident
Batching	Continuous, request-level; no fixed batch window

Platform

Regions	12, anycast-routed, automatic failover
Models	Open-weight, fine-tuned, and custom; hot-swappable
Interfaces	REST, gRPC, and OpenAI-compatible endpoints
Billing	Per token served; scale-to-zero between requests

Security & compliance

Certifications	SOC 2 Type II; HIPAA available
Isolation	Per-tenant compute and network isolation
Data	Prompts and weights never leave your boundary; zero retention by default
Uptime	99.99% SLA, measured and published

stream.py

from substrate import Client

client = Client(api_key="sk-...")

# same call, routed to the nearest region
for token in client.stream(
    model="llama-3.1-70b",
    input=prompt,
):
    print(token, end="")

# first token in ~11ms · billed per token

Developer experience

Boring on purpose.

The best infrastructure is the kind you forget is there. Substrate's SDK is a thin wrapper over a stable HTTP API, drop-in compatible with the OpenAI client, so most teams migrate by changing one base URL.

Get an API key

Read the full architecture.

Request access and we'll share the deep-dive: the compiler, the batching scheduler, and the silicon.

Request access