Capability 05 of 10 · Fast, reliable, affordable at scale
AI that answers in milliseconds, stays up, and costs what you planned.
AI Infrastructure & Cloud builds the backend that makes AI fast, reliable and cheap to run at scale: inference platforms, cloud foundations, data and model pipelines, and the SRE and FinOps practice that keeps latency and spend in check.
A live telemetry dashboard for an example cluster called your-platform-prod across three regions, with illustrative values: GPU utilisation for eight nodes averaging 72 percent with one idle warm-pool node, p95 latency of about 842 ms against a 1.5 second objective, output throughput of 18.4 thousand tokens per second, a cost of $1.84 per thousand requests against a $2.10 budget, and three regions with their share of traffic and grid carbon intensity.
02Token economicswhere latency and spend come from
Where AI bills and latency surprise teams.
A model call is priced by the token and timed by the token. Most overruns come from tokens nobody decided to send: prompts that grew, retries nobody capped and capacity nobody turned down.
Anatomy of one request
A support answer grounded in your documents
- Cost per 1k requests
- $22.20
- baseline
- Time to first token
- 0.62 s
- baseline
- Monthly at 1.5M requests
- $33,300
- baseline
| Part | Tokens | Rate per 1M | Cost |
|---|---|---|---|
| System prompt & tools | 1,200 | $3.00 | $0.0036 |
| Retrieved context | 3,800 | $3.00 | $0.0114 |
| User input | 150 | $3.00 | $0.0005 |
| Output | 450 | $15.00 | $0.0068 |
| Per request | 5,600 | $0.0222 |
Time to first token is prefill: the model reads every input token before it can say anything, so long prompts wait longer.
Tokens per second is decode: the answer streams at a steady rate, so long answers take longer to finish.
Illustrative list prices: $3 per million input tokens, $0.30 per million cached input tokens, $15 per million output tokens. Real rates vary by model, provider and commitment. Illustrative
Showing the request as shipped: 5,600 tokens, $22.20 per thousand requests, 0.62 seconds to first token.
-
01
Context grows silently
Chat history and retrieved passages are resent on every turn. By turn 12 a request can carry seven times the input tokens of turn 1, and the bill follows.
FixSummarise history, cap retrieval, cache the stable prefix.
-
02
Retries multiply spend
A slow provider plus three blind retries can bill the same answer up to four times, and adds load exactly when capacity is short.
FixRetry budgets, jittered backoff, idempotency keys and a fallback route.
-
03
GPUs sit idle between peaks
Self-hosted capacity sized for the evening peak can average about a third utilisation across the day, and idle GPU-hours cost the same as busy ones.
FixAutoscale on queue depth, scale to zero off-peak, move batch work into the troughs.
03Reference architectureclients → gateway → router → serving → data
A reference architecture that scales and fails safely.
Every model call goes through one gateway you own. That single seam is what makes AI observable, affordable and replaceable: routing, fallback, caching, budgets and evals live there, not scattered through product code.
- 01
Clients
Web and mobile apps, internal tools, and agents or scheduled jobs call one internal AI API.
- 02
API gateway
Authenticates every call (OIDC, JWT), enforces rate limits and quotas per tenant, and logs requests.
- 03
AI gateway and router
Routes each request by cost, latency, quality and data class; opens a circuit breaker and falls back when a provider degrades; caches prompts and answers; enforces token budgets per team.
- 04
Model serving
Managed model APIs from more than one provider, plus self-hosted open-weight models on vLLM in Kubernetes GPU node pools that autoscale on queue depth.
- 05
Evals and canary
A candidate model takes shadow traffic, then a small canary share, and is promoted only when eval scores, p95 latency and cost hold.
- 06
Data and storage
A vector database for retrieval, a feature store, and object storage for model weights, documents and logs.
- 07
Rails across every tier
OpenTelemetry traces, metrics and logs; secrets in Vault or a cloud KMS with short-lived credentials; policy as code for IAM, network and data residency.
-
Fastest to start
Managed model APIs
Choose when
- Volume is modest or spiky
- You need frontier-model quality today
- Nobody on the team runs GPUs
WatchPer-token cost at scale, provider rate limits, data-processing terms.
-
Most control
Self-hosted open weights
Choose when
- Steady volume makes GPU-hours cheaper than tokens
- Data must stay in your account or in India
- You need tight latency or a fine-tuned model
WatchGPU capacity, on-call, model and driver upgrades.
-
Our default
Both, behind one gateway
Choose when
- One API for every product team
- Routes by cost, latency and data class
- Fallback when a provider degrades
Governed asRouter configuration is code, reviewed and tested like any other.
AI-native pace
Swap models in days, not quarters.
Because every call goes through the gateway, a new or cheaper model is a configuration change. It takes shadow traffic, then a 5% canary, and is promoted only when evals, p95 latency and cost hold. Infrastructure changes ship weekly through reviewed Terraform and Helm pipelines.
- Day 1Candidate added to router config · offline evals on the golden set
- Day 2Shadow traffic · responses scored, never shown
- Day 35% canary · p95, cost and error budget watched
- Day 5Promoted, or rolled back in one commit
Technologies we work with
- AWS
- Microsoft Azure
04Failure drillchaos engineering, before your customers do it for you
We break it on purpose, so it holds when it matters.
Pick a failure and inject it into a ten-minute window of production-like traffic. Run it with the resilience patterns off, then on, and watch latency, errors and the error budget.
Interactive failure drill. Choose one of three injected failures, turn the resilience patterns on or off, then press Play or move the timeline slider. Four charts show successful requests per second, p95 latency, error rate and the error budget left for the month; the architecture map shows which component failed and where traffic goes; the incident log lists what happened and when. All values are illustrative.
10:00/ 10:00
Successful requests req/s
235
p95 latency ms
888 ms
Error rate %
0.08%
Error budget left % of month
62.0%
Architecturegpu-node-3 ejected · replica added
Incident log8 events
- Drill started · 240 req/s · p95 860 ms
- gpu-node-3 stops responding (injected)
- Health check fails 3 × 3 s → node ejected from the pool
- In-flight requests retried once on healthy nodes
- Queue depth 180 · backpressure on · p95 1.3 s
- Autoscaler: +1 replica from the warm pool
- Replica ready · weights loaded from local cache
- Recovered without a page · budget used 0.03%
- Peak client-visible errors
- 4.0%
- Peak p95 latency
- 1.38 s
- Error budget used
- 0.03% of the month
- Fast-burn page
- Not triggered
-
The objective
99.9% monthly availability
Leaves 0.1% of requests to fail: the equivalent of 43.2 minutes of full downtime in a 30-day month. That allowance is the error budget, and spending it is a decision, not an accident.
-
The page
Burn-rate alerts, not noise
On-call is paged when the budget burns at 14.4× the sustainable rate over both 1 hour and 5 minutes, which spends 2% of the month in an hour. Slower burns (6× over 6 hours) page too; gentle ones open a ticket.
-
The patterns
What “on” switches on
- Health checks that eject failed nodes
- A router with circuit breakers and fallback providers
- A request queue with backpressure and capped, jittered retries
- A semantic cache and an autoscaler with a warm pool
- Automated failover to a warm standby region
Let’s build what happens next.
Tell us what you’re building. We’ll answer straight.