Service 05 of 10 · Technology & Intelligence
Cloud set-up that keeps your systems and AI fast, reliable and affordable.
We set up and run the cloud servers behind your apps and AI, for business owners and IT heads. We keep speed, cost and time online within agreed targets.
A live dashboard for an example set-up called your-platform-prod, with illustrative values. Six servers with GPUs (the chips that run AI) are about 68 per cent busy, with two spares ready. Full response time is about 842 ms, against a 1.5 second target. It produces 18.4 thousand tokens (pieces of words) per second. Cost is $1.21 per thousand requests, against a $1.45 budget. Three regions show their share of traffic and how clean their power grid is.
02Cost and speedboth grow with every word
Where AI bills and delays surprise teams.
AI is billed and timed per token, a piece of a word. Most overspend is waste: growing prompts, unlimited retries and idle GPUs (the chips that run AI).
Anatomy of one request
A support answer grounded in your documents
- Cost per 1k of this request
- $22.20
- baseline
- Time to first token
- 0.62 s
- baseline
- Monthly · 1.5M req
- $33,300
- baseline
| Part | Tokens | Rate per 1M | Cost |
|---|---|---|---|
| System prompt & tools | 1,200 | $3.00 | $0.0036 |
| Retrieved context | 3,800 | $3.00 | $0.0114 |
| User input | 150 | $3.00 | $0.0005 |
| Output | 450 | $15.00 | $0.0068 |
| Per request | 5,600 | $0.0222 |
Time to first token is reading time: the model reads all the input first, so long prompts wait longer.
Tokens per second is writing speed: the answer streams out steadily, so long answers take longer.
Screens marked Illustrative are examples, not client results. Example prices per million tokens: $3 in, $0.30 cached in and $15 out. Illustrative
This is one costly request: an answer from your documents. Across all features, the example averages $1.21 per 1,000 requests.
Showing the request as shipped: 5,600 tokens, $22.20 per thousand of this request, 0.62 seconds to first token.
-
01
Growing prompts
Each turn resends the whole chat and document extracts. Turn 12 can carry 7 times the text of turn 1.
FixSummarise the chat, trim extracts and reuse the fixed instructions.
-
02
Unlimited retries
3 automatic retries on a slow provider can bill one answer up to 4 times, and slow it further.
FixLimit and space out retries, and switch to a backup provider.
-
03
Idle GPUs between peaks
GPUs sized for evening peaks run at a tenth of capacity overnight, at full cost.
FixAdd GPUs as the queue grows, switch them off when quiet and run bulk jobs at night.
03How it fits togetherapps → gateway → models → data
Our standard design grows with demand and fails safely.
Every AI request passes one gateway you own: a single link to all your models. It picks the model, caps spend and switches provider if one fails.
- 01
Your apps
Websites, apps, staff tools, AI agents and scheduled jobs all call one internal AI link.
- 02
Access control
Checks who is calling, logs every call (OIDC, JWT) and sets usage limits per customer.
- 03
AI gateway
Picks the model by cost, speed, quality and data type. Switches provider if one fails, reuses answers and caps each team’s use.
- 04
AI models
Pay-per-use models from several providers, plus open models on vLLM in your cloud. GPUs are added or removed as demand changes.
- 05
Testing new models
A new model answers copied traffic unseen, then a small live share. It stays only if quality, speed and cost hold.
- 06
Data and storage
A search index so AI can answer from your documents, plus storage for models, files and logs.
- 07
Checks across every part
Monitoring with OpenTelemetry. Keys locked in Vault or a cloud key store. Rules as code for access, networks and where data stays.
-
Fastest to start
Pay-per-use AI models
Choose when
- Use is low or uneven
- You need the strongest models today
- Nobody on the team runs GPUs
Watch forPrices at high volume, provider usage limits and how they treat your data.
-
Most control
Open models you host
Choose when
- Steady use makes your own GPUs cheaper
- Data must stay in your account or in India
- You need top speed or a custom-trained model
Watch forGPU supply, on-call cover and updates.
-
Our default
Both, behind one gateway
Choose when
- One AI link for every team
- Picks by cost, speed and data type
- Switches when a provider fails
How we run itIts rules are code, reviewed and tested like the rest.
Model swaps
Swap a model in days, without a rebuild.
Every request goes through the gateway, so a new or cheaper model is a settings change. We test it on your own examples first.
- Day 1New model added to the gateway · tested on your own examples
- Day 2Copied traffic · answers scored, never shown
- Day 35% of traffic · speed, cost and errors watched
- Day 5Goes live, or undone in one change
Technologies we work with
- AWS
- Microsoft Azure
04Failure drilla planned failure, in a safe window
Failure drills that test recovery before an outage does.
Pick a failure and add it to 10 minutes of realistic traffic. Run it with the safeguards off, then on. Compare speed, errors and the error budget.

Interactive failure drill. Choose one of three injected failures, turn the resilience patterns on or off, then press Play or move the timeline slider. Four charts show successful requests per second, p95 latency, error rate and the error budget left for the month; the architecture map shows which component failed and where traffic goes; the incident log lists what happened and when. All values are illustrative.
10:00/ 10:00
Successful requests req/s
39
p95 latency ms
888 ms
Error rate %
0.08%
Error budget left % of month
62.0%
Architecturegpu-node-3 ejected · replica added
Incident log8 events
- Drill started · 40 req/s at peak · p95 860 ms
- gpu-node-3 stops responding (injected)
- Health check fails 3 × 3 s → node ejected from the pool
- In-flight requests retried once on healthy nodes
- Queue depth 180 · backpressure on · p95 1.3 s
- Autoscaler: +1 replica from the warm pool
- Replica ready · weights loaded from local cache
- Recovered without a page · budget used 0.03%
- Peak client-visible errors
- 4.0%
- Peak p95 latency
- 1.38 s
- Error budget used
- 0.03% of the month
- Fast-burn page
- Not triggered
-
The target
99.9% online each month
That leaves 0.1% of requests free to fail: 43.2 minutes of full downtime in a 30-day month. This allowance is the error budget. Your team decides how to spend it.
-
The alert
An alert when the budget drains
The alert calls an engineer if the budget drains fast: 2% of the month’s budget in an hour. A slow drain opens a ticket.
-
The safeguards
What “on” switches on
- Checks that take failed servers out
- A gateway that switches to a backup provider
- A queue that holds excess requests, and limited retries
- Reused answers and spare servers on standby
- Automatic switch to a backup region
05Faster, cheaper AIthe same answers, on fewer GPUs
4 ways to run AI models faster and for less.
Each one runs the same model on less GPU time. It goes live only after a test on your own examples shows quality has held.
-
01
Continuous batching
2–4× the requests per GPU
A scheduler with six GPU lanes, each decoding a request at about 75 tokens a second, 450 tokens a second between them, and a queue of three waiting requests. As a lane finishes, the next queued request joins the batch at once; a fixed batch would hold the queue for another 5.8 seconds.
The GPU handles many requests at once. A new one joins when a slot frees, without waiting for the slowest.
Quality check quality 0.94 → 0.94 · reply time −38% · pass
-
02
Prompt caching
replies start 53% sooner
Two requests shown as rows of twelve token blocks. The first request computes every block. The second request shares the same seven-block prefix, which is served from the cache, and only computes the five new blocks.
The fixed instructions and examples repeat on every call. We keep them ready, so only the new text is read.
Quality check quality 0.94 → 0.94 · first word 0.62 s → 0.29 s · pass
-
03
Quantisation
50–75% less memory
8B model · one 24 GB GPU
- Weights
- 16 GB
- 4k-token seqs
- ~12
- Decode
- 1.0×
- Golden set
- 0.94
- Weights
- 8 GB
- 4k-token seqs
- ~28
- Decode
- 1.6×
- Golden set
- 0.93
- Weights
- 4.5 GB
- 4k-token seqs
- ~35
- Decode
- 2.1×
- Golden set
- 0.90
FP8 selected: 8 bits per weight, 8 gigabytes of weights on a 24 gigabyte GPU, room for about 28 concurrent 4,000-token sequences, 1.6 times decode speed, golden-set score 0.93, inside tolerance, ships.
The model is stored in fewer bits (FP8 or INT4), leaving room for more requests. The quality check measures any loss.
Quality check baseline · ships−0.01 · inside the 0.02 tolerance · ships−0.04 · below tolerance · held for review
-
04
Speculative decoding
45–60% less writing time
A draft model proposes four tokens at a time. The target model checks all four in one pass, keeps the accepted ones and adds one token of its own; the rest is discarded. The output line grows by the accepted tokens: twelve tokens take three target passes instead of twelve.
A small model guesses the next few words and the main model checks them in one step. Same answer, sooner.
Quality check same output by design · writing time −56% · pass
Engines we run models on
These engines update monthly. We load-test each one and run the quality check on a few servers before live traffic.
- vLLM PagedAttention · continuous batching · prefix caching
- NVIDIA TensorRT-LLM in-flight batching · FP8 · speculative decoding
- SGLang RadixAttention prefix cache · structured output
- Ray Serve multi-model serving · autoscaling replicas
- Hugging Face open-weight models · TGI
Gains are typical ranges for chat and answers from documents. They depend on the model, prompts and traffic. Illustrative
068 layersfrom GPUs at the bottom to security at the top
Your AI set-up, from GPUs to security.
Each layer has one job, one owner and its tools. Open one to see what it holds and what we decide with you.
U08 · tray open0 critical CVEs
Security & policy
Vault or the cloud key store gives each service short-lived passwords, so none is saved in code. We scan software before release, watch it while it runs and block anything unsafe or unsigned.
Technologies we work with here
- Vault
- Trivy
- Falco
- Snyk
- OPA Gatekeeper
- Kyverno
Decided with you at this layer
- Short-lived access instead of fixed keys
- Only signed software in production
- Models, prompts and logs treated as sensitive data
U07 · tray open14 SLO alerts
Monitoring & costs
OpenTelemetry tracks every request from gateway to GPU, with its size, reused answers and cost. Any slow or costly request points to its feature, team and model.
Technologies we work with here
- OpenTelemetry
- Prometheus
- Grafana
- Datadog
- PagerDuty
- OpenCost
Decided with you at this layer
- Standard AI tracking on every model call
- Alerts when the budget drains
- Daily spend per team and feature
U06 · tray open31% cache hits
AI gateway
Every AI request passes through it. It checks access, picks the model, switches on failure, reuses answers and caps team budgets.
Technologies we work with here
- Kong
- Cloudflare
- Redis
- LiteLLM
- Envoy AI Gateway
Decided with you at this layer
- Which data may leave the country
- Backup providers and time limits
- When saved answers can be reused
U05 · tray open12.4M vectors
Data & search
Good answers from your documents need fast, complete search. We size the index for speed and update it within minutes of a change.
Technologies we work with here
- Qdrant
- PostgreSQL
- Milvus
- Apache Kafka
- pgvector
Decided with you at this layer
- Search tuned to find the right passages
- Re-index only what changed
- Each customer’s data kept apart
U04 · tray open3 models live
AI models
Open models run on vLLM or TensorRT-LLM behind a standard API, so the gateway treats them like pay-per-use ones. Copies are added as the queue grows.
Technologies we work with here
- vLLM
- NVIDIA
- Hugging Face
- Ray
- SGLang
- Triton
Decided with you at this layer
- Which model and settings for each route
- Prompt caching and batch limits
- Scaling on the waiting queue
U03 · tray open6 node pools
Kubernetes & code
GPU servers are kept for AI work. Terraform builds the cloud, Helm packages the apps and Argo CD keeps each cluster matched to the code. Nothing live is changed by hand.
Technologies we work with here
- Kubernetes
- Helm
- Terraform
- Argo
- Docker
- Karpenter
- KEDA
Decided with you at this layer
- GPU servers with one spare ready
- Test systems switched off when unused
- Every change reviewed before it goes live
U02 · tray open3 regions · 2 in India
Cloud & regions
We build on the cloud you already use, in regions your data rules allow. A secure base sets log-ins, networks, logs and security rules once. A web firewall and caching sit in front.
Technologies we work with here
- AWS
- Microsoft Azure
- Google Cloud
- Cloudflare
Decided with you at this layer
- Primary and standby regions
- Reserved, pay-as-you-go and spot servers
- Private links to AI providers where offered
U01 · tray open8 × L40S · 68%
GPUs & AI chips
We pick the smallest chip that fits the model and your traffic. An 8B model at FP8 suits an L4 or L40S. 70B models and long inputs need an H100 or H200. Inferentia and TPU are tested on your traffic first.
Technologies we work with here
- NVIDIA
- L4 · 24 GB
- L40S · 48 GB
- H100 · 80 GB
- H200 · 141 GB
- AWS Inferentia2
- Google Cloud TPU
Decided with you at this layer
- GPU type for each model and input size
- Reserved base, pay-as-you-go peaks, spot for bulk
- GPUs kept 60–80% busy

Where data stays
Cloud regions we use in India
The gateway sets where data goes. Personal data cannot go to a model or region outside India. Bulk jobs on other data may run in cheaper, lower-carbon regions.
| Cloud | Region | Location | Code |
|---|---|---|---|
| AWS | Asia Pacific (Mumbai) | Maharashtra | ap-south-1 |
| AWS | Asia Pacific (Hyderabad) | Telangana | ap-south-2 |
| Microsoft Azure | Central India | Pune | centralindia |
| Microsoft Azure | South India | Chennai | southindia |
| Google Cloud | Mumbai | Maharashtra | asia-south1 |
| Google Cloud | Delhi | Delhi NCR | asia-south2 |
GPU supply varies by region and changes often. We check it before a design is final.
07Cloud costscost per request · per team · forecast · spikes
Cost per request, visible before the invoice.
Every AI request is tagged with its feature, team and route, so you see what each one costs. An AI agent checks costs daily. A named person approves any change it suggests.
- Spend to date
- $26.3k
- day 19 of 30
- Forecast
- $41.8k
- ± $2.9k at 80%
- Budget
- $45.0k
- 7% headroom
- Cost per 1k requests
- $1.21
- −14% vs August
- Unallocated
- 3%
- of spend without a tag
Unit cost
Cost per 1k requests against the cost model
| Name | Against target | Actual | Target | Trend |
|---|---|---|---|---|
| Support assistant412k requests | $1.42 | $1.60 | −12% | |
| Document search1.1M requests | $0.38 | $0.50 | −4% | |
| Sales email drafts58k requests | $2.94 | $2.20 | +31% | |
| Invoice extraction96k documents | $0.71 | $0.90 | −9% | |
| Internal copilot203k requests | $1.18 | $1.20 | +2% |
Spend forecast
$k · cumulative
Month-to-date spend of 26.3 thousand dollars on day 19, forecast to reach 41.8 thousand dollars by day 30, with an 80 percent range of 38.9 to 44.6 thousand, under a budget of 45 thousand.
GPUs in use against reserved capacity
last 7 days · 2 h buckets
Over seven days, serving used an average of 5.4 of 8 reserved GPUs, about 68 percent. 4 afternoon peaks rose above the reservation onto on-demand GPUs, and nightly batch jobs ran on 4 spot GPUs for 168 GPU-hours.
- 5.4 avgserving on reserved · 68% of the reservation
- 4 peaksburst to on-demand
- 168 GPU-hbatch on spot, about 65% cheaper
Anomaly · yesterday 01:00–03:10
+$1,240 vs forecast
Embedding job re-ran 3×
embed-docs-nightly timed out listing the object store and retried the whole job instead of the failed batch, three times over.
Proposed by finops-agent · confidence 0.92
jobs/embed-docs.yaml - retries: 3 + retries: 1 + retryScope: batch + idempotencyKey: "{{ .source.version }}"
Awaiting your approval · the agent cannot merge
- finops-agentAnomaly: embedding spend 3.1× forecast (z = 4.2)
- finops-agentCause: 3 full re-runs after an object-store timeout
- finops-agentFix proposed · awaiting human approval
- youApproved · PR #482 opened · change CHG-1182 logged
- youHeld for review · assigned to platform on-call
-
Cost reports per team
The gateway labels all AI use and GPU time by feature and team. Each team sees its costs weekly, before finance sees the total.
-
Reserved, pay-as-you-go and spot
Steady work runs on reserved servers and peaks on pay-as-you-go ones. Bulk jobs run on cheap spare capacity (spot) and save progress, so an interruption costs minutes.
-
Off when idle
Test and preview systems switch off outside working hours. The first request after that waits for start-up, so live systems never switch off.
Cost standards we work to
- FinOps FrameworkFinOps Foundation · inform, optimise, operate
- FOCUSFinOps Open Cost and Usage Specification · one bill format for every cloud
08Sustainabilitycarbon per request, measured and cut
AI carbon emissions, cut by scheduling.
The same job emits more or less carbon depending on when and where it runs. Live requests stay near users. Bulk work moves to cleaner hours and regions your data rules allow.
A chart of grid carbon intensity across one day for Mumbai and Chennai, both highest overnight and in the evening and lowest around midday, when solar output peaks; Chennai sits lower throughout. The embedding refresh is scheduled in the shaded four-hour window.
Inferencelatency-sensitive · stays near usersall day
Embedding refresh4 h · 22 kWh · drag the block or use the slider11:00–15:00
Backup snapshotfixed · keeps the RPO02:00 · 14:00
- Grid intensity, I
- 465g/kWh
- Job emissions
- 12.1kgCO₂e
- SCI per 1k documents
- 30.3gCO₂e
- −33% vs the 01:00 Mumbai cron
SCI = ((E × I) + M) per RE 22 kWh · I 465 g/kWh · M 1.9 kg · R 1k documents
Both regions are in India, so documents stay in the country. Carbon figures here are examples; live ones come from grid data and GPU energy readings. Illustrative
09Reliabilitytargets · error budgets · on-call
Reliability targets we set with you, and an error budget we track.
Time online is a target you choose and pay for. We agree it with you, and the error budget decides when to release features and when to fix things.
Availability
Successful responses ÷ all responses, measured at the gateway, excluding client 4xx.
- Objective
- 99.9%
- over 30 days
- Observed
- 99.96%
- this window
- Budget
- 43.2 min
- total for 30 days
- Remaining
- 25.8 min
- 59.7% of budget
An error-budget burn-down for Availability across thirty days, falling from the full budget to 59.7% remaining, plotted against a straight ideal-burn line, with deployment markers along the bottom and one step drop where an incident burned budget quickly.
One provider incident on day 12 burned 6.4 minutes. The multi-provider router carried the rest of the traffic, so the budget survived it.
p95 latency
Share of non-streaming requests completed end to end within the target, per endpoint class.
- Objective
- 99% < 1.5 s
- p95 observed 1.21 s
- Observed
- 99.38%
- this window
- Budget
- 345k req
- total for 30 days
- Remaining
- 131k req
- 37.9% of budget
An error-budget burn-down for p95 latency across thirty days, falling from the full budget to 37.9% remaining, plotted against a straight ideal-burn line, with deployment markers along the bottom and one step drop where an incident burned budget quickly.
A day-20 release raised p95 by 340 ms. Fast burn paged on-call inside the hour, the release was rolled back, and the budget is being repaid before the next feature ships.
Time to first token
Share of streaming responses that put their first token on the wire within the target.
- Objective
- 95% < 800 ms
- p95 observed 612 ms
- Observed
- 96.8%
- this window
- Budget
- 965k streams
- total for 30 days
- Remaining
- 623k streams
- 64.6% of budget
An error-budget burn-down for Time to first token across thirty days, falling from the full budget to 64.6% remaining, plotted against a straight ideal-burn line, with deployment markers along the bottom.
Prompt caching and continuous batching hold first token steady through the evening peak, so the budget burns close to the ideal line.
How alerts work
A fast burn (14.4× the safe rate for an hour) calls the on-call engineer. A slow burn (6× over 6 hours) opens a ticket for the next working day. Alerts track what users feel, not single servers.
The freeze rule
With under 25% of the budget left, releases stop and the next 1- or 2-week cycle goes on reliability. We agree this in advance, so nobody argues it during an outage.
Illustrative figures
The window, targets and limits are the ones we use. The plotted values are sample data for an example system. Illustrative
-
Fix guides for every alert
Written during the build, each guide gives the first 5 commands, the dashboards to open and how to undo.
-
A shared on-call rotation
A main and a backup engineer, a clear escalation path and handover notes at each shift. Your engineers can join the rotation.
-
No-blame reviews after outages
Within 5 working days: what happened, why, what monitoring missed and who fixes what, planned into the next cycle.
-
Tested disaster recovery
Recovery time and data-loss limits are set per system. Each quarter, a timed rebuild from code and backups tests them.

Standards we build to
The security, continuity and server standards we build to. Certifying your own systems is a separate project we can prepare you for.
- ISO/IEC 27001:2022Information security management systems
- SOC 2Trust Services Criteria
- CIS Controls v8.1Critical Security Controls and Benchmarks
- NIST CSF 2.0Cybersecurity Framework
- ISO 22301:2019Business continuity management systems
Support hours are agreed per project (business, extended or 24×7) and written into the fix guides with who to call.
10How it worksmeasure · design · build · run
Measure first, then cut delays, downtime and cost.
We measure speed, downtime, spend and carbon first, so you can check every change against them. We fix what the numbers point to, not what looks old. Weeks shown are typical.
-
01
Measure
Wk 01–02
We map how things run today and measure traffic, speed, outages, spend and carbon. Then we agree targets with you.
- Today’s numbers
- Agreed targets
- Cost breakdown
-
02
Design
Wk 02–04
We design the new set-up, the capacity you need and how to move across. We write down each decision and why.
- New set-up design
- Capacity plan
- Written decisions
-
03
Build
Wk 04–10
We build the set-up as code and test it under heavy load. Monitoring and cost tracking start on day one.
- Set-up as code
- Load test results
- Dashboards
-
04
Run
Wk 10–12
We switch traffic over and write the fix guides. A monthly review of targets, spend and capacity sets the next fixes.
- Fix guides
- Monthly reviews
- Cost report
The 3 numbers a phase is judged on
Measured on your traffic before the first change and after each release. Bars show the cut from the starting figure. One example project shown. Illustrative
-
Chat reply time (p95)
2.9 s 1.21 s
−58%
Continuous batching, prompt caching and sending short requests to a smaller model.
-
Cost per 1,000 requests
$2.95 $1.21
−59%
Servers matched to demand, quantised models and caching, with bulk work on reserved capacity.
-
Idle GPU hours a week, test systems
96 h 12 h
−88%
Test and preview systems switch off outside working hours.
After launch, reviews continue. Each month we review the error budget, outages and costs per request. Before busy seasons, we check capacity.
11What you getyour set-up as code, in your accounts
Everything we build is saved as code in your accounts.
What we make for you is yours once it is paid for. Tools we already had stay ours, and you get a free, permanent licence to use them.
A browsable listing of the infrastructure repository handed over at the end of the engagement, with 9 files you can open: Terraform for the GPU pool and the residency policy, Helm values for model serving and the AI gateway, a Grafana cost dashboard, the SLO objectives with their burn-rate alert rules, the GPU node-failure runbook, a k6 peak-load profile and the disaster-recovery drill script.
Arrow keys move, → opens a directory, ← closes it, Enter shows a file.
terraform/envs/prod/main.tf
01module "gpu_pool" {02 source = "../../modules/gpu-pool"03 04 region = "ap-south-1" # Mumbai, primary05 instance_type = "g6e.2xlarge" # 1 × L40S, 48 GB06 min_nodes = 607 max_nodes = 10 # 6 serving + 2 warm + 2 burst08 warm_nodes = 209 taints = ["nvidia.com/gpu=present:NoSchedule"]10 tags = local.cost_tags # feature, team, model route11}
11 linesPR #482 · platform · plan attached, 26 policy checks passed
terraform/policy/region.rego
01package terraform.region02 03# Personal data never leaves India. Any resource planned04# outside the approved regions fails the policy check in CI.05approved := {"ap-south-1", "ap-south-2"}06 07deny[msg] {08 r := input.resource_changes[_]09 r.change.after.region10 not approved[r.change.after.region]11 msg := sprintf("%s targets %s, outside the residency set",12 [r.address, r.change.after.region])13}
13 linesPR #455 · security · ap-south-2 added to the residency set
helm/vllm-serving/values.yaml
01model:02 name: your-platform/chat-8b03 quantization: fp8 # eval gate: −0.01, inside the 0.02 tolerance04 maxModelLen: 819205 gpuMemoryUtilization: 0.9206 enablePrefixCaching: true07 08autoscaling:09 metric: vllm:num_requests_waiting # queue depth, never CPU10 target: "4"11 minReplicas: 612 maxReplicas: 1013 14podDisruptionBudget:15 minAvailable: 5
15 linesPR #501 · serving · FP8 after the eval gate came back −0.01
helm/ai-gateway/values.yaml
01routes:02 - match: { dataClass: personal }03 provider: self-hosted # residency rule, enforced here04 - match: { tokensIn: "<800" }05 provider: small-model06 - default: provider-a07 08retries:09 budget: 10% # of a route's requests per minute10 backoff: exponential-jitter11 idempotencyKey: required12 13cache:14 prompt: { ttl: 1h }15 semantic: { threshold: 0.93, ttl: 15m }
15 linesPR #498 · platform · retry budget capped at 10% of the route
dashboards/finops.json
01{02 "title": "Cost per 1,000 requests",03 "datasource": "prometheus",04 "targets": [05 {06 "expr": "sum by (feature) (rate(gen_ai_cost_usd_total[1h]))07 / sum by (feature) (rate(gen_ai_requests_total[1h])) * 1000",08 "legendFormat": "{{feature}}"09 }10 ],11 "thresholds": [{ "value": 1.43, "colorMode": "text" }],12 "description": "Cost tags are set at the gateway. Spend without a13 tag shows up as {feature=\"\"} and gets chased."14}
14 linesPR #471 · finops · cost tag added to the gateway span
slo/objectives.yaml
01- name: availability02 sli: successful_responses / all_responses # at the gateway, 4xx excluded03 objective: 99.904 window: 30d # 43.2 minutes of budget05 burnRateAlerts:06 - { factor: 14.4, long: 1h, short: 5m, severity: page }07 - { factor: 6, long: 6h, short: 30m, severity: ticket }08 freezeBelowBudget: 25 # % remaining — feature releases stop09 10- name: time_to_first_token11 sli: streams_first_token_under_800ms / streams12 objective: 9513 window: 30d
13 linesPR #440 · SRE · burn-rate pair agreed with you, then merged
runbooks/gpu-node-failure.md
01# GPU node failure02 03Alert: vllm_node_unhealthy · Dashboard: SLO / serving04Owner: platform on-call · Expected impact: none, the warm pool absorbs it05 06## First five commands07 081. kubectl get nodes -l nvidia.com/gpu=present -o wide092. kubectl describe node $NODE | sed -n '/Conditions/,/Events/p'103. kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data114. kubectl -n serving get hpa vllm-serving125. k6 run loadtest/k6/smoke.js13 14## If p95 stays above 1.5 s for ten minutes15 16Shift the chat route to provider A in the gateway, page the model17owner, and open an incident. The drill for this is in section 04.
17 linesPR #466 · on-call · the five commands verified in a drill
loadtest/k6/peak.js
01import http from 'k6/http';02import { check } from 'k6';03 04export const options = {05 scenarios: {06 peak: { executor: 'constant-arrival-rate', rate: 40,07 timeUnit: '1s', duration: '20m', preAllocatedVUs: 300 },08 },09 thresholds: {10 'http_req_duration{endpoint:chat}': ['p(95)<1500', 'p(99)<3000'],11 'http_req_failed': ['rate<0.01'],12 },13};14 15export default function () {16 const res = http.post(`${__ENV.BASE}/v1/chat`, PAYLOAD,17 { tags: { endpoint: 'chat' } });18 check(res, { 'first token under 800 ms': (r) => r.timings.waiting < 800 });19}
19 linesPR #489 · quality · peak profile raised to 40 req/s
dr/restore-drill.sh
01#!/usr/bin/env bash02set -euo pipefail03 04# Quarterly drill: rebuild production in the standby region from05# code and backups only, and time it against the agreed RTO.06START=$(date +%s)07 08terraform -chdir=terraform/envs/dr apply -auto-approve09helm upgrade --install platform helm/platform -f helm/platform/dr.yaml10./dr/restore-vectors.sh --snapshot "$(date -u +%F)" # RPO: hourly11k6 run loadtest/k6/smoke.js12 13echo "RTO $(( $(date +%s) - START ))s against a 3600s target."14echo "Evidence written to dr/evidence/$(date -u +%F)/."
14 linesPR #430 · platform · last quarter's RTO evidence attached
Open any file to see what is inside. Paths and values are examples. You get the structure and the maths behind the limits. Illustrative
Handover pack
- Server set-up as codeTerraform · Helm
- Diagrams and written decisionsDiagrams · records
- AI model hosting set-upContainers · config
- Automatic test and releasePipelines · Argo CD
- Targets, dashboards & alertsGrafana · alert rules
- Load and outage test reportsk6 · report
- Cost and carbon report per teamDashboard · spreadsheet
- Change goes through reviewEvery change is reviewed, with a plan, an automatic rules check and a named approver. Nothing goes live from a laptop.
- Your accounts, your keysCloud accounts, code stores, password stores and AI keys are yours from day one. We work inside them with limited, logged access.
- Decisions are written downShort records explain what we chose, what we rejected and what would change our mind.
Built and run with
- Terraform
- Helm
- Kubernetes
- Argo
- GitHub Actions
- Vault
- Grafana
- k6
12Outcomeschecked monthly, the same way each time
5 numbers, measured every month.
We agree these targets and how to measure them at the start. Each monthly review shows every reading with its evidence, misses included.
-
Chat reply time (p95)
99 per cent of chat replies inside the agreed time, measured for each type of request.
1.21 s was 1.34 s Inside the objective
0 s agreed band 0–1.5 s 3 s
Set for each type of request and held under load tests at forecast peak, not guessed from average traffic.
-
Cost per 1,000 requests
Within ten per cent either side of the cost model agreed at forecast volume.
$1.21 was $1.41 On the cost model
$0.00 agreed band $1.17–$1.43 $2.40
Tracked per feature and per team, so a price rise or a prompt that grew shows up as a line, not a surprise invoice.
-
GPU utilisation at peak
Between 55 and 75 per cent at the daily peak — busy enough to be economic, with headroom left.
81% was 68% Above the band
0% agreed band 55–75% 100%
Below the band you pay for idle GPUs. Above it, queues build and the slowest replies get much slower.
Why it missed, and what happens next. A launch in week 3 tripled one feature’s traffic. More GPUs are being added and scaling now starts sooner. The owner and date are in the review log.
-
Carbon per 1,000 requests
Lower each quarter against a ceiling agreed at the start of it, measured the same way each time.
26.4 g was 29.8 g Trending down
0 g agreed band 0–32 g 60 g
Carbon falls with the same levers as cost: batching, caching, quantisation, and batch work scheduled into cleaner hours.
-
Error budget spent
No month where the error budget runs out without a decision behind it.
40.3% was 52% Inside the allowance
0% agreed band 0–60% 100%
A planned burn during a migration is a decision. An unplanned one is a design or a monitoring gap, and it gets a review.
-
01
Speed on target
Each feature gets a speed target, even for its slowest answers, tested under peak traffic.
-
02
Reliability you can quote
Agreed targets and an error budget. When it runs low, fixes come before features.
-
03
Spend each team can see
Each team sees the cost per request, user and AI model it controls, so it can cut it.
Ranges and readings are sample data, including the miss. Your targets come from your own starting numbers in the first phase. Illustrative
Offers & plans
Pick the cloud services you need, alone or combined.
We cover cloud set-up, app hosting, AI models, reliability and cost. Each offer says who it suits.
- Categories
- 04
- Offers
- 16
- Plans
- 04
How to buy
- 01Pick offers. Ask about one, or add several to one enquiry.
- 02Choose a plan. A small job, a fixed project or ongoing help.
- 03Send your enquiry. We reply within one working day.
How we work with you
Ways to start, from a question to a quote.
Ask a quick question, send your project details or request a quote. The person who will lead the work reads each one in full. Services in your enquiry go with it.
Or book a 30-minute call-
01
For a first chat, a press request or anything not ready to plan yet.
You get A reply from a named lead
-
02Recommended
Goals, who it is for, budget and timing, so our first reply can outline the work.
You get Options and prices after one call
-
03
Your documents, deadlines and the purchasing and security rules to meet.
You get Confirmation and a named lead
How it is priced
Each plan shows how it is priced. Work starts once we agree a written scope and price.
Opens the project form with this plan chosen.
-
In your enquiry
- Typical length
- 1–3 weeks
- Pricing
- Fixed fee
-
In your enquiry
- Typical length
- 3–9 months
- Pricing
- Fixed price per stage
-
In your enquiry
- Typical length
- Ongoing · 6-month minimum
- Pricing
- Monthly fee
-
In your enquiry
- Typical length
- 6–18 months
- Pricing
- Priced per written scope
Compare what each plan includes
| Plan | What you get | Best for |
|---|---|---|
| SprintA short, fixed job that answers one clear question. |
|
An audit, a brand check or a test version of an app |
| MilestoneA bigger build in stages you approve and pay for one at a time. |
|
Work too big for one contract, such as a large app |
| RetainerA monthly team that runs and improves what we built. |
|
Monthly social media and ads, or care for live sites and apps |
| EnterpriseA large, long project with its own team and agreed response times. |
|
Large companies making changes across markets, brands or business units |
13Questionsanswered plainly
Cloud set-up, your questions answered.
Short answers on hosting, reliability, cloud choice, capacity, cost and carbon.
-
01 Host our own AI models, or pay per use?
Pay per use is fastest to start and cheapest at low volume. Hosting your own open models pays off with steady, high volume, strict data-location rules or tight speed needs. We work out the break-even on your traffic first.
-
02 What is a reliability target and error budget?
A target such as 99.9% of requests answered within 0.8 seconds over 30 days. The error budget is the 0.1% left over: about 43 minutes of downtime in a 30-day month. When it is spent, fixes come before new features.
-
03 Which cloud do you recommend?
The one that fits your contracts, data rules, skills and AI needs. We work with AWS, Azure and Google Cloud. Where it is worth the cost, we build so you can switch later.
-
04 How do you reduce GPU costs?
We match servers to demand and do less work per request. We batch and cache requests, shrink models and send simple requests to smaller models. Where the work allows, idle servers switch off and jobs run on cheaper capacity.
-
05 Do you measure carbon?
Yes, per request, with the Software Carbon Intensity (SCI) method. The fixes that cut cost also cut carbon.
-
06 Do you work out how many servers we need?
Yes, from your own traffic. We plan the Kubernetes cluster, server groups for each kind of work and GPU sharing. The plan shows what happens at peak and what spare servers cost.
Tell us what you need built.
You will talk to the lead who would run your work.
Three ways to start
We always agree a written scope and price before work begins.
Choose one of the three ways above



