The Four Cloud Decisions That Decide Whether Your AI Workload Survives Production

Almost every AI program we review has a competent model strategy and an incompetent infrastructure strategy. The pilot works on a notebook against a copy of production data, then it moves to a shared cluster, then latency triples, then the cloud bill arrives, then someone asks whether the vector store is even in the same region as the source of truth. The interesting part is that none of these failures are AI failures. They are cloud architecture decisions — usually four of them — taken quietly by whoever stood up the first environment. Gartner projects AI-optimized IaaS spending will grow 96% in 2026 to roughly $42 billion, which means these decisions are about to get expensive at scale rather than expensive in a sandbox.

Decision one: where the data lives, not where the model runs

Model endpoints are portable. Data is not. Once you have tens of terabytes of transactional, document and telemetry data anchored in one provider's storage layer, the gravity of that data determines the economics of everything you build on top of it. Retrieval-heavy AI amplifies this: a RAG pipeline doesn't read data once for training, it reads slices of it on every request, forever.

The practical test is simple. Trace a single production inference call end to end and count the region and provider boundaries it crosses. If the answer is more than zero, you have a recurring tax — in milliseconds and in egress charges — baked into every future user of that capability. Analysis published by Sedai puts egress at a material share of total cloud spend for many organizations, and retrieval traffic is precisely the pattern that turns an occasional charge into a fixed operating cost.

Decision two: the shape of your compute commitment

Training and inference have opposite consumption curves, and treating them as one procurement line is the most common financial mistake in enterprise AI. Training is bursty and tolerant of scheduling; inference is continuous, latency-sensitive and intolerant of queueing. Reserved capacity that flatters your training bill can be badly wrong for a 24/7 agent workload — and vice versa.

• Separate the curves. Commit capacity for steady-state inference; keep training and fine-tuning on burst, spot or scheduled capacity where interruption is acceptable.
• Model tokens, not instances. Unit economics should be expressed per thousand tokens or per business transaction, otherwise nobody can tell whether a feature is profitable.
• Assume scarcity has a lead time. Industry reporting through 2025 and 2026 consistently shows accelerator capacity being reserved months ahead. Architecture that assumes on-demand GPUs will always be there is a bet, not a design.
• Right-size the model to the task. A smaller domain-tuned model behind a router often beats a frontier model on cost, latency and predictability for the 80% of requests that are routine.

Decision three: the inference topology

Centralised inference is simple and often correct. But if your AI capability sits inside a store, plant, warehouse or field application, the round trip to a central region becomes the product experience. This is where an honest tiering exercise pays for itself: which inferences must be sub-100ms and local, which can be regional, and which are asynchronous and can run wherever capacity is cheapest overnight. Most enterprises discover that only a small fraction of their calls genuinely need premium, low-latency, always-warm capacity — and that fraction is where the architecture effort belongs. We cover this pattern in more depth across our architecture and AI advisory work.

Decision four: the control plane you'll wish you had built

The fourth decision is the one nobody makes deliberately: how AI workloads are governed at runtime. Without a gateway layer, every team invents its own authentication, model choice, logging, rate limiting and prompt handling. Within a year you have a dozen shadow integrations and no way to answer basic questions — which model answered this customer, on what data, at what cost. A single ingress for model traffic gives you observability, cost attribution, model substitution and policy enforcement in one place. It is far cheaper to build before the tenth use case than after it.

What to do next

Treat these four decisions as a formal architecture gate, not a platform-team detail: data placement, commitment shape, inference topology, control plane. Document the answers, cost them per transaction, and revisit them each time a new use case enters the pipeline. The organisations pulling ahead in AI are rarely the ones with the best models — they are the ones whose infrastructure choices allow the tenth and fiftieth use case to be cheaper than the first. If you're weighing these trade-offs against an existing enterprise landscape, our delivery experience is a useful reference point.