Infrastructure
The physical plant — data centers, power and cooling, the cluster network, the clouds that rent it out, and the scheduling software that keeps expensive silicon busy.
Dossier
What this layer is
The models are software. This is the building they live in.
Every token a model produces was computed somewhere physical: a chip on a board, the board in a server, the server in a rack, the rack in a hall, the hall drawing power from a substation and shedding heat into water or air. The Infrastructure layer is everything between the silicon (Layer I, below it) and the model (Layer III, above it): the data centers, the power and cooling plant, the networks that lace thousands of accelerators into one machine, the clouds that rent all of it out by the hour, and the scheduling software that decides which job runs where.
It is the least glamorous layer of the stack and the one where the constraints are hardest, because its constraints are physical. Software scales by copying. Buildings, substations, transformers, and water permits do not. When you read that a frontier lab's roadmap slipped, or that a region's grid operator is alarmed, or that a chip in high demand is sitting in a warehouse waiting for a powered shell to put it in — you are reading an infrastructure story. This layer is where the AI boom stops being a software story and becomes a construction project.
One framing to hold onto for the whole dossier: a GPU costs the same whether it is working or idle. Most of what this layer does — the networking, the scheduling, the cooling, the redundancy — exists to keep extremely expensive silicon busy. Utilization is the quiet variable underneath every economic claim made about AI, and it is decided here.
The data center
A hyperscale data center is a warehouse engineered around three flows: electricity in, data in and out, heat out. Strip away the branding and every facility is the same diagram.
The hierarchy. Chips sit on boards; boards in servers (a typical AI training server carries 8 accelerators); servers in racks; racks in rows; rows in halls (often called data halls or pods); halls in a building; buildings on a campus. Each level of the hierarchy has its own power distribution, its own cooling loop, and its own tier of network. The campus is the new unit of account: frontier training sites are now planned as multi-building campuses measured in hundreds of megawatts to gigawatts of power capacity, not in square feet. ⚑ unverified
What changed with AI. A conventional cloud data center was designed around racks drawing roughly 5–15 kW each — web servers, databases, storage. ⚑ unverified An AI training rack is a different animal: a current- generation rack-scale system (72 tightly coupled GPUs sharing a switched high-speed interconnect) draws on the order of 120 kW in a single rack, roughly ten times the old design point. ⚑ unverified That single number — rack density — cascades through everything: it obsoletes air cooling (air physically cannot move that much heat out of that little space), it concentrates power delivery into busbars and liquid-cooling manifolds the old buildings never had, and it means most of the world's existing data-center floor space cannot host frontier AI hardware without a rebuild. The AI buildout is not a repurposing of old capacity. It is new construction, on new sites, chosen for power.
Why sites are chosen where they are. The old data-center site checklist was network latency (near users) and land cost. The AI checklist is: available grid power first, water or dry-cooling feasibility second, land third, and latency to users barely at all — a training cluster serves no users; it can live anywhere electrons are cheap. This is why the map of AI construction looks nothing like the map of the internet: the new capacity goes where hundreds of megawatts can actually be interconnected this decade.
Power
Power is the binding constraint of this layer, and increasingly of the whole stack. Not chips — power. Accelerators can be manufactured in quarters; the electrical infrastructure to run them is built in years.
The load. A single modern training accelerator draws on the order of 700–1,200 watts at the chip, and roughly double that once you count its share of networking, storage, cooling, and power-conversion overhead. ⚑ unverified Multiply by a hundred thousand accelerators and a frontier training campus is a load comparable to a heavy-industrial plant or a small city — hundreds of megawatts, headed toward gigawatts on announced plans. ⚑ unverified Data centers as a category consume a low-single-digit percent of electricity in most grids today, but the AI-driven growth is concentrated in a handful of regions, and concentration, not the global average, is what breaks things. ⚑ unverified
The queue. To connect a large new load to the grid you file for interconnection, and in most developed markets that queue now runs multiple years — often longer than it takes to erect the building itself. ⚑ unverified Behind the queue sit real hardware lead times: large power transformers are quoted years out, and high-voltage switchgear, generators, and skilled electrical labor are all in deficit. ⚑ unverified This is why the industry's most watched metric quietly shifted from GPUs-on-order to megawatts-energized, and why operators pursue workarounds: siting next to existing power plants, contracting directly with nuclear operators, building on-site gas turbines, or buying stranded capacity (a shuttered smelter's grid connection is worth more than the smelter). Each workaround trades speed against cost, emissions accounting, or regulatory exposure — mechanics here; what it means for everyone else on the grid is the ENERGY lane's beat.
The load's shape. Training is not a steady draw. A synchronized training run steps in lockstep — thousands of accelerators compute, then exchange gradients, then compute — and the aggregate load can swing tens of megawatts within seconds as the cluster oscillates between compute-bound and communication-bound phases, or when a run halts and restarts. Grids were engineered for loads that ramp gently. A gigawatt-class computer that steps its draw like a strobe light is a new kind of customer, and grid operators have begun writing rules for it; some operators add on-site energy storage or dummy-load smoothing purely to make the facility grid-polite. ⚑ unverified
Energy per unit of intelligence. Two numbers anchor the layer. Training: a frontier-scale run consumes gigawatt-hours — an amount of energy meaningfully compared to the annual consumption of thousands of homes, spent once, up front, to produce an artifact (the weights). ⚑ unverified Inference: a single chat query is commonly estimated in the range of a few tenths of a watt-hour to a few watt-hours depending on model size and serving efficiency — small per query, enormous in aggregate, and unlike training it recurs with every use forever. ⚑ unverified The industry's own disclosed per-query figures are best treated as lower bounds: they are measured under favorable assumptions (high batch efficiency, excluding some overheads). The honest statement is that per-query energy is falling fast on efficiency gains while total energy rises faster on volume — both true at once.
Cooling
Every watt that enters a chip leaves it as heat, and the heat must go somewhere within milliseconds or the silicon throttles. Cooling is not a support function; at AI rack densities it is co-equal engineering.
The progression. Air cooling — cold aisle in, hot aisle out, big chillers on the roof — carried the industry for decades and tops out around the old rack densities. Above roughly 30–50 kW per rack, air gives way to liquid. ⚑ unverified The dominant AI approach is direct-to-chip liquid cooling: a cold plate clamped to each accelerator, a loop of treated water or coolant carrying heat to a rack manifold, then to a coolant distribution unit (CDU), then out of the building. Rear-door heat exchangers (a radiator on the rack's back door) are the halfway house; full immersion cooling (servers submerged in dielectric fluid) is the maximal version, real but still niche. The current-generation rack-scale training systems are liquid-cooled as designed — there is no air-cooled variant of the frontier — which is a second reason legacy facilities can't simply host new chips. ⚑ unverified
Water. The building's final heat rejection is either evaporative (cooling towers — cheap, efficient, and water-hungry: large facilities can evaporate on the order of millions of liters a day in warm weather ⚑ unverified) or dry (closed-loop radiators — water-frugal but less efficient exactly when it's hot, raising power draw). Water consumption also hides upstream: thermoelectric power generation evaporates water per megawatt-hour, so even a "dry-cooled" data center has a water footprint through its electricity. The siting math trades water, power price, and climate against each other; the mechanics are here, the watershed politics are the ENERGY lane's.
PUE, and what it doesn't say. Power Usage Effectiveness = total facility power ÷ power delivered to IT equipment. A PUE of 1.0 would mean zero overhead; older enterprise facilities ran 1.5–2.0; modern hyperscale runs roughly 1.1–1.3, and operators advertise fleet averages near the bottom of that range. ⚑ unverified Three honest caveats. First, PUE says nothing about how efficiently the IT power is used — a hall of idle GPUs at PUE 1.1 is a monument to waste. Second, marginal PUE in hot weather is worse than the annualized average that gets published. Third, PUE improvements are largely mined out: going from 2.0 to 1.2 was the industry's great efficiency story, but from 1.2 the theoretical remaining gain is under 20%, so future demand growth passes through to the grid almost one-for-one. The era in which efficiency gains absorbed demand growth is over; that is the single most important fact this layer exports to the site's energy coverage.
Networking
A training run's defining trick is making thousands of accelerators impersonate one computer. The impersonation is performed by the network, and the network is where scale goes to die.
Two networks. Every AI cluster runs (at least) two distinct fabrics. The frontend network is ordinary data-center Ethernet: it moves training data in, checkpoints out, and connects the cluster to the world. The backend (or compute) fabric exists solely so accelerators can exchange model state with each other, and it is built to a different standard entirely: 400–800 Gb/s per accelerator, microsecond latencies, and lossless delivery. ⚑ unverified The backend fabric is the expensive one — optics, switches, and cabling for a large cluster run to a meaningful fraction of the cost of the GPUs themselves. ⚑ unverified
Scale-up vs scale-out. Within a server or rack, accelerators talk over a proprietary short-range interconnect (NVLink being the canonical example) at bandwidths an order of magnitude above the network — this is the scale-up domain, and growing it (8 GPUs → 72 GPUs behaving as one memory-coherent island) is the current architectural race. Between racks, you scale out over InfiniBand or Ethernet. The programming model cares intensely about the boundary: whatever must communicate constantly (tensor parallelism) stays inside the scale-up island; whatever tolerates coarser exchange (data parallelism, pipeline stages) crosses the scale-out fabric.
InfiniBand vs RoCE. The scale-out choice of the decade. InfiniBand is a purpose-built lossless fabric with RDMA (remote direct memory access — NIC-to-NIC transfers that bypass the CPU) native to its design; it is the incumbent for training, at a price premium and effectively a single-vendor supply chain. RoCE (RDMA over Converged Ethernet) delivers the same RDMA semantics over Ethernet switches, cheaper and multi-vendor, but it demands careful lossless tuning (priority flow control, congestion control) that InfiniBand gives you by default; badly tuned RoCE fabrics produce exactly the tail-latency stalls that kill synchronized training. The hyperscalers have driven RoCE (and successor Ethernet efforts) hard because nobody wants their fabric single-sourced. Both work at frontier scale today; the engineering cost just lives in different places.
Why the network is the whole game at scale. The dominant communication pattern of training is the all-reduce: every accelerator must receive the sum of every other accelerator's gradients every step. It is a synchronized, global operation — which means every step completes at the pace of the slowest participant. One congested link, one flapping optic, one straggler GPU, and ten thousand accelerators wait together. This is why AI fabrics are built lossless and rail-optimized, why topology (fat-tree/Clos designs with full bisection bandwidth) is a first-order economic decision, and why "the network is fine on average" is a meaningless sentence — training runs on the tail of the latency distribution, not the mean.
The cluster
The cluster is where the hardware stops being inventory and becomes a machine. It is also where the failure math becomes the design driver.
Failure is the steady state. Take a component with excellent reliability and multiply it by a hundred thousand, then add optics (the most failure-prone part in the building), DRAM, fans, pumps, firmware, and cosmic-ray bit flips. At frontier scale, something in the cluster fails on the order of every few hours; published post-mortems of large training runs read as catalogs of GPU faults, HBM errors, and network flaps, with unhealthy hardware being swapped continuously throughout the run. ⚑ unverified A synchronized training job, by construction, can be halted by any single participant. So the layer's real deliverable is not peak performance but goodput: the fraction of wall-clock time the run spends making forward progress.
The countermeasures. Checkpointing (periodically writing the full model and optimizer state — terabytes — to storage, so a failure costs minutes of recomputation instead of weeks), hot spares (idle-but-warm nodes swapped in when one dies), health-checking and pre-flight burn-in (a large share of faults appear in hardware's first weeks), straggler detection (a GPU that is slow-but-alive is worse than a dead one, because nothing alarms), and topology-aware scheduling (place a job so its heaviest communication stays inside the cheapest bandwidth domain). None of this is exotic; all of it is mandatory; and every hour spent checkpointing, restarting, or waiting on a straggler is purchased at the full hourly price of the entire cluster.
Training vs inference clusters. The same silicon, opposite machines. A training cluster is one huge synchronized job: it wants maximum interconnect, tolerates no stragglers, runs for months, and can live in the desert. An inference fleet is millions of small independent jobs: it wants to sit near users (latency), scale with the diurnal traffic curve, and maximize throughput per dollar via batching. The economics invert too — training is capex spent once before any revenue; inference is the recurring marginal cost of every customer forever. Purpose-built inference infrastructure (smaller interconnect domains, cheaper accelerators, aggressive batching and caching) is the fastest-growing part of the buildout, because that is where the demand curve actually recurs.
Cloud
Almost nobody who uses AI compute owns it. The cloud layer is the market structure of infrastructure: who holds the asset, who rents it, and on what terms.
The hyperscalers. AWS, Microsoft Azure, and Google Cloud own the largest fleets, sell every layer of abstraction from bare GPU to managed model API, and are simultaneously the landlords, the biggest customers (their own models), and — through custom silicon (TPU, Trainium, Maia) — competitors to their chip supplier. Oracle emerged as the surprise fourth, winning frontier-scale training contracts by moving fast on dedicated capacity. The hyperscaler AI business runs substantially on committed contracts: multi-year reserved capacity deals, some in the tens of billions of dollars, which convert uncertain future demand into today's construction financing — and which transfer demand risk onto whoever signed the commitment. ⚑ unverified
The neoclouds. A new species — CoreWeave, Lambda, Crusoe, and peers — that sells GPUs and nothing else. No decades of managed services, just dense AI clusters, often financed by debt collateralized on the GPUs themselves and de-risked by anchor contracts from a handful of AI labs and hyperscalers. ⚑ unverified They exist because hyperscaler GPU capacity was scarce and list-priced, and because a pure-play can stand up a cluster faster. Their structural questions are the interesting part: customer concentration (a few labs are most of revenue), asset life (debt terms versus how fast this generation of GPU depreciates), and what happens to rental rates whenever supply catches demand. That set of questions is a load-bearing input to the site's Bubble Watch — flagged here, adjudicated there.
The rental economics. GPU-hours are quoted like a commodity but priced like a market squeeze: on-demand rates carry large premiums over committed rates, and spot/preemptible capacity floats beneath both. A serious buyer's actual unit cost is dominated by commitment terms and, above all, by utilization of what they committed to. Meanwhile an aggregation layer (serverless GPU platforms, cross-cloud brokers, inference-as-a-service) is commoditizing the raw hour from above — the classic cloud pattern, replaying at higher voltage.
Orchestration infrastructure
The last piece is the software that turns a warehouse of servers into a schedulable resource. (Distinct from Layer IV's model-level orchestration — RAG and agents wire up a model; this wires up a fleet.)
Slurm is the HPC world's batch scheduler, decades old, and still the default for large training: it thinks in whole-node allocations, gang-scheduled jobs, queues, and topology awareness — exactly the shape of a training run. Kubernetes is the cloud-native scheduler that won everything else: it thinks in long-running services, autoscaling, and declarative state — exactly the shape of inference serving. The industry's awkward secret is that AI needs both shapes and each system is being bent toward the other (gang-scheduling and queueing retrofitted onto Kubernetes; container workflows bolted onto Slurm). Ray sits above either: a distributed-compute framework that lets Python code fan out across a cluster, widely used for data preprocessing, reinforcement-learning pipelines, hyperparameter sweeps, and serving. Above all three sit the launcher/broker tools (SkyPilot and kin) that chase available capacity across clouds — because in a shortage, the scheduler that matters is the one that can find you a GPU anywhere.
The stakes of this unglamorous software are the layer's core number: a scheduler that leaves gaps between jobs, packs topology badly, or recovers slowly from failure burns money at the rate of the whole cluster's hourly cost. Scheduling is utilization policy, executed.
Where it breaks — the failure-mode ledger
- Power before chips. Interconnection queues, transformer lead times, and substation construction gate the buildout more than silicon supply. A GPU without a powered shell is inventory, and "powered shell" is now the scarce asset class. ⚑ unverified
- The grid meets a strobe light. Synchronized training load swings stress grid equipment engineered for gentle ramps; mitigation (storage, smoothing, curtailment agreements) adds cost that never appears in per-GPU pricing.
- Heat density outruns buildings. Each accelerator generation raises rack power; each rise strands more of the existing air-cooled fleet. Infrastructure depreciates by obsolescence here, not wear.
- The tail kills the run. Synchronized training runs at the speed of the slowest of ~100,000 participants. Congestion, stragglers, and failed optics convert directly into idle time on the entire cluster.
- Failure math at scale. Component-level reliability that rounds to perfect still produces hours-frequency faults at cluster scale; goodput, checkpoint overhead, and recovery time are the real performance metrics.
- Utilization is fragile. Committed capacity + spiky demand + scheduling friction = idle silicon at full price. Every layer above this one inherits its unit economics from this single ratio.
- Concentration risk everywhere. One dominant chip vendor, one dominant fabric vendor, a handful of landlord clouds, a few anchor tenants behind the neocloud debt stack. The physical layer is the least diversified part of the industry.
The economics — capex, PUE, utilization, and the buildout
The infrastructure layer is where the AI industry's money actually lands. The hyperscalers' combined capital expenditure — the majority of it AI infrastructure — is running at a scale of hundreds of billions of dollars per year, an industrial buildout with few peacetime precedents. ⚑ unverified
The cost stack of a compute-hour. Take one delivered GPU-hour and cut it open. The largest slice is depreciation of the accelerator and its server — the silicon is roughly half or more of total cost of ownership, amortized over a useful life the industry keeps arguing about (3, 4, 5, 6 years — each year added flatters every AI income statement that depends on it). ⚑ unverified Then the backend network (a meaningful double-digit percent), then the building and power infrastructure (amortized over decades — the shell outlives many generations of chips inside it), then electricity (a surprisingly modest share at today's silicon prices — which is why operators chase power availability far harder than power price), then cooling, operations, and financing. ⚑ unverified
The two multipliers. Whatever the stack sums to, two ratios scale it. PUE multiplies the energy line (mostly optimized-out already, as above). Utilization divides everything: a cluster at 40% useful output has 2.5× the effective unit cost of the same cluster at 100%, and real-world figures — model-FLOPs utilization on training runs commonly cited around 35–45%, fleet-level utilization on inference lower and jealously unpublished — sit far enough from 100% that utilization, not efficiency, is the industry's biggest hidden cost lever. ⚑ unverified
The mismatch to watch. The asset lives 3–5 years (chips) inside a shell that lives 20–30, financed today against demand projected for the 2030s, with the recurring revenue (inference) still small relative to the recurring cost of standing ready. Whether that gap is a timing artifact or a structural overbuild is precisely the question this site's Bubble Watch exists to track — this dossier's job is only to make the machine legible enough that the reader can follow that argument with the mechanics in hand.
Concepts & Guides
What a hyperscale data center actually is
The full anatomy, walked physically: utility feed → substation → switchgear → UPS → busway → rack → server → chip, and the mirror path for heat. The hierarchy (chip/server/rack/hall/building/campus), what a megawatt of IT load physically looks like, why rack density is the single number that dates a facility, and why AI flipped the site-selection logic from "near users" to "near power." Failure modes: stranded legacy floor space, powered-shell scarcity, the years-long gap between announcing a campus and energizing it. The engineer-nod detail: redundancy tiers (N, N+1, 2N) and why training clusters often accept less facility redundancy than a bank's web servers — a checkpointed training job can die and resume; a payment system cannot.
Power, cooling, and PUE — the thermodynamics bill
Every watt in becomes heat out; the guide follows the watt. Chip power draw by generation, the doubling from chip-power to facility-power once overheads stack, why air cooling hit a wall at AI rack densities, and the liquid-cooling progression (rear-door → direct-to-chip → immersion) with the actual plumbing: cold plates, CDUs, facility loops, evaporative vs dry heat rejection and the water math of each. PUE defined honestly: what it measures, what it hides (idle IT load looks "efficient"), why fleet averages near 1.1 mean the efficiency era is over and demand growth now passes straight through to the grid. ⚑ unverified Failure modes: hot-weather marginal PUE, coolant leaks in dense racks, the grid-side strobe-light problem of synchronized training load.
The training cluster vs the inference fleet
Same chips, opposite machines — the guide most readers need before any AI-economics argument makes sense. Training: one synchronized job, all-reduce every step, speed of the slowest participant, goodput vs peak FLOPs, checkpointing math (how often to save terabytes so a failure costs minutes, not weeks), why the cluster can live anywhere. Inference: millions of independent requests, latency vs throughput, batching (the single biggest serving lever), prefill vs decode phases and why they stress hardware differently, the diurnal demand curve, and why inference wants to be near users. The economic punchline: training is capex spent once; inference is the marginal cost of every customer forever — and the buildout is quietly pivoting toward the second.
Networking — InfiniBand, RoCE, and why the fabric is half the fight
The two-network truth (frontend Ethernet vs backend compute fabric), scale-up vs scale-out domains and why the boundary dictates how models are parallelized, RDMA explained properly (NIC-to-NIC memory transfers, no CPU in the path), InfiniBand vs RoCE as engineering-cost-relocation rather than good-vs-bad, lossless fabrics (priority flow control, congestion control) and what happens when tuning is wrong, fat-tree/Clos topologies and bisection bandwidth as a purchasing decision. Failure modes: the all-reduce tail-latency trap, optics as the highest-failure-rate component in the building, congestion collapse on mis-tuned RoCE, single-vendor fabric risk. Engineer-nod: rail-optimized topology and why collective-communication libraries (NCCL and kin) are secretly some of the most performance-critical code in AI.
Scheduling and orchestration — Slurm, Kubernetes, Ray
Fleet-level orchestration, disambiguated from Layer IV's model-level orchestration. Why training jobs are gang-scheduled batch work (Slurm's home turf: whole-node allocation, queues, topology awareness) and inference is long-running autoscaled services (Kubernetes' home turf: declarative state, health checks, horizontal scaling), why each system is being bent toward the other, where Ray fits (distributed Python for preprocessing, RL, sweeps, serving), and what the capacity-broker layer (SkyPilot and kin) solves in a shortage. Failure modes: scheduling gaps as pure money-burn, fragmentation (free GPUs in unusable scatterings), priority inversion between research and production, the multi-tenancy problem of sharing a cluster without anyone's job stalling everyone's.
Utilization — why it is the whole game
The layer's capstone argument, made with arithmetic. A GPU's cost accrues every hour regardless of output, so effective unit cost = sticker cost ÷ utilization — and every real-world force pushes utilization down: failure recovery, checkpoint overhead, stragglers, scheduling gaps, demand spikes sized-for, capacity committed-to-but-unused, and the training-MFU gap (the difference between "the GPUs were allocated" and "the GPUs did useful model-FLOPs"). Walks the compounding: a plausible stack of individually reasonable losses lands a fleet at a fraction of its theoretical output, which is the difference between an AI business model that clears and one that doesn't. Ends by tracing utilization assumptions through the layers above: every per-token price, every API margin, every "cost of intelligence is collapsing" chart inherits this denominator. ⚑ unverified
The Kit
The real tools, vendors, and models of the Infrastructure layer — what each one is, when to reach for it, and the honest caveats. This is the layer’s directory slice: 9 entries are flagged affiliate-eligible, and none carry a live link yet — mentions stay plain until partner programs exist, and links will activate by data change, not re-edit.
The full directory lives on its own page.
Open The Kit — the Infrastructure directory →No affiliate links are live on this page yet — no partner programs have been joined. Tool mentions are unlinked until they are; when links activate, this disclosure applies.
Winner Matrix
Platforms that host and serve models for you — per-token APIs over open models, and serverless/dedicated GPU serving where you bring the model. The layer between renting raw GPUs and calling a closed frontier API. Two pricing models dominate: per-token (provider eats utilization risk) and per-second compute (cold starts are your problem). Ease is editorial: 1 = platform engineering project, 5 = API key and one curl.
| Provider | Pricing model | Open-model coverage | Cold-start / latency | Ease (1–5) | Best for |
|---|---|---|---|---|---|
| Together AI | Per-token serverless (≈$0.05–7.00/1M by model) + per-minute dedicated (H100 $6.49/hr, B200 $11.95/hr) ⚑ unverified | Among the broadest — 100+ chat/vision/image/embedding models + fine-tuning API ⚑ unverified | Always-warm serverless for popular models; solid but not the TTFT leader ⚑ unverified | 4/5OpenAI-compatible API, instant serverless; dedicated-endpoint sizing adds a step. | One vendor for open-model APIs, fine-tuning, and a scale-up path to dedicated capacity. together.ai/pricingartificialanalysis.ai |
| Replicate | Mostly per-second hardware (H100 $0.001525/sec) + per-token on some LMs; cold-boot billed on private deployments ⚑ unverified | Enormous community catalog — strongest in image/video/audio; anyone can push via Cog ⚑ unverified | The known weakness — cold models take tens of seconds to minutes; min-instances ≈$36.60/day for a warm H100 ⚑ unverified | 5/5The simplest run-a-model experience in the category — one API call against thousands of ready models. | Shipping image/video/audio features fast; prototyping exotic community models. Acquired by Cloudflare Dec 2025; pricing unchanged so far. replicate.com/pricingreplicate.com/docs |
| Fireworks AI | Per-token serverless (from ≈$0.07/1M input; cached input 50%; batch 50%) + on-demand per-GPU-second ⚑ unverified | Broad text/vision/embeddings across major open families + fine-tuning with serverless deploy ⚑ unverified | Strong — consistently near the top on TTFT among GPU-based providers ⚑ unverified | 4/5OpenAI-compatible, generous free credits, quick start; dedicated deployments need more config. | Production apps on open models where TTFT matters and you may fine-tune later. fireworks.ai/pricingartificialanalysis.ai |
| Groq | Per-token only, aggressively cheap on small models (8B-class $0.05/1M input; 70B ≈$0.59/$0.79); batch and caching at 50% ⚑ unverified | Narrow by design — curated open models compiled for LPU hardware; no BYO model, no fine-tuning ⚑ unverified | The speed leader — ≈280–1,000 tok/sec by model, several times typical GPU throughput ⚑ unverified | 4/5Free tier, OpenAI-compatible, instant start; the catalog constraint is the only friction. | Latency-critical chat and agent loops where a supported open model fits. groq.com/pricingartificialanalysis.ai |
| Modal | Per-second serverless compute (H100 ≈$3.95/hr, A100-80 ≈$2.50/hr), billed only while code runs ⚑ unverified | Not a catalog — bring model + serving code in Python; anything in a container runs ⚑ unverified | Good for serverless — fast container snapshots keep cold starts low; warm pools cost money ⚑ unverified | 3/5Superb Python DX, but it's a code platform, not an inference API — you write the serving layer. | Python teams running custom models, batch jobs, and pipelines on serverless economics. modal.com/pricingmodal.com docs |
| Baseten | Per-minute dedicated while active (H100-80 ≈$6.50/hr · A100 $4.00/hr · B200 ≈$9.98/hr); scale-to-zero free when idle ⚑ unverified | Popular open models + BYO via Truss; positioned around dedicated deployments ⚑ unverified | Strong optimized-serving reputation (TensorRT-LLM); scale-to-zero cold starts actively engineered down ⚑ unverified | 3/5Polished deploy flow, but replicas/autoscaling thinking is on you. | Production teams serving their own or customized models with SLAs on managed dedicated capacity. baseten.co/pricing |
| Anyscale | Platform markup on cloud compute (effective H100 commonly 1.5–2× bare rates); old per-token Endpoints discontinued — confirm status ⚑ unverified | Unrestricted in principle — Ray Serve runs anything; no curated per-token catalog ⚑ unverified | Not serverless-inference — cluster spin-up, not cold start, is the relevant delay ⚑ unverified | 2/5Powerful for distributed workloads, but you're adopting Ray as a framework — heaviest lift here. | Teams already invested in Ray running mixed training-plus-serving pipelines at scale. anyscale.com/pricing |
| Hugging Face Inference | Inference Providers: per-token routed to partners at list price, no HF markup, monthly credits on paid plans · Endpoints: per-minute dedicated from ≈$0.50/GPU/hr ⚑ unverified | The widest surface in the ecosystem — the Hub's catalog via Providers; virtually any Hub model on Endpoints ⚑ unverified | Providers inherits the routed provider's latency; Endpoints has real cold starts on scale-to-zero ⚑ unverified | 4/5If the model's on the Hub, deployment is a few clicks; routing abstracts many vendors behind one token. | Developers in the HF ecosystem wanting one account across providers, or cheap endpoints for niche Hub models. huggingface.co/pricing |
Per-token providers (Groq, serverless Together/Fireworks) are the cheapest way to serve popular open models at low-to-medium volume because you pay nothing for idle. Per-second platforms (Modal, Baseten, Replicate deployments, HF Endpoints) win when the model is custom, traffic is sustained, or you need hardware control — at which point cold-start engineering becomes your job. The crossover is utilization: a dedicated H100 at $4–6.50/hr beats per-token pricing only when kept meaningfully busy.
Affiliate disclosure: where a placement slot exists, it is labelled and links to our Disclosure page. Partner status never influences scores or rankings — editorial only.
News & Video
What this layer's feed covers
- The buildout — campus announcements, hyperscaler capex disclosures (the quarterly earnings numbers are this layer's scoreboard), construction starts and slips, powered-shell and land deals.
- Power and grid — interconnection decisions, utility filings and rate cases, nuclear/gas/renewable PPAs, on-site generation moves, grid-operator rule changes for large flexible loads, transformer and switchgear supply news. (Mechanics and market facts here; community and environmental consequences route to AI Impact's ENERGY lane — same events, different desk.)
- Cooling and water — liquid-cooling adoption milestones, water-permit fights as facility-siting facts, dry-cooling and heat-reuse engineering.
- Cloud and neocloud markets — GPU rental pricing moves, committed-contract announcements, neocloud financings and debt structures, capacity gluts or squeezes, new entrants. (Anything smelling of circularity or fragility also feeds Bubble Watch — the feed tags it for both.)
- Networking and cluster engineering — fabric announcements (InfiniBand generations, Ethernet-for-AI efforts), published training post-mortems and cluster papers (the honest ones are gold), scheduler and orchestration releases.
Example source classes (curation pool, not endorsements)
- Primary/disclosed: hyperscaler earnings calls and capex guidance; utility and grid-operator filings; operator engineering blogs (the cloud providers and large labs publish real cluster engineering); published training reports and infrastructure papers.
- Trade and analyst: data-center trade press (construction, siting, power deals); semiconductor/infra analyst shops that model the cost stack; energy-sector press where it intersects large loads.
- Community/technical: HPC and networking conference talks (the fabric debates happen there); systems conference papers; the practitioner blogs that dissect cluster failures honestly.
Video (curated embeds, tagged to this layer)
The layer suits video unusually well — this is the one part of the AI stack you can point a camera at. Curate: data-center hall and campus tours (several operators have produced genuinely substantive walk- throughs), liquid-cooling teardowns and rack-scale system breakdowns, grid-and-power explainers from the energy-engineering channels, and conference talks on cluster networking and training-at-scale post-mortems. Selection bar matches the text: real mechanism, real numbers, no hype reels. Sponsored placements in the feed are permitted under the site's disclosure rule; the analysis above the feed stays clean.