Hardware

Dossier

What this layer is

Everything in AI that you can drop on your foot. Below the models, below the clouds, below the frameworks, there is a physical machine doing arithmetic — and this layer is that machine: the accelerator chips (GPUs, TPUs, custom ASICs), the high-bandwidth memory stacked next to them, the interconnect fabric that lashes tens of thousands of them into one logical computer, and the manufacturing chain — fabs, lithography, advanced packaging — that produces all of it.

The hardware layer is where the AI boom stops being software economics and becomes industrial economics. Software scales at near-zero marginal cost. Silicon does not. Every model you have ever prompted was trained on, and is served from, a finite pool of physical accelerators that took years to manufacture, cost tens of thousands of dollars each, draw kilowatts under load, and depreciate whether or not anyone uses them. When people argue about whether AI spending is sustainable, this is the layer they are actually arguing about — it is where most of the money lands.

What it does

One job, at absurd scale: multiply large matrices of numbers, very fast, over and over.

A transformer — training or inference — decomposes almost entirely into matrix multiplications and a thin garnish of element-wise operations (activations, normalizations, softmax). Training a frontier model means performing on the order of 10²⁵–10²⁶ floating-point operations ⚑ unverified spread across months on thousands of accelerators. Serving that model means, for every single token generated, streaming most of the model's weights out of memory and through the multiply units — billions of parameters read per token, per user, per request.

That workload profile — massive, regular, parallel arithmetic with enormous data movement — is what this layer is shaped around. Everything distinctive about AI hardware (thousands of small cores instead of a few big ones, memory stacked millimetres from the die, chip-to-chip links running at terabytes per second) exists because of that profile.

Why it exists

Because CPUs are the wrong tool. A modern server CPU is a marvel of making one instruction stream fast: branch prediction, speculative execution, deep caches — silicon spent on latency, on guessing what a single thread will do next. AI arithmetic doesn't need any of that. It needs throughput: the same multiply-accumulate operation applied to millions of independent numbers where the entire "control flow" is known in advance.

GPUs, which had spent two decades learning to do exactly that for 3D graphics (shading millions of pixels is also embarrassingly parallel arithmetic), turned out to be almost accidentally the right substrate. The 2012 AlexNet moment — a neural net trained on two consumer gaming GPUs beating the field at image recognition — was the industry's proof of concept. Everything since is that insight compounding: from repurposed gaming cards, to data-center GPUs with the graphics plumbing stripped out, to dedicated tensor units (Nvidia's Tensor Cores, Google's TPU systolic arrays) that do nothing except dense matrix math, to today's rack-scale systems where the "chip" is really 72 GPUs pretending to be one.

The layer exists, in short, because the physics of neural networks demanded a different trade-off than general-purpose computing ever made: sacrifice flexibility, buy throughput.

Where it breaks

This is the honest part, and it matters more than the spec sheets.

The memory wall. Compute has outrun memory. Over the past two decades, peak FLOPs per chip grew roughly 3× every two years while memory bandwidth grew roughly 1.6× ⚑ unverified — and the gap compounds. The result is the defining imbalance of modern AI hardware: the multiply units are usually starving. In LLM inference specifically, generating a token requires reading essentially all active model weights from memory, so the speed limit is bandwidth, not FLOPs — an H100's compute units can sit largely idle while its memory system works flat out. This is why vendors now lead their marketing with terabytes-per-second rather than teraflops, why HBM exists at all, and why a huge fraction of software effort at this layer (quantization, batching, KV-cache tricks, FlashAttention) is really just memory-wall mitigation. See the Concepts section for the mechanism.

The HBM and packaging bottleneck. High-bandwidth memory is stacked DRAM — up to 12 dies, thinned, drilled through with thousands of vertical vias, and bonded next to the GPU on a silicon interposer. Only three companies on Earth make it (SK Hynix, Samsung, Micron), and joining it to the GPU requires advanced packaging — TSMC's CoWoS process being the chokepoint of record. Through 2023–2025, the binding constraint on global GPU supply was repeatedly not wafer capacity but CoWoS packaging capacity and HBM yield ⚑ unverified. This is a structural fragility: the industry's growth rate is gated by two or three specific production lines in Taiwan and Korea. When you read "GPU shortage," the accurate translation is usually "packaging and memory shortage."

Fab concentration. Every leading-edge AI chip — Nvidia's, AMD's, Google's, Amazon's, Apple's — is manufactured by one company, TSMC, overwhelmingly in Taiwan. TSMC holds roughly 90% of leading-edge (≤5nm-class) foundry capacity ⚑ unverified. One step upstream it gets narrower still: every EUV lithography machine, without which no leading-edge fab operates, is built by exactly one company, ASML, in the Netherlands. The entire global AI buildout — hundreds of billions of dollars of announced capex — bottlenecks through a single island with an active geopolitical dispute attached and a single Dutch toolmaker. There is no plan B at current volumes; Intel's foundry effort and TSMC's own Arizona fabs are real but years from absorbing the load ⚑ unverified. This is the least-diversified critical supply chain of any industry this size, and every serious AI risk model has to price it.

Power and depreciation. Two slower-burning failure modes. First, power: a single B200-class GPU draws around 1,000–1,200W ⚑ unverified, a rack-scale GB200 NVL72 system on the order of 120kW ⚑ unverified — pushing AI data centers into direct collision with grid capacity (that story lives in the Infrastructure layer, but it is hardware that creates it). Second, depreciation: accelerators are commonly depreciated over five to six years ⚑ unverified, but Nvidia now ships a materially better generation roughly every year. If the economic life of a GPU is shorter than its accounting life, a lot of reported cloud and hyperscaler earnings are quietly flattered. This is one of the sharpest open questions in AI economics, and it is a hardware question.

The economics — who captures the margin

Follow one dollar of AI capex down the stack and the capture is wildly uneven.

Nvidia takes the largest single cut. Data-center GPUs sell for roughly $25,000–$40,000 per unit ⚑ unverified against manufacturing costs a small fraction of that; Nvidia's corporate gross margin has run in the ~70–75% range through the boom ⚑ unverified. The margin is not "for the silicon" — TSMC makes the silicon. It is the rent on the CUDA software moat, the networking stack, and the absence (so far) of a substitute buyers trust at scale. Nvidia's data-center revenue went from roughly $15B in FY2023 to over $115B in FY2025 ⚑ unverified — the single largest and fastest margin capture in the history of the semiconductor industry.

TSMC takes a solid, lower cut — gross margins in the ~50s ⚑ unverified — on every leading-edge chip regardless of whose logo it carries. TSMC is the layer's true toll collector: whoever wins the accelerator war, the wafers clear through the same fabs.

The HBM makers take a newly fat cut. Memory was historically a brutal commodity business; HBM broke the pattern. It is supply-constrained, technically hard, and sold into buyers who cannot say no — SK Hynix rode HBM to record profitability ⚑ unverified, and HBM pricing per gigabyte runs at a multiple of conventional DRAM ⚑ unverified.

ASML takes a small-volume, high-price cut — EUV machines at roughly $180–220M each, High-NA EUV at ~$350–400M ⚑ unverified — with a literal monopoly at the leading edge.

The buyers carry the risk. Hyperscalers and AI labs — combined capex guidance for the big four running past $300B for 2025 ⚑ unverified — pay all of the above margins and then must recover the money by selling compute or model access. Every layer above this one is, in accounting terms, an attempt to earn back what was paid into this one. Whether the revenue at the top of the stack ever justifies the capex at the bottom is the bubble-watch question, and it is asked in this layer's units: dollars per GPU, tokens per second, watts per rack.

Concepts & Guides

GPU vs CPU vs TPU — three answers to one question

The question is: given a fixed silicon budget, what do you spend it on?

CPU: spend it on latency. A server CPU dedicates most of its die to making individual instruction streams fast — big caches, branch predictors, out-of-order execution engines. It has a few dozen powerful cores. It is the right answer when your workload is branchy, serial, and unpredictable: databases, web servers, operating systems. For dense linear algebra it is catastrophically inefficient — most of the transistors you paid for are solving problems (control-flow guessing) that matrix multiplication doesn't have.

GPU: spend it on throughput. A GPU inverts the ratio: minimal control logic, maximal arithmetic. An H100 carries 132 streaming multiprocessors and tens of thousands of parallel lanes ⚑ unverified, organized around a SIMT model — thousands of threads executing the same instruction on different data. It hides memory latency not with caches and prediction but with sheer occupancy: when one group of threads stalls waiting on memory, another is swapped in within a clock cycle. Since Volta (2017), Nvidia GPUs also carry Tensor Cores — dedicated units that consume small matrix tiles per instruction rather than scalars, which is where virtually all AI FLOPs now execute. The GPU keeps enough generality to run arbitrary CUDA kernels, which is why it won: researchers could try anything.

TPU: spend it on one workload. Google's TPU deletes the remaining generality. Its core is a systolic array — a large fixed grid of multiply-accumulate units (256×256 in recent generations ⚑ unverified) through which operands flow in a rhythmic wave, each unit passing partial results to its neighbour. No instruction fetch per operation, no thread scheduling: data marches through a fixed pipeline. For dense matrix math it delivers superior performance-per-watt ⚑ unverified; for anything that isn't dense matrix math it is the wrong machine. The TPU bet only works because Google controls its whole stack (XLA compiler, JAX/TensorFlow, its own data centers) and can guarantee the workload fits the silicon.

The engineering summary: CPU is a few brilliant generalists, GPU is an army with a shared drill sergeant, TPU is a factory line built for exactly one product. AI is currently that one product — which is why the money flows toward the right-hand end of this spectrum, and why the risk concentrates there too: specialized silicon is stranded capital if the workload ever changes shape.

HBM and the memory wall

Start with the imbalance. An H100 SXM can perform roughly 989 trillion dense BF16 operations per second, but can move only about 3.35 TB/s between its memory and its compute ⚑ unverified. Divide those numbers: to keep the multiply units fed, you would need to perform on the order of ~300 floating-point operations for every byte you fetch. Most of an LLM's inference workload performs fewer than 1 per byte in the decode phase — every weight is read, used in a couple of multiplies, and discarded until the next token. The compute units are a firehose; memory is a straw.

HBM is the industry's best answer, and it is a packaging answer, not a chemistry answer. Take ordinary DRAM dies, thin them to tens of microns, stack 8–12 of them vertically, drill thousands of through-silicon vias (TSVs) straight down through the stack, and mount the stack millimetres from the GPU die on a silicon interposer. The interposer allows a memory bus thousands of bits wide — versus 64 bits for a conventional DIMM channel — so bandwidth goes up ~an order of magnitude while energy-per-bit-moved goes down, because the bits travel millimetres instead of centimetres. HBM3 → HBM3e → HBM4 ⚑ unverified is the ladder: more stacks per GPU, more layers per stack, faster pins.

The failure modes are exactly where you'd expect for something this baroque: yield (one bad die in a 12-high stack can kill the stack), heat (stacked dies insulate each other), cost (HBM runs at a multiple of DDR5 per gigabyte ⚑ unverified), and supply (three manufacturers, and every AI chip vendor bidding for the same output). HBM capacity per GPU is also the hard ceiling on what model fits on a chip without sharding — which is why "141GB vs 192GB" is a headline war between Nvidia and AMD, and why memory capacity, not FLOPs, increasingly decides procurement.

Compute-bound vs memory-bound — the roofline, and why it decides everything

The single most useful mental model at this layer. Every kernel has an arithmetic intensity: FLOPs performed per byte of memory traffic. Every chip has a ridge point: peak FLOPs divided by memory bandwidth. If your kernel's intensity is above the ridge, you are compute-bound — buying more FLOPs helps. Below it, you are memory-bound — more FLOPs are useless; only bandwidth (or restructuring the computation) helps.

For an H100 SXM the ridge sits around 295 FLOPs/byte at BF16 ⚑ unverified. Now place the two halves of LLM inference on that chart:

• Prefill (processing the prompt): every weight fetched is used across all prompt tokens at once. Intensity scales with sequence length — easily thousands of FLOPs/byte. Compute-bound. This is why time-to-first-token tracks FLOPs.
• Decode (generating tokens one at a time, batch of one): every weight is fetched to do ~2 FLOPs. Intensity ≈ 1–2. Memory-bound by two orders of magnitude. Tokens-per-second ≈ bandwidth ÷ bytes-of-active-weights — you can estimate a GPU's single-stream decode speed on a napkin, and the napkin will be right within ~20%.

Almost every major inference optimization is a scheme to climb the roofline: batching (share each weight fetch across many users' tokens), quantization (fewer bytes per weight → fewer bytes fetched), speculative decoding (turn several sequential decode steps into one parallel verification, which looks like prefill), FlashAttention (restructure attention to keep intermediates in on-chip SRAM instead of round-tripping HBM), MoE sparsity (fetch only the experts a token routes to). None of these add FLOPs. All of them dodge bytes. Once you see the layer this way, GPU marketing decodes itself: a spec-sheet FLOPs number quoted "with sparsity" at FP4 is a compute-bound number for a mostly memory-bound world.

Interconnect — why the cluster is the computer

No frontier model fits on one GPU, and no frontier training run fits on one node. GPT-4-class training runs used tens of thousands of GPUs ⚑ unverified; the model's weights, activations, and optimizer state are sharded across all of them, which means partial results must be exchanged constantly — after essentially every layer, every microbatch. The interconnect is not plumbing; it is a first-order determinant of how fast the whole system runs, because a stalled network stalls every GPU on it simultaneously at ~$30,000 ⚑ unverified a socket.

The topology comes in tiers:

• Scale-up (inside the node/rack): NVLink. Direct GPU-to-GPU links at 900 GB/s per GPU in generation 4 (H100), 1.8 TB/s in generation 5 (Blackwell) ⚑ unverified — roughly an order of magnitude beyond what PCIe offers. NVSwitch chips extend this to all-to-all connectivity: 8 GPUs in an HGX baseboard, and in the GB200 NVL72 rack, 72 GPUs sharing one NVLink domain — close enough to uniform memory access that software can treat the rack as a single giant accelerator.
• Scale-out (across the cluster): InfiniBand or Ethernet. 400–800 Gb/s per link ⚑ unverified — note the bits: ~50–100 GB/s, an order of magnitude below NVLink — with RDMA so GPUs exchange memory without touching CPUs. Nvidia sells the dominant InfiniBand fabric (via the Mellanox acquisition); the rest of the industry, organized under the Ultra Ethernet banner plus Broadcom's switch silicon, is fighting to make Ethernet good enough, because nobody wants the GPU monopolist owning the network too.
• The collective operations. The workhorse is all-reduce: every GPU holds a partial gradient; every GPU must end with the sum. Bandwidth-optimal ring and tree algorithms (NCCL) make the cost proportional to data size regardless of GPU count — but latency-sensitive and jitter-sensitive. One slow link, one flaky optical transceiver among thousands, degrades the entire step time for the whole cluster. At scale, transceiver and link failures are a daily operational fact ⚑ unverified, and large training runs checkpoint obsessively because of them.

The failure mode worth internalizing: parallelism strategies are chosen by the interconnect, not despite it. Tensor parallelism (splitting individual matrix multiplies) needs NVLink-class bandwidth, so it stays inside the rack. Data and pipeline parallelism tolerate the slower inter-node fabric. When someone says a model was trained with "TP8, PP4, DP128," they are reading you the network topology. The cluster is the computer; the chip is just its arithmetic unit.

The chain: sand → fab → packaging → board → cloud

The full gestation of an accelerator, because each stage is a separate industry and a separate failure point.

1. Design (Nvidia, AMD, Google/Broadcom): years of architecture work compiled into mask sets. The designers own the margin but no factories — "fabless."
2. Lithography tools (ASML, monopoly): EUV machines — 13.5nm light generated by pulsing tin droplets with lasers 50,000 times a second, focused by the flattest mirrors ever made ⚑ unverified. ~$200M per machine, ~$400M for High-NA ⚑ unverified. Every leading-edge fab on Earth queues for them.
3. Fabrication (TSMC, ~90% of leading edge ⚑ unverified): three to four months per wafer start-to-finish ⚑ unverified, thousands of process steps, in Taiwan. A wafer of dies emerges; imperfect dies are binned or discarded — yield is the fab's whole economics.
4. Memory (SK Hynix, Samsung, Micron): HBM stacks manufactured in parallel on their own risky supply line — this stage, not logic, was the boom's binding constraint through much of 2023–2025 ⚑ unverified.
5. Advanced packaging (TSMC CoWoS, chokepoint of record): GPU die + HBM stacks bonded onto a silicon interposer. Capacity here gated global GPU shipments repeatedly ⚑ unverified.
6. Board and system (Foxconn, Quanta, Supermicro, Dell): packaged chips onto HGX baseboards, into servers, into racks with power and liquid-cooling plumbing.
7. Deployment (hyperscalers, neoclouds): racks into data centers, onto grids — where the Hardware layer hands off to the Infrastructure layer.

End to end: on the order of a year from wafer start to a GPU serving tokens, with the two narrowest points (EUV tools, CoWoS+HBM) each controlled by one to three companies. This chain is why AI hardware supply cannot surge on demand, why export controls work at all (choke stages 2–3 and nothing downstream matters), and why every serious forecast of AI compute growth is really a forecast of TSMC's packaging expansion schedule.

The Kit

The real tools, vendors, and models of the Hardware layer — what each one is, when to reach for it, and the honest caveats. This is the layer’s directory slice: 9 entries are flagged affiliate-eligible, and none carry a live link yet — mentions stay plain until partner programs exist, and links will activate by data change, not re-edit.

Winner Matrix

News & Video

What the Hardware feed covers. Chip launches and real benchmarks (announcement specs vs. delivered performance — MLPerf rounds are the honest scoreboard); the supply chain (TSMC capex and CoWoS expansion, HBM roadmaps and allocation fights, ASML orders); the market structure (Nvidia data-center revenue, AMD's traction, hyperscaler custom-silicon programs, neocloud financing); export controls and the geopolitics of Taiwan; GPU rental price movements (our own tracked series — [asset: B20 GPU price/availability]); and the physical-limits beat: power per rack, liquid cooling, packaging yield. The framing test for whether an item belongs here: does it change the cost, supply, or speed of a FLOP? If yes, it's Hardware news; if it changes where the FLOP runs, it's Infrastructure.

Example source types (4–6):

1. Semiconductor analysis outlets — SemiAnalysis, TechInsights-style teardowns: die-level and supply-chain reporting well ahead of mainstream coverage.
2. Vendor primary sources, read adversarially — Nvidia GTC/earnings, AMD Advancing AI, TSMC and ASML earnings calls: the raw claims, tagged against later independent numbers.
3. Benchmark bodies — MLPerf training/inference rounds; peer-reviewed systems papers (ISCA/Hot Chips): where marketing meets measurement.
4. Financial/trade press on the capex cycle — earnings-season coverage of hyperscaler capex, HBM contract pricing, neocloud debt structures.
5. Video explainers (curated, embedded) — chip-architecture channels of the Asianometry class: fab economics, packaging, lithography, visualized properly.
6. Policy/export-control trackers — BIS rule changes, CSIS-style analysis: the regulatory layer that can reprice this entire stack in one announcement.