Articles AI of tomorrow
EN

Living notebook · Codex no. 2

Chips of tomorrow

What will the chips that train the next AIs look like?

An AI model is only as fast as the chip that feeds it. This notebook looks at the chips that train AI today, the ones being built right now, and our hypotheses for the next generations: more power, less energy, less heat. With diagrams, so you can see the chips from the inside.

Version 1 · updated

I

The chips that train AI today

A modern AI accelerator is not one chip but a small city: compute dies, memory stacks and wiring, all sitting on one package. Fig. 1 shows the layers; Fig. 2 compares the main chips on the figure that matters most, memory.

Fig. 1 — Anatomy of an AI accelerator

123456789
  1. Compute dies (each close to the reticle limit)
  2. HBM memory stacks, 8 to 12 layers of DRAM
  3. HBM base die: the controller at the bottom of each stack
  4. Silicon interposer with vertical connections (TSV)
  5. Package substrate that carries power
  6. Die-to-die link between the two compute dies
  7. Thermal paste and metal lid
  8. Cold plate with circulating liquid
  9. Solder balls to the circuit board
A cut through the package. The compute dies and the memory stacks sit side by side on a silicon interposer; everything is cooled from the top.

Fig. 2 — Memory bandwidth and capacity of the main chips

Memory bandwidth (TB/s)

  1. H1003.35
  2. B2008
  3. B3008
  4. MI355X8
  5. Trainium34.9
  6. TPU Ironwood7.37
  7. Maia 2007
  8. Rubin19.2 → 22
  9. MI455X23.3

Memory capacity (GB)

  1. H10080
  2. B200192
  3. B300288
  4. MI355X288
  5. Trainium3144
  6. TPU Ironwood192
  7. Maia 200216
  8. Rubin288
  9. MI455X432

Shipping2026 generationRubin’s first target (22 TB/s)

Bandwidth is how fast data reaches the compute; capacity is how much a chip can hold. The 2026 generation roughly doubles to triples bandwidth.
II

The chips being built

The 2026–2028 generation is already taped out or in production. Its pattern is clear: much more memory bandwidth, more dies per package, and racks that draw hundreds of kilowatts.

Fig. 3 — Chips, memory and processes, 2024–2029

202420252026202720282029

Nvidia

  • B200
  • B300
  • Rubin
  • Rubin Ultra
  • Feynman

AMD

  • MI355X
  • MI455X

Google

  • Ironwood
  • TPU 8t · 8i

AWS

  • Trainium3
  • Trainium4

Microsoft

  • Maia 200

TSMC

  • N2
  • A16
  • A14
  • A12

HBM

  • HBM3E
  • HBM4
  • HBM4E
  • HBM5
Nov. 2026

ShippingRampingPlanned

Dates for future items are announced or reported, and often move. Read it as an order of arrival, not a schedule.

Established

Nvidia Rubin (Vera Rubin NVL72)

Two dies on TSMC’s N3P process, 288 GB of HBM4 at up to 19.2 TB/s (22 TB/s was the first target; HBM4 pin speed was the sticking point), about 50 PFLOPS in NVFP4 inference, roughly 1,800 W. First production hardware arrived in autumn 2026.

Source (3)

Promising

Google TPU 8: two chips instead of one

Reported at Google Cloud Next 2026: the next TPU splits into 8t for training and 8i for inference. A first sign that one design cannot be the best at both jobs.

Source (1)
III

Four walls

Faster chips are no longer a matter of shrinking transistors. Four limits decide what the next generations can do. Fig. 4 shows the first one.

Fig. 4 — The scissors: compute pulls away from memory

×1×10×1000246810Years×243×10×5
  • Peak compute
  • Memory bandwidth

After ten years at these rates, compute has grown about 24 times more than memory bandwidth (×243 against ×10).

Growth per two years measured by Gholami and colleagues over two decades (compute ×3.0, memory bandwidth ×1.6, interconnect ×1.4), extended here for ten years. A trend, not a forecast.

Established

The memory wall

Over two decades, peak compute grew about 3× every two years, memory bandwidth about 1.6× and interconnect about 1.4×. Chips increasingly wait for data instead of calculating. This is why HBM now sits inside the package.

Source (1)
IV

New materials and transistors

The transistor itself is being redesigned: its shape, where power enters, and eventually what it is made of.

Promising

Nanosheets, then power from behind

Gate-all-around “nanosheet” transistors wrap the gate around the channel for better control and less leakage; TSMC’s N2 uses them. Next comes backside power (TSMC A16 “Super Power Rail”): power wires move under the transistors, freeing the top for signals. The schedule is reported to have slipped to 2027.

Source (2)
V

New architectures

Where the silicon sits matters as much as what it is. Fig. 5 contrasts the package of today with a stacked one.

Fig. 5 — The same chip, side by side or stacked

2.5D package (today)

1234

3D stack (tomorrow)

5678
  1. Compute die
  2. HBM stack
  3. Silicon interposer
  4. Long wires: millimetres, so more energy
  5. Hybrid-bonding pads at micrometre pitch
  6. Compute die stacked on top
  7. Very short vertical path
  8. Heat must cross the whole stack
Today the data crosses millimetres of interposer. Stacked, it crosses micrometres. The price is that heat must now cross the stack.

Promising

Light inside the package

Co-packaged optics (CPO) place optical engines next to the switch or the accelerator, replacing pluggable transceivers and long copper links. Nvidia claims up to 3.5× better power efficiency for its photonic switches. Optics for the links between GPUs inside a rack is the next step.

Source (3)
VI

Smarter software

The cheapest watt is the one never spent. Software gets more out of the same silicon, and Fig. 6 shows how much is at stake per watt.

Fig. 6 — Performance per watt at peak

FP8 (8-bit numbers)

  1. H1002.8
  2. B2004.5

FP4 (4-bit numbers)

  1. B2009.0
  2. B30010.7
  3. Rubin27.8
  4. Rubin Ultra27.8

TFLOPS / W — peak, denseAnnounced figures, not measured

Peak throughput divided by chip power. Halving the number format roughly doubles the figure; Rubin and Rubin Ultra use announced values, and the ratio between them is flat.

Established

Better kernels

FlashAttention-3 reorganises attention so data stays in on-chip memory: about 75% utilisation on an H100 in FP16, 1.5–2× faster than version 2. Same mathematics, fewer trips to HBM.

Source (1)
VII

Heat and power

Every watt a chip uses becomes heat, and every watt of heat must travel out. Fig. 7 shows how power per chip and per rack is climbing; Fig. 8 shows two ways to get the heat out.

Fig. 7 — Power keeps climbing

Power per chip (W)

  1. H100700 W
  2. B2001000 W
  3. B3001400 W
  4. Rubin≈ 1800 W
  5. Rubin Ultra≈ 3600 W

Power per rack (kW)

  1. GB200 / GB300 NVL72≈ 120–130 kW
  2. Vera Rubin NVL72≈ 190–230 kW
  3. Rubin Ultra NVL576≈ 600 kW

Announced or estimated

Each generation draws more. Rubin and Rubin Ultra values are announced or estimated.

Fig. 8 — Two ways to take the heat out

Cold plate (today)

12345

Microfluidic (prototype)

1567
  1. Hot spot in the die
  2. Thermal paste
  3. Metal lid
  4. Cold plate
  5. Coolant
  6. Channels etched in silicon
  7. Liquid a few tens of micrometres from the transistors
A cold plate pulls heat through several layers. Microfluidics puts the liquid next to the transistors. Microsoft measured up to 3 times better heat removal in the lab.
VIII

Our hypotheses

What follows is our own reasoning, not a published result. The confidence figure says how much we believe it today; it will move.

H1

Memory bandwidth stays the main limit

Through 2030, every generation will be defined by memory first: more stacks, faster stacks, closer to the compute. Peak FLOPS will keep growing faster than the data that feeds it.

Confidence80 %
For
The measured trend since 2000; HBM4 and HBM4E roadmaps; MI455X and Rubin spend most of their novelty on memory.
Against
Smaller number formats and sparsity reduce bytes per operation, which could ease the pressure for a while.
Source (2)

H2

Hybrid bonding becomes mainstream by 2029

3D stacking with copper-to-copper bonding will move from a few products to most high-end packages: first in 16-high HBM, then logic on cache or memory.

Confidence55 %
For
Already announced in HBM roadmaps; it shortens links and saves energy.
Against
Heat trapped between layers, yield of bonded stacks and cost are unsolved at volume.
Source (2)

H3

Optics enter the package, then the rack

Optical links will replace copper first between racks, then between GPUs inside a rack, around the turn of the decade.

Confidence55 %
For
Nvidia is shipping CPO switches; copper reach shrinks as speeds grow; Google already uses optical switching in TPU pods.
Against
Reliability, serviceability and laser cost. Copper keeps surprising by lasting longer.
Source (2)

H4

FP4 becomes the default format

Most large training runs and nearly all inference will use 4-bit formats within three years, with a few sensitive layers kept at higher precision.

Confidence55 %
For
NVFP4 pretraining at 10 trillion tokens; hardware support in Blackwell, Rubin and MI355X.
Against
Accuracy risks on the hardest tasks, and models that reach FP4 quality only with careful recipes.
Source (1)

H5

In-chip liquid cooling reaches products by 2030

A first commercial accelerator will use microchannels etched into or directly under the silicon.

Confidence40 %
For
A 3× lab gain; the 3,600 W Rubin Ultra class is near what cold plates can do.
Against
Leaks, reliability and a new supply chain; plates and immersion may stretch further than expected.
Source (2)

H6

Efficiency will come more from design and software than from the process

Each new process node still helps, but most of the gain per watt through 2030 will come from formats, memory placement, scheduling and specialised chips.

Confidence70 %
For
Peak FP4 per watt is about flat between Rubin and Rubin Ultra by our arithmetic (Fig. 6); Perseus-type savings need no new silicon.
Against
A real breakthrough in transistors (backside power at scale) would change the balance.
Source (3)

H7

A 2D-material transistor in a commercial AI chip before 2031

We think this is unlikely on this horizon: the lab results are real, but manufacturing at AI-chip scale is another matter.

Confidence5 %
For
50 nm-pitch complementary 2D transistors on 300 mm wafers in 2026.
Against
Contacts, defects and yield; the industry still has CFET and backside power to deploy first.
Source (2)
IX

Provisional conclusions

What we think at this stage. “Provisional” is the key word.

  1. The next gain is in the memory, not the compute

    Everything points the same way: more HBM, custom base dies, stacking, and in-package optics. A chip is now judged on how fast it can be fed.

    Source (2)
  2. Watts will be saved in three places at once

    Fewer bits (FP4), less data movement (stacking, optics) and smarter scheduling add up. No single trick is enough.

    Source (3)
  3. Heat and power delivery are now design problems

    At 600 kW per rack, cooling and 800 V distribution decide what can be built, as much as transistors.

    Source (2)
  4. The field is diversifying

    Rubin, MI455X, TPU 8, Trainium4, Maia 200: training and inference chips are drifting apart, and the big clouds design their own.

    Source (2)
X

Open questions

Next leads

XI

Lexicon

The words of this notebook, in plain language.

TDP
Thermal design power: the heat, in watts, that the cooling system must be able to remove from a chip.
HBM
High-bandwidth memory: layers of DRAM stacked vertically and placed right beside the chip, joined by thousands of short wires.
Interposer
A thin slab of silicon under the chips that carries the fine wiring between compute dies and memory (TSMC’s version is called CoWoS).
Chiplet
A small die that is one piece of a larger processor. Several chiplets are assembled in one package.
Reticle limit
The largest area a lithography machine can print in one exposure, about 850 mm². It caps the size of a single die.
Hybrid bonding
Joining two dies by bonding their copper pads directly, without solder bumps. The connections are much smaller and closer together.
FP4
A floating-point number written with only 4 bits. Less precise, but it halves the data moved and speeds up calculation.
Gate-all-around
A transistor whose gate surrounds the channel on all sides, which gives better control than the older fin shape.
Backside power
Delivering power from the underside of the chip, so the top side is free for signal wires.
Co-packaged optics
Optical transceivers built into the same package as the chip, so signals leave as light after a very short electrical path.
Thermal interface material
The thermal paste or pad between the chip and its lid or cold plate. It fills tiny gaps so heat can pass.

Notebook log

  1. Notebook created: today’s chips, the 2026–28 generation, four walls, materials, architectures, software, heat; eight diagrams and seven hypotheses.

AI of tomorrow →

Sources (45)

Back to top