Deep dive · Folio IV
Since the 1940s, computers have been built in two parts: memory on one side, the calculator on the other, and a corridor in between. For AI, which reads billions of weights for every word, that corridor has become the main cost.
Processors have become enormously faster, memory much less so, and moving data costs ever more compared with computing. A model with 70 billion weights reads them all for every word it writes: the calculator waits, and the energy goes into the corridor.
The idea: stop moving the weights. Put them where the calculation happens, or make the memory itself compute.
Three families. Digital near-memory: many small cores, each with its own memory right next to it, and no external memory. Analog in-memory: the weight is stored as an electrical conductance and Ohm's law does the multiplication: apply a voltage, read a current. Photonic: light crosses a grid of optical components that performs a matrix product on its way.
In all three cases the gain comes from the same place, fewer trips, and so does the difficulty: memory on a chip is limited, analog signals are noisy, and these machines must fit into a world of digital software.
Source: Horowitz, Computing's Energy Problem (ISSCC 2014)
Fig. J — The corridor, or no corridor
Classic computer
Each weight makes the trip: about 1,300 to 2,600 pJ to read 64 bits from main memory, for about 4 pJ of calculation.
Compute-in-memory
Each memory cell computes where its weight is stored: the corridor disappears.
Illustrative diagram: it shows the idea, not measurements.
Promising
22 billion transistors, no external memory. On image recognition (ResNet-50), against a GPU made with the same 12 nm process: 25 times more energy-efficient, 22 times lower latency, 5 times more output per transistor; still 5 times more efficient than a 4 nm H100. Digital, so free of analog noise. Its limit: everything must fit in the chip's memory.
Speculative
35 million phase-change memory cells on 34 tiles, in 14 nm. On speech recognition, accuracy close to digital and more than 14 times the energy efficiency, up to 12.4 trillion operations per second per watt. Still a research chip: no digital cores on board, some steps done outside.
Source: Ambrogio et al., An analog-AI chip for speech recognition, Nature (2023)
Speculative
Six chips stacked in 3D, 50 billion transistors and a million photonic components: 65.5 trillion operations per second for 78 watts. For the first time, a light-based processor ran unmodified models (ResNet, BERT, an Atari player) with accuracy close to 32-bit electronics.
Source: Ahmed et al., Universal photonic AI acceleration, Nature (2025)
Capacity: a large language model does not fit in the memory of a single NorthPole; it must be split across many chips, and the corridor returns between them. Precision: analog and optical computing drift with temperature and age. Ecosystem: twenty years of software were written for GPUs. That is why this lead is “speculative”: the physics works, the industry is not following yet.
Fourth factor of H6: the energy per bit moved. And the clearest illustration of the new conclusion: the real cost is moving data. A hierarchy of data is the software version of the same idea: keep what is used often close to the calculation.
We cannot build such a chip, but we can measure the wall: on an ordinary computer, time a small model when its weights fit in the processor's cache and when they do not. The jump should show, at small scale, the barrier these chips try to remove.