Deep dive · Folio IV
Each number in a model is stored with some precision, like a ruler marked in millimetres or in centimetres. For years models used 16 or 32 bits per number. It turns out they can often make do with 8, 4, or even fewer than 2.
A model's weights are numbers. Storing them with fewer bits is like rounding: a little detail is lost, but each number takes less space, is read faster and is multiplied by smaller circuits.
The real gain is less in the calculation than in the movement: as the energy figure on the main page shows, reading a number from memory costs hundreds of times more than computing with it. Halving the bits halves that bill.
The trick is scaling by blocks: numbers are grouped in small packets that share a common scale, so a few bits are enough to describe each one relative to its neighbours. The most sensitive parts (first and last layers, sums) stay in higher precision.
There are two routes: compress after training (quick, but quality falls at very low precision), or train directly at low precision, so the model learns to live with rounding from the start.
Source: Horowitz, Computing's Energy Problem (ISSCC 2014)
Fig. I — Same model, fewer bits
Memory needed to store 1 billion weights
With three values, multiplication disappears:
+1add the number0skip it−1subtract it
Illustrative diagram: it shows the idea, not measurements.
Established
First validation of 8-bit training on a giant model: most of DeepSeek-V3's matrix calculations are done in FP8, with fine-grained scaling. The training loss stays within 0.25 % of the 16-bit reference. It is one reason for its low training cost.
Promising
NVIDIA trained a 12-billion-parameter model on 10,000 billion words in 4 bits, the longest such run published. On MMLU-Pro: 62.58 % against 62.62 % in 8 bits. On recent chips, 4 bits computes 2 to 3 times faster than 8 bits with half the memory. About 16 % of layers stay in higher precision, and code is slightly weaker.
Source: NVIDIA, Pretraining Large Language Models with NVFP4 (2025)
Promising
Each weight is −1, 0 or +1. From 3 billion parameters, it matches a 16-bit model trained on the same data, while being 2.71 times faster and using 3.55 times less memory; at 70 billion, 4.1 times faster. Estimated arithmetic energy for matrix products: 71 times less.
Source: Ma et al., The Era of 1-bit LLMs: BitNet b1.58 (2024)
No shortcut for existing models: ternary weights must be learned from the start; rounding an existing model this hard breaks it. Hardware lags: BitNet's gains are only fully real on chips designed for it. Fragile tasks: code and precise calculation suffer first. And the more data a model has seen, the more fine detail its weights carry, which leaves less room for rounding.
Third factor of H6: the bits per weight. The parallel with our notebook is striking: a ternary model keeps only the direction of each connection (for, against, neutral), as if fine details did not belong in the weights. That is exactly the split we propose: coarse principles in the model, precise details in an external memory.
Take one open model in 16, 8 and 4 bits and ask the three versions the same questions about OrigoTech articles. Our guess: facts (dates, figures) degrade before reasoning does. If so, it is an argument for storing facts outside the weights.