Floating Point

2026-08-10T00:00:00+00:00
Run this anywhere:</p>
>>></span> 0.1</span> +</span> 0.2</span></span>
0.30000000000000004</span></span>
>>></span> 0.1</span> +</span> 0.2</span> ==</span> 0.3</span></span>
False</span></span></code></pre>
And in real training runs, this happens:</p>
step 39,998   loss 2.41</span></span>
step 39,999   loss 2.40</span></span>
step 40,000   loss nan</span></span></code></pre>
Both outputs have the same cause: the way computers store numbers.
By the end of this page you will be able to predict both, explain
them to someone else, and read a line like "MXFP4, E2M1, block 32,
E8M0 scale" part by part. No prior knowledge is assumed. Every measured number on this page comes
from a small script you can run, linked where the number appears,
measured on an Apple M3 Max with torch 2.11.0 [1</a>]</span>.</p>
A fixed set of points</a></h2>
A computer stores a number in a fixed number of bits. Each bit
has 2 states, so 32 bits can form 2 x 2 x ... x 2 = 2^32 =
4,294,967,296 different patterns. A number format assigns one
number to each pattern. So for a 32-bit machine only those
4,294,967,296 numbers exist; anything else you compute is moved
to the nearest one of them, and the distance moved is the
rounding error. A format is exactly this: a choice of which
numbers get a pattern. This page is about how that choice is
made.</p>
First, see the size of the problem. Between 1 and 2 there are
infinitely many numbers. Zoom in anywhere and you find more.
The line is never empty, at any depth. A format is a finite
list of points. So almost no computed number is on the list,
and almost every operation ends with a move to the nearest
point:</p>

  
    
    
  </div>
  Figure 1.</span> infinitely many numbers on the line, 8 points in the format. every number off the list moves to the nearest point.</figcaption>
</figure>
The simplest choice is equal spacing: 0, 1, 2, 3, and so on, the
same distance 1 between every pair of neighbors. That is what
integers are, and for counting it works. Now take two numbers
that one real neural network produced in one training step: an
activation of 10.5 (a value flowing through the network) and a
gradient of 0.0000004 (a value that steers learning). Store both
with equally spaced points:</p>

  
    
    
  </div>
  Figure 2.</span> equally spaced points meet 2 real numbers. 10.5 keeps a 4.8% error; 0.0000004 becomes 0, a 100% error.</figcaption>
</figure>
Those 2 numbers are not invented. They come from 1 training step
of a small network, and this page keeps returning to that step,
so here it is, drawn:</p>

  
    
    
  </div>
  Figure 3.</span> one training step of the measured network, drawn as the network. the bottom axis carries the step's 2 flows on one scale: the backward numbers live far lower than the forward ones.</figcaption>
</figure>
3 kinds of numbers move in that step. Activations are the values
flowing forward through the layers. Weights are the numbers the
network is learning; multiplying by them is what a layer does.
Gradients are computed backward from the loss, 1 for every
weight: each one says how much that weight should move. Train
the network for a few seconds and measure all 3:</p>

  
    
    
  </div>
  Figure 4.</span> the measured ranges of one small network's numbers in one training step. thin line: full range. box: the middle 99.8%. thick mark: the median.</figcaption>
</figure>

  
    </span>
    p1_where_numbers_live.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: where a real network's numbers live. Train a small MLP for</span></span>
two seconds, then measure the spans of its weights, activations, and</span></span>
gradients in one step. The same model needs numbers nine orders of</span></span>
magnitude apart; a uniform grid cannot serve both ends.</span>"""</span></span>
import</span> torch</span>,</span> torch</span>.</span>nn</span> as</span> nn</span></span>
</span>
torch</span>.</span>manual_seed</span>(</span>0</span>)</span></span>
net</span> =</span> nn</span>.</span>Sequential</span>(</span>nn</span>.</span>Linear</span>(</span>64</span>,</span> 256</span>),</span> nn</span>.</span>ReLU</span>(),</span></span>
                    nn</span>.</span>Linear</span>(</span>256</span>,</span> 256</span>),</span> nn</span>.</span>ReLU</span>(),</span> nn</span>.</span>Linear</span>(</span>256</span>,</span> 1</span>))</span></span>
opt</span> =</span> torch</span>.</span>optim</span>.</span>Adam</span>(</span>net</span>.</span>parameters</span>(),</span> lr</span>=</span>1e-3</span>)</span></span>
X</span> =</span> torch</span>.</span>randn</span>(</span>4096</span>,</span> 64</span>)</span>; y</span> = (</span>X</span>[:, :</span>3</span>].</span>prod</span>(</span>1</span>,</span> keepdim</span>=</span>True</span>)</span> *</span> 10</span>)</span></span>
for</span> step</span> in</span> range</span>(</span>300</span>):</span></span>
    opt</span>.</span>zero_grad</span>()</span></span>
    loss</span> = ((</span>net</span>(</span>X</span>)</span> -</span> y</span>)</span> **</span> 2</span>).</span>mean</span>()</span></span>
    loss</span>.</span>backward</span>()</span>; opt</span>.</span>step</span>()</span></span>
</span>
acts</span> = {}</span></span>
h</span> =</span> X</span></span>
tensors</span> = {</span>"</span>weights</span>"</span>:</span> torch</span>.</span>cat</span>([</span>p</span>.</span>detach</span>().</span>flatten</span>()</span> for</span> p</span> in</span> net</span>.</span>parameters</span>()]),</span></span>
           "</span>grads</span>"</span>:</span> torch</span>.</span>cat</span>([</span>p</span>.</span>grad</span>.</span>flatten</span>()</span> for</span> p</span> in</span> net</span>.</span>parameters</span>()])}</span></span>
with</span> torch</span>.</span>no_grad</span>():</span></span>
    a</span> =</span> net</span>[</span>1</span>](</span>net</span>[</span>0</span>](</span>X</span>))</span></span>
    tensors</span>[</span>"</span>activations</span>"</span>] =</span> net</span>[</span>2</span>](</span>a</span>).</span>flatten</span>()</span></span>
</span>
print</span>(</span>f</span>"final loss </span>{</span>loss</span>.</span>item</span>()</span>:.3f</span>}</span>"</span>)</span></span>
for</span> name</span>,</span> t</span> in</span> tensors</span>.</span>items</span>():</span></span>
    nz</span> =</span> t</span>[</span>t</span> !=</span> 0</span>].</span>abs</span>()</span></span>
    q</span> =</span> torch</span>.</span>quantile</span>(</span>nz</span>,</span> torch</span>.</span>tensor</span>([</span>0.001</span>,</span> 0.5</span>,</span> 0.999</span>]))</span></span>
    print</span>(</span>f</span>"</span>{</span>name</span>:12s</span>}</span> |min..max| </span>{</span>nz</span>.</span>min</span>()</span>:.2e</span>}</span> .. </span>{</span>nz</span>.</span>max</span>()</span>:.2e</span>}</span>   "</span></span>
          f</span>"p0.1/median/p99.9  </span>{</span>q</span>[</span>0</span>]</span>:.2e</span>}</span> / </span>{</span>q</span>[</span>1</span>]</span>:.2e</span>}</span> / </span>{</span>q</span>[</span>2</span>]</span>:.2e</span>}</span>"</span>)</span></span>
span</span> =</span> tensors</span>[</span>"</span>grads</span>"</span>].</span>abs</span>()[</span>tensors</span>[</span>"</span>grads</span>"</span>]</span> !=</span> 0</span>]</span></span>
import</span> math</span></span>
print</span>(</span>f</span>"gradient span: </span>{</span>math</span>.</span>log2</span>(</span>span</span>.</span>max</span>()</span> /</span> span</span>.</span>min</span>())</span>:.1f</span>}</span> powers of two "</span></span>
      f</span>"in one tensor family, this small a net, this early"</span>)</span></span>
</span>
print</span>(</span>"</span>\n</span>decade histogram (percent of values per log10 decade; </span>"</span></span>
      "</span>the drawn silhouettes):</span>"</span>)</span></span>
edges</span> =</span> list</span>(</span>range</span>(</span>-</span>8</span>,</span> 2</span>))</span></span>
for</span> name</span>,</span> t</span> in</span> tensors</span>.</span>items</span>():</span></span>
    nz</span> =</span> t</span>[</span>t</span> !=</span> 0</span>].</span>abs</span>().</span>log10</span>()</span></span>
    counts</span> = [(</span>100.0</span> *</span> ((</span>nz</span> >=</span> a</span>)</span> &</span> (</span>nz</span> <</span> a</span> +</span> 1</span>)).</span>sum</span>()</span> /</span> nz</span>.</span>numel</span>())</span></span>
              for</span> a</span> in</span> edges</span>]</span></span>
    row</span> =</span> "  "</span>.</span>join</span>(</span>f</span>"</span>{</span>c</span>:4.1f</span>}</span>"</span> for</span> c</span> in</span> counts</span>)</span></span>
    print</span>(</span>f</span>"</span>{</span>name</span>:12s</span>}</span> [</span>{</span>edges</span>[</span>0</span>]</span>}</span>..</span>{</span>edges</span>[</span>-</span>1</span>]</span> +</span> 1</span>}</span>]  </span>{</span>row</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Read the two computed lines in the drawing. Why divide? The
ratio says how many of the small number fit into the big one.
With equal spacing, that is the number of steps between them.
The gradients alone run from 0.00000037 to 0.71, and
0.71 / 0.00000037 is about 1,900,000, which is 2^20.9. So if
the spacing is fine enough to keep the smallest gradient, the
largest one sits 1,900,000 steps away. Across kinds it is
worse: 10.5 / 0.00000037 = 28,000,000 steps. A larger model
makes both numbers bigger. Equal spacing cannot hold both
ends. Set the spacing to 0.0000004 so the smallest gradient
survives, and even 2^31 = 2,147,483,648 points in the positive
direction reach only 0.0000004 x 2,147,483,648 = 859: the format
ends before 1,000. Set the spacing to 1 so that large values
fit, and every gradient becomes 0, as the first drawing showed.
The spacing itself has to change with the size of the number:
points packed close near zero, spread out far from it.</p>
Floating the point</a></h2>
One small tool first: what a dot means in base two, because from
here on every number is bits. In decimal, each place is worth a
tenth of the place before it: the 7 in 204.75 means seven tenths.
Binary uses the same rule with 2 in place of 10. Left of the dot
the places are worth 8, 4, 2, 1; right of it they are worth a
half, a quarter, an eighth:</p>

  
    
    
  </div>
  Figure 5.</span> the dot in base two. same reading you have done since childhood, with each place worth half the one before instead of a tenth.</figcaption>
</figure>
Read 0010.1100 with those place values: the 1s sit on 2, on a
half and on a quarter, and 2 + 0.5 + 0.25 = 2.75. That is all a
string of bits with a dot in it can mean, and you can now read
any of them.</p>
One question hides here, and it decides half of this page:
which fractions can these places write exactly? The places are
halves, quarters, eighths. Any finite sum of them is a fraction
whose denominator is a power of 2: 1/2, 3/4, 5/8. Those end
cleanly. Now take 1/3, 1/5 or 1/10. Their denominators carry a
3 or a 5. No finite sum of halves, quarters and eighths equals
them. Watch both cases run:</p>

  
    
    
  </div>
  Figure 6.</span> building fractions from halves, quarters, eighths. 5/8 is hit exactly in 3 bits. 1/3 leaves a gap after every row, forever: its bits repeat. 1/10 carries a 5 and repeats the same way.</figcaption>
</figure>
Now a first attempt that fails. Take 1 byte, which is 8 bits,
and fix the dot in the middle: 4 bits for the whole part, 4 for
the fraction:</p>

  
    
    
  </div>
  Figure 7.</span> fixed point: the same 8 bits, 3 dot positions. each choice reaches the same 255 steps; moving the dot slides the reach, never widens it, and no position covers both target numbers.</figcaption>
</figure>
This is called fixed point, and the drawing shows its problem:
the dot's position decides the largest value and the smallest
step at the same time, and no single position makes both good. The
repair is in the name of this page. Let the dot move, and store
where it went.</p>
Where should the machine write down "where the dot went"? You
already know the answer, because you already do this on paper.
To write a very small or a very large number, you split it into
its digits and its size: 0.00000037 becomes 3.7 x 10^-7. The 3.7
says what the digits are; the 10^-7 says how big the number is.
2 separate jobs, written separately:</p>

  
    
    
  </div>
  Figure 8.</span> the heart of the format: the digits-and-size split you already write, moved to base 2, one bit field per half.</figcaption>
</figure>
A float makes exactly this split, in base 2. Any positive number
is some power of 2 times a number between 1 and 2: 10.5 is
1.3125 x 2^3 (check it: 2^3 = 8, and 1.3125 x 8 = 10.5), and
0.00000037 is about 1.552 x 2^-22. The power of 2 goes into one
field, named the exponent</strong>. The 1.something goes into the
other field, named the mantissa</strong>. The 2 words are only names
for the 2 halves of the split: the exponent is the number's
size, the mantissa is its digits.</p>
What does each field buy on the number line? Draw them [2</a>]</span>:</p>

  
    
    
  </div>
  Figure 9.</span> what each field does. the exponent counts doubling intervals, drawn at true scale so you can see each one really is twice as wide; the mantissa cuts the chosen interval into 2^M equal steps.</figcaption>
</figure>
Where do the intervals come from? From the split itself. Fix
the power at 8. The digits part runs from 1.00 to 1.99. So the
values run from 8 to just under 16: every number with power 8
lives between 8 and 16. Fix the power at 16 and you fill 16 to
32. That stretch starts twice as high, so it is twice as wide.
That is all the doubling is. Choosing the exponent is choosing
the interval. 8 exponent bits count 2^8 = 256
intervals, and because every interval doubles, 256 of them span
an enormous distance: that is how 8 bits reach from 1e-38 to
3e38. The mantissa then cuts the chosen interval into equal
steps: 3 bits make 2 x 2 x 2 = 8 steps, binary32's 23 bits make
8,388,608. And the growing step we asked for at the start falls
out by itself: [4, 8) is twice as wide as [2, 4), both are cut
into the same number of steps, so every step in [4, 8) is twice
as long. Small numbers get fine steps, large numbers get coarse
ones, with nobody managing it.</p>
Now store one number end to end. Move 1: pick the interval.
6.1 = 1.525 x 4, and 1.525 is between 1 and 2, so the interval
is [4, 8). Move 2: say where inside. 6.1 sits 2.1 past 4, and
2.1 / 4 = 0.525. That fraction, 0.525, is what the mantissa
holds:</p>

  
    
    
  </div>
  Figure 10.</span> storing 6.1 is 2 moves: pick the interval, then cut it into equal steps and take the nearest. the fraction 0.525 is what the mantissa holds.</figcaption>
</figure>
In a real 32-bit float the split gets 1 bit for the sign, 8 for
the exponent, 23 for the mantissa. Here is 6.1, every field
decoded:</p>

  
    
    
  </div>
  Figure 11.</span> binary32, decoded by hand: the actual bits of 6.1, the hidden 1 as a ghost cell, and the bias on the exponent's path.</figcaption>
</figure>
Two details in that card deserve their own sentences, because
every format on this page inherits both. First, the decode row
says "1 + 0.525", but the mantissa field only stored the .525.
The leading 1 costs nothing: every number in a power-of-two
interval is 1.something times a power of two, so the format does
not store the 1 and gets 24 bits of position for the price of 23.
(The subnormals, coming 2 sections from now, are 0.something
and are the one exception.) Second, the bias</strong>. The exponent field is unsigned: 8 bits,
0 to 255, no minus sign. But interval numbers need minus
signs: 0.00000037 lives at interval -22. The fix is one
constant, 127, the middle of the range. When the machine
stores, it adds 127: the -22 above went in as -22 + 127 = 105.
When it reads, it subtracts 127: this card's 129 means
129 - 127 = 2. Same 127, two directions. Every interval number
from -126 to 127 now fits in 1 to 254. (Stored 0 and 255 are
reserved; they build the edge cases 2 sections ahead.)</p>

  
    
    
  </div>
  Figure 12.</span> the bias, both directions: add 127 when storing, subtract 127 when reading. why 127: it is the middle of what 8 unsigned bits can hold.</figcaption>
</figure>
The unsigned choice has a quiet payoff. A bigger float always
carries a bigger bit pattern, so a chip can compare 2 floats
with the integer circuits it already owns [3</a>]</span>.</p>
Notice what the last decode row admits: the machine does not
store 6.1. It cannot. 6.1 is 61/10, and the 10 carries a 5.
The fractions figure showed what a 5 does in base 2: the bits
repeat forever. The mantissa keeps 23 of them and cuts the
rest. What remains decodes to 6.0999999..., the nearest
float32. The gap is the rounding error. The next three
sections are about the size and the consequences of that gap. This layout, the
bias and the field widths, was standardized in 1985 as IEEE 754 [3</a>]</span>, and it is the
format your float32</code> tensors use today, unchanged.</p>
The map</a></h2>
You cannot draw all 4,294,967,296 points of float32. But the
same design at 8 bits has only 2^8 = 256 patterns, and its
entire positive half fits in one picture. The 8-bit format is
called E4M3, and the name is just its recipe: E4 means 4
exponent bits, M3 means 3 mantissa bits. Every format name on
this page reads the same way. This is the most important drawing on
this page, because every format you will ever meet is this drawing
with different counts:</p>

  
    
    
  </div>
  Figure 13.</span> every positive number an FP8 E4M3 can be. each power-of-two interval holds the same 8 points; the step doubles from interval to interval, forever.</figcaption>
</figure>

  
    </span>
    p2_e4m3_map.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: every FP8 E4M3 value, decoded from its bits by hand, and</span></span>
checked bit-for-bit against torch.float8_e4m3fn. 256 codes: this is</span></span>
the entire format, and the decoder is fifteen lines.</span>"""</span></span>
import</span> torch</span></span>
</span>
def</span> decode_e4m3</span>(</span>byte</span>):</span></span>
    s</span> = (</span>byte</span> >></span> 7</span>)</span> &</span> 1</span></span>
    e</span> = (</span>byte</span> >></span> 3</span>)</span> & 0x</span>F</span></span>
    m</span> =</span> byte</span> & 0x</span>7</span></span>
    if</span> e</span> == 0x</span>F</span> and</span> m</span> == 0x</span>7</span>:</span></span>
        return</span> float</span>(</span>"</span>nan</span>"</span>)</span>            # the single NaN code per sign</span></span>
    if</span> e</span> ==</span> 0</span>:</span>                          # subnormal: no hidden 1</span></span>
        return</span> (</span>-</span>1</span>)</span>**</span>s</span> *</span> 2</span>**</span>(</span>1</span> -</span> 7</span>)</span> *</span> (</span>m</span> /</span> 8</span>)</span></span>
    return</span> (</span>-</span>1</span>)</span>**</span>s</span> *</span> 2</span>**</span>(</span>e</span> -</span> 7</span>)</span> *</span> (</span>1</span> +</span> m</span> /</span> 8</span>)</span></span>
</span>
agree</span>,</span> values</span> =</span> 0</span>, []</span></span>
for</span> byte</span> in</span> range</span>(</span>256</span>):</span></span>
    mine</span> =</span> decode_e4m3</span>(</span>byte</span>)</span></span>
    ref</span> =</span> torch</span>.</span>tensor</span>([</span>byte</span>],</span> dtype</span>=</span>torch</span>.</span>uint8</span>).</span>view</span>(</span>torch</span>.</span>float8_e4m3fn</span>).</span>float</span>().</span>item</span>()</span></span>
    ok</span> = (</span>mine</span> !=</span> mine</span> and</span> ref</span> !=</span> ref</span>)</span> or</span> mine</span> ==</span> ref</span></span>
    agree</span> +=</span> ok</span></span>
    if</span> not</span> ok</span>:</span></span>
        print</span>(</span>f</span>"DISAGREE byte </span>{</span>byte</span>:08b</span>}</span>: mine </span>{</span>mine</span>}</span>, torch </span>{</span>ref</span>}</span>"</span>)</span></span>
    values</span>.</span>append</span>(</span>mine</span>)</span></span>
print</span>(</span>f</span>"</span>{</span>agree</span>}</span>/256 bit patterns agree with torch.float8_e4m3fn"</span>)</span></span>
</span>
finite</span> =</span> sorted</span>(</span>v</span> for</span> v</span> in</span> values</span> if</span> v</span> ==</span> v</span>)</span></span>
pos</span> = [</span>v</span> for</span> v</span> in</span> finite</span> if</span> v</span> ></span> 0</span>]</span></span>
print</span>(</span>f</span>"largest value          : </span>{</span>max</span>(</span>finite</span>)</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"smallest positive      : </span>{</span>min</span>(</span>pos</span>)</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"positive values        : </span>{</span>len</span>(</span>pos</span>)</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"step just above 1.0    : </span>{</span>min</span>(</span>v</span> for</span> v</span> in</span> pos</span> if</span> v</span> ></span> 1.0</span>)</span> -</span> 1.0</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"step just below 448    : </span>{</span>448</span> -</span> max</span>(</span>v</span> for</span> v</span> in</span> pos</span> if</span> v</span> <</span> 448</span>)</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Read the map slowly; every later section builds on it. Every
interval holds exactly 8 points, one per mantissa pattern. Neighbors
inside a interval are 1/8 of the interval's starting value apart, so
rounding moves a number by at most half of that: 1/16 of its own
size, about 6%, and this is the same in every interval, from 0.02 to
400. The step inside [1, 2) is 0.125; the step inside [256, 448]
is 32. So the absolute error grows with the number, but the
relative error does not. Every float format works this way.</p>
The local step also has a proper name, and you will meet it in
every numerics discussion: the ulp</strong>, the unit in the last
place [4</a>]</span>. "The ulp at x" means the step between neighboring
points where x lives. E4M3's ulp at 1.0 is 0.125; its ulp at 300
is 32. One number, the ulp at 1.0, summarizes a format's whole
precision, because every other interval's step is that number times a
power of two; the field guide later on this page prints it on
every format's card.</p>
The proof under the map is 15 lines of Python that decode all
256 bit patterns from first principles, and torch agrees with
every single one. The largest value is 448. The smallest
positive value is 0.001953125. Divide them: 448 / 0.001953125 =
229,376, about 2^17.8, so the top of this format is about 18
doublings above the bottom. Now compare that with the second
drawing: one tensor's gradients spanned 20.9 doublings. The
whole format is narrower than one tensor's spread, and that gap
runs the entire training half of this page.</p>
The edges of the map</a></h2>
Four places on the map need special handling, and every format
must decide all four.</p>
Zero.</strong> The position field always means "1.something", so no
pattern naturally means zero. The all-zeros pattern is simply
assigned the value zero, as a special case. The sign bit still exists, so there is
a +0 and a -0, and they compare equal.</p>
The ramp.</strong> Just above zero there is trouble. Every normal
point is 1.something x a power of 2. The exponent has a lowest
power, so there is a smallest normal point: in E4M3 it is
1.0 x 2^-6 = 0.015625. Below it, a naive format has nothing
until 0. A cliff. The fix: in the lowest interval only, drop
the hidden 1. Read the mantissa as 0.something instead of
1.something. Then 0.000 x 2^-6 is 0 itself, and the other 7
patterns walk up from 0 in the same equal steps. These points
are the subnormals</strong>. They turn the cliff into a ramp:</p>

  
    
    
  </div>
  Figure 14.</span> the gap at zero, empty and then filled. gradual underflow means the difference of two unequal numbers always has somewhere to land.</figcaption>
</figure>
Infinity.</strong> When a result overflows the ceiling, float32 returns
a dedicated pattern called inf</code>, which then behaves lawfully:
anything finite divided by inf</code> is 0, and inf - inf</code> is the next
special value. E4M3 made a harder choice: it has no infinity at
all. Those patterns were used for one more doubling of range,
and overflow stops at 448 instead. Keep that choice in mind; it
is the first sign of how little room 8 bits leave.</p>
NaN.</strong> Not a Number is the pattern returned when no answer is
defensible: 0/0, inf - inf, the square root of -1. It has one
deliberately strange law: nan == nan</code> is false, so x != x</code> is
the honest test for it. When your loss prints nan</code>, this value
is what you are looking at, and it arrived through an overflow or
an undefined step somewhere upstream. We will catch it in the act
in a few sections.</p>
Rounding</a></h2>
Between any 2 neighboring points of the map lies everything the format cannot
say. When a result lands there, one rule decides its fate: go to
the nearer point. When it lands exactly halfway, go to the point
whose last mantissa bit is 0, so that ties break upward and
downward equally often and a long chain of them does not drift.
This pair of rules is round-to-nearest, ties-to-even, the default
of every machine you will touch [3</a>]</span>:</p>

  
    
    
  </div>
  Figure 15.</span> between two points, the nearer one wins; a tie goes to the even one. the whole error model of numerical computing is this picture.</figcaption>
</figure>
The worst this rule can do is move a value by half the local
step: half an ulp. For float32 the ulp at 1.0 is 2 to the power -23, which is
about 1.2e-7 (read e-7 as: move the decimal point 7 places left,
so 0.00000012), and that number is called the machine epsilon
(torch.finfo(torch.float32).eps</code> prints it). So every operation lands
within about 6e-8 of the true result, measured relative to the
result's size. Both numbers read straight
off the map: find the interval that starts at 1, take its step, halve
it. The standard error analysis of floating point is this one
bound, applied once per operation [4</a>]</span>.</p>
How can hardware apply the rule without computing every bit of
the exact answer first? It keeps 3 extra bits, and that is all
it ever needs:</p>

  
    
    
  </div>
  Figure 16.</span> guard, round, sticky: the whole rounding rule runs on 3 extra bits, and the result is exactly as if the machine had kept all 48.</figcaption>
</figure>
This explains the first code block at the top of the page, and
you can check every step of it by hand.
To write 0.1 in binary you double it, again and again, and each
time the whole part of the result is the next bit. 0.1 doubles to
0.2: the first bit is 0. Then 0.4, 0.8: two more 0s. Then 0.8
doubles to 1.6: the first 1, and the 0.6 carries on. 0.6 doubles
to 1.2: another 1, and 0.2 carries on. But 0.2 is where the
second step started. The process is in a loop, and the bits
repeat forever:</p>

  
    
    
  </div>
  Figure 17.</span> 0.1 converted by hand. 5 doublings and the state returns to 0.2: the tail 0011 repeats forever, and no finite mantissa can hold it.</figcaption>
</figure>
1/3 does the same thing in decimal, and for the same reason: the
denominator has a prime factor the base does not. Ten is 2 times
5, so 0.1 is 1/(2 times 5), and base two has no 5. The format
cuts the repetition at its mantissa width and stores the nearest
representable number, which for float64 is exactly
0.1000000000000000055511151231257827... (proof</a>
prints every digit). The same happens to 0.2 and to 0.3, and the
sum of the two stored numbers is not the stored number nearest
0.3.
Nothing malfunctioned. 3 numbers you typed do not exist,
and the printed digits show exactly which nearby numbers were
stored instead.</p>

  
    </span>
    p0_riddles.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the two riddles. 0.1 + 0.2 is not 0.3 because 0.1 and 0.2</span></span>
do not exist in binary; the sum of the two nearest citizens is not</span></span>
the nearest citizen to 0.3. And at 1e8, float32's local step is 8,</span></span>
so adding 1 moves nothing.</span>"""</span></span>
from</span> fractions</span> import</span> Fraction</span></span>
import</span> struct</span>,</span> torch</span></span>
</span>
# the exact value a float64 stores for 0.1</span></span>
exact</span> =</span> Fraction</span>(</span>struct</span>.</span>unpack</span>(</span>"</span><q</span>"</span>,</span> struct</span>.</span>pack</span>(</span>"</span><d</span>"</span>,</span> 0.1</span>))[</span>0</span>]</span> &</span> ((</span>1</span><<</span>52</span>)</span>-</span>1</span>)</span> |</span> (</span>1</span><<</span>52</span>),</span> 1</span>)</span></span>
# simpler and fully honest: print the stored values to 30 digits</span></span>
print</span>(</span>f</span>"stored 0.1  = </span>{</span>0.1</span>:.30f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"stored 0.2  = </span>{</span>0.2</span>:.30f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"stored sum  = </span>{</span>0.1</span> +</span> 0.2</span>:.30f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"stored 0.3  = </span>{</span>0.3</span>:.30f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"0.1 + 0.2 == 0.3 : </span>{</span>0.1</span> +</span> 0.2</span> ==</span> 0.3</span>}</span>"</span>)</span></span>
print</span>()</span></span>
t</span> =</span> torch</span>.</span>tensor</span>(</span>1e8</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
print</span>(</span>f</span>"float32: 1e8 + 1 - 1e8 = </span>{</span>((</span>t</span> +</span> 1</span>)</span> -</span> t</span>).</span>item</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"the local step at 1e8  = </span>{</span>torch</span>.</span>nextafter</span>(</span>t</span>,</span> torch</span>.</span>tensor</span>(</span>2e8</span>)).</span>item</span>()</span> -</span> t</span>.</span>item</span>()</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
One more piece of standard equipment: the fused multiply-add, 1
instruction computing a times b plus c with a single rounding at
the end instead of 2. The result is as if the product had been
computed exactly and only the final sum were rounded, and on
modern chips it costs about as much as 1 multiply [27</a>]</span>. The
difference is not small print. Choose a = b = 1 + 2^-27 and
c = -(1 + 2^-26): the separate path rounds the product first and
returns exactly 0, while the fused path returns the true answer,
2^-54 (proof</a> runs both on this
machine). Every tensor core in the training half of this page is
a lattice of these fused units, and the sums they accumulate
stay in float32 even when the products arrive in 8 bits. Keep
that sentence; it is why low-precision training is possible at
all.</p>

  
    </span>
    p21_fma_one_rounding.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the fused multiply-add really is one rounding instead of</span></span>
2, and the difference is visible. Choose a, b and c so that the</span></span>
exact product a*b carries a tail the separate path must round away:</span></span>
  a = b = 1 + 2^-27, so a*b = 1 + 2^-26 + 2^-54 exactly</span></span>
  c = -(1 + 2^-26)</span></span>
The separate path rounds a*b to 1 + 2^-26 (the 2^-54 tail is under</span></span>
half an ulp), then adds c and returns exactly 0.0: it reports that</span></span>
a*b + c is zero when it is not. The fused path computes the exact</span></span>
product, adds c, rounds once, and returns the true answer, 2^-54.</span></span>
Both answers are legal for the source line "a*b + c", which is why</span></span>
a compiler's fusion choice can change a program's bits.</span></span>
"""</span></span>
import</span> math</span></span>
import</span> torch</span></span>
</span>
a</span> =</span> b</span> =</span> 1.0</span> +</span> 2.0</span> ** -</span>27</span></span>
c</span> =</span> -</span>(</span>1.0</span> +</span> 2.0</span> ** -</span>26</span>)</span></span>
</span>
separate</span> =</span> a</span> *</span> b</span> +</span> c</span></span>
if</span> hasattr</span>(</span>math</span>,</span> "</span>fma</span>"</span>):</span></span>
    fused</span> =</span> math</span>.</span>fma</span>(</span>a</span>,</span> b</span>,</span> c</span>)</span>          # Python 3.13+</span></span>
    how</span> =</span> "</span>math.fma</span>"</span></span>
else</span>:</span></span>
    import</span> ctypes</span>,</span> ctypes</span>.</span>util</span></span>
    libm</span> =</span> ctypes</span>.</span>CDLL</span>(</span>ctypes</span>.</span>util</span>.</span>find_library</span>(</span>"</span>m</span>"</span>))</span></span>
    libm</span>.</span>fma</span>.</span>restype</span> =</span> ctypes</span>.</span>c_double</span></span>
    libm</span>.</span>fma</span>.</span>argtypes</span> = [</span>ctypes</span>.</span>c_double</span>]</span> *</span> 3</span></span>
    fused</span> =</span> libm</span>.</span>fma</span>(</span>a</span>,</span> b</span>,</span> c</span>)</span>          # the C library's fma</span></span>
    how</span> =</span> "</span>libm fma</span>"</span></span>
</span>
print</span>(</span>f</span>"a = b = 1 + 2^-27, c = -(1 + 2^-26)"</span>)</span></span>
print</span>(</span>f</span>"separate (a*b) + c : </span>{</span>separate</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"fused fma(a, b, c) : </span>{</span>fused</span>}</span>   (via </span>{</span>how</span>}</span>)"</span>)</span></span>
print</span>(</span>f</span>"true answer        : </span>{</span>2.0</span> ** -</span>54</span>}</span>"</span>)</span></span>
assert</span> separate</span> ==</span> 0.0</span></span>
assert</span> fused</span> ==</span> 2.0</span> ** -</span>54</span></span>
</span>
t</span> = (</span>torch</span>.</span>tensor</span>(</span>a</span>,</span> dtype</span>=</span>torch</span>.</span>float64</span>)</span> *</span></span>
     torch</span>.</span>tensor</span>(</span>b</span>,</span> dtype</span>=</span>torch</span>.</span>float64</span>)</span> +</span></span>
     torch</span>.</span>tensor</span>(</span>c</span>,</span> dtype</span>=</span>torch</span>.</span>float64</span>)).</span>item</span>()</span></span>
print</span>(</span>f</span>"torch eager a*b+c  : </span>{</span>t</span>}</span>   (separate ops: also 0.0)"</span>)</span></span>
print</span>(</span>"</span>\n</span>the separate path says the answer is exactly 0 when it is</span>"</span>)</span></span>
print</span>(</span>"</span>not; the fused path keeps it. both are legal readings of the</span>"</span>)</span></span>
print</span>(</span>"</span>same source line, which is why fusion decisions (a compiler</span>"</span>)</span></span>
print</span>(</span>"</span>flag, an inlined function, torch.compile) can change a</span>"</span>)</span></span>
print</span>(</span>"</span>program's last bits without anyone touching the math.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
The unit itself is worth one look, because the training half of
this page keeps returning to it:</p>

  
    
    
  </div>
  Figure 18.</span> one fused multiply-add unit. the product keeps every bit, the running sum lives in float32, and the single rounding happens at the end. a tensor core is a lattice of these.</figcaption>
</figure>
Where arithmetic goes wrong</a></h2>
Three rules of ordinary arithmetic stop holding once every
result is rounded to a point, and all three cause real bugs.</p>
Absorption.</strong> At 100,000,000, which is 1e8, float32's step is 8, so
adding 1 offers a move smaller than half the gap, and the sum
rounds straight back (proof</a>):</p>

  
    
    
  </div>
  Figure 19.</span> 1e8 + 1 = 1e8, exactly. the addend was real; the local step is 8; there is nowhere closer to land.</figcaption>
</figure>

  
    </span>
    p4_absorb_cancel.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the two classic accidents. Absorption: below half the</span></span>
local step, an addend vanishes. Cancellation: subtracting two close</span></span>
numbers deletes their shared leading digits and promotes the noise.</span>"""</span></span>
import</span> torch</span></span>
t</span> =</span> torch</span>.</span>tensor</span>(</span>1e8</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
print</span>(</span>f</span>"float32 step at 1e8      : </span>{</span>(</span>torch</span>.</span>nextafter</span>(</span>t</span>,</span> t</span>*</span>2</span>)</span> -</span> t</span>).</span>item</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"1e8 + 1                  : </span>{</span>(</span>t</span> +</span> 1</span>).</span>item</span>()</span>:.1f</span>}</span>   (absorbed)"</span>)</span></span>
print</span>(</span>f</span>"1e8 + 5                  : </span>{</span>(</span>t</span> +</span> 5</span>).</span>item</span>()</span>:.1f</span>}</span>   (rounds up: past half the step)"</span>)</span></span>
a</span> =</span> torch</span>.</span>tensor</span>(</span>1.0000001</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
b</span> =</span> torch</span>.</span>tensor</span>(</span>1.0000000</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
d</span> = (</span>a</span> -</span> b</span>).</span>item</span>()</span></span>
print</span>(</span>f</span>"</span>\n</span>(1.0000001 - 1.0) in float32 = </span>{</span>d</span>:.10e</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"true answer                  = 1.0000000e-07"</span>)</span></span>
print</span>(</span>f</span>"relative error               = </span>{</span>abs</span>(</span>d</span> -</span> 1e-7</span>)</span>/</span>1e-7</span>:.1%</span>}</span>"</span>)</span></span>
print</span>(</span>"</span>seven leading digits matched and left; only the rounding</span>"</span>)</span></span>
print</span>(</span>"</span>noise of the inputs remained.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Cancellation.</strong> Subtract two nearly equal numbers and their
shared leading digits leave together, promoting whatever rounding
noise the inputs carried into the leading digits of the answer.
The proof above subtracts 1.0000001 from 1.0 in float32 and gets
1.19e-7 where the truth is 1.0e-7: a 19% error from one
subtraction of two almost-exact inputs. The strange part: the
subtraction itself commits no error at all. When 2 numbers are
within a factor of 2 of each other, their difference is always
exactly representable (Sterbenz's lemma [4</a>]</span>;
proof</a> checks it on this very case).
Cancellation never creates error; it exposes the error the
inputs already carried.</p>
Order.</strong> Because every addition rounds, (a + b) + c and
a + (b + c) are different numbers. Sum 50,000 values
forward and backward and the answers disagree
(proof</a>); a parallel machine that splits the
same sum across cores picks yet another order, and another answer.
torch's own .sum()</code> adds pairwise in a tree, which is both
faster and about 130 times closer to the true sum than a
one-by-one loop (measured: 0.24 error against 0.0018), and compensated summation (Kahan's trick of carrying
the rounding error in a second variable) closes most of the rest
[4</a>]</span>.</p>

  
    </span>
    p5_sum_order.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: addition order changes the answer. One million float32</span></span>
values, three orders, three sums; then the repairs: pairwise</span></span>
(what torch does), Kahan compensation, and fsum as ground truth.</span>"""</span></span>
import</span> math</span>,</span> torch</span></span>
</span>
torch</span>.</span>manual_seed</span>(</span>0</span>)</span></span>
x</span> = (</span>torch</span>.</span>randn</span>(</span>1_000_000</span>)</span> *</span> 100</span>).</span>float</span>()</span></span>
truth</span> =</span> math</span>.</span>fsum</span>(</span>x</span>.</span>double</span>().</span>tolist</span>())</span></span>
</span>
fwd</span> =</span> torch</span>.</span>tensor</span>(</span>0.0</span>)</span></span>
for</span> c</span> in</span> x</span>.</span>split</span>(</span>100_000</span>):</span>          # sequential in chunks, forward</span></span>
    for</span> v</span> in</span> c</span>:</span> pass</span></span>
# a plain python loop over 1e6 floats is slow; do exact fp32 fold in torch</span></span>
def</span> fold</span>(</span>t</span>):</span></span>
    s</span> =</span> torch</span>.</span>tensor</span>(</span>0.0</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
    for</span> v</span> in</span> t</span>.</span>split</span>(</span>4096</span>):</span></span>
        for</span> u</span> in</span> v</span>.</span>tolist</span>():</span></span>
            s</span> =</span> s</span> +</span> torch</span>.</span>tensor</span>(</span>u</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
    return</span> s</span>.</span>item</span>()</span></span>
</span>
seq_fwd</span> =</span> fold</span>(</span>x</span>[:</span>50_000</span>])</span>          # smaller slice keeps runtime sane</span></span>
seq_rev</span> =</span> fold</span>(</span>x</span>[:</span>50_000</span>].</span>flip</span>(</span>0</span>))</span></span>
tsum</span>    =</span> x</span>[:</span>50_000</span>].</span>sum</span>().</span>item</span>()</span>   # torch's pairwise-style reduction</span></span>
truth50</span> =</span> math</span>.</span>fsum</span>(</span>x</span>[:</span>50_000</span>].</span>double</span>().</span>tolist</span>())</span></span>
print</span>(</span>f</span>"ground truth (fsum)       : </span>{</span>truth50</span>:.6f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"sequential, forward       : </span>{</span>seq_fwd</span>:.6f</span>}</span>   err </span>{</span>seq_fwd</span>-</span>truth50</span>:+.6f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"sequential, reversed      : </span>{</span>seq_rev</span>:.6f</span>}</span>   err </span>{</span>seq_rev</span>-</span>truth50</span>:+.6f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"torch .sum() (pairwise)   : </span>{</span>tsum</span>:.6f</span>}</span>   err </span>{</span>tsum</span>-</span>truth50</span>:+.6f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"forward == reversed       : </span>{</span>seq_fwd</span> ==</span> seq_rev</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
One more fact turns this whole section from a list of failures
into a toolbox: the rounding error is catchable, exactly. The
error of an addition is itself a number the format can hold, and
3 operations recover it whole: s = a + b, z = s - a, e = b - z,
and now s + e equals the true sum of a and b with no error at
all. Run it on this page's opening example and the 3 operations
hand back the exact rounding error of 0.1 + 0.2
(proof</a> checks it against exact rational
arithmetic). A fused multiply-add does the same for a product in
1 instruction. Kahan's compensated sum is this trick run in a
loop, and the double-word arithmetic that stretches precision in
software is this trick kept instead of thrown away [4</a>]</span>.</p>

  
    </span>
    p22_exact_error.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the rounding error is catchable, exactly. 3 facts, each</span></span>
checked against exact rational arithmetic (fractions.Fraction):</span></span>
1. Fast2Sum: for |a| >= |b|, 3 operations recover the rounding</span></span>
   error of a + b perfectly: s = a+b; z = s-a; e = b-z. Then</span></span>
   s + e equals the true sum of the stored a and b, exactly.</span></span>
   Run on the page's opening example, 0.1 + 0.2.</span></span>
2. Sterbenz: when x and y are within a factor of 2 of each</span></span>
   other, x - y is computed exactly: cancellation commits no</span></span>
   error of its own; it only exposes error the inputs carried.</span></span>
3. 2MultFMA: 1 fused multiply-add recovers a product's whole</span></span>
   rounding error: e = fma(a, b, -a*b).</span></span>
"""</span></span>
from</span> fractions</span> import</span> Fraction</span> as</span> F</span></span>
import</span> ctypes</span>,</span> ctypes</span>.</span>util</span></span>
</span>
# ---- 1. Fast2Sum on 0.1 + 0.2</span></span>
a</span>,</span> b</span> =</span> 0.2</span>,</span> 0.1</span></span>
s</span> =</span> a</span> +</span> b</span></span>
z</span> =</span> s</span> -</span> a</span></span>
e</span> =</span> b</span> -</span> z</span></span>
print</span>(</span>f</span>"s = a + b        = </span>{</span>s</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"e = (b - (s - a)) = </span>{</span>e</span>!r</span>}</span>"</span>)</span></span>
assert</span> F</span>(</span>a</span>)</span> +</span> F</span>(</span>b</span>)</span> ==</span> F</span>(</span>s</span>)</span> +</span> F</span>(</span>e</span>)</span></span>
print</span>(</span>"</span>checked: stored 0.2 + stored 0.1 == s + e, EXACTLY.</span>"</span>)</span></span>
print</span>(</span>"</span>the 3 float operations caught the rounding error whole.</span>\n</span>"</span>)</span></span>
</span>
# ---- 2. Sterbenz: close subtraction is exact</span></span>
x</span>,</span> y</span> =</span> 1.0000001</span>,</span> 1.0</span></span>
d</span> =</span> x</span> -</span> y</span></span>
assert</span> F</span>(</span>x</span>)</span> -</span> F</span>(</span>y</span>)</span> ==</span> F</span>(</span>d</span>)</span></span>
print</span>(</span>f</span>"x - y = </span>{</span>d</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>"</span>checked: equals the exact difference of the stored inputs.</span>"</span>)</span></span>
print</span>(</span>"</span>the 19</span>% e</span>rror of the cancellation section was already in</span>"</span>)</span></span>
print</span>(</span>"</span>the inputs; the subtraction added nothing.</span>\n</span>"</span>)</span></span>
</span>
# ---- 3. 2MultFMA: a product's error in one instruction</span></span>
libm</span> =</span> ctypes</span>.</span>CDLL</span>(</span>ctypes</span>.</span>util</span>.</span>find_library</span>(</span>"</span>m</span>"</span>))</span></span>
libm</span>.</span>fma</span>.</span>restype</span> =</span> ctypes</span>.</span>c_double</span></span>
libm</span>.</span>fma</span>.</span>argtypes</span> = [</span>ctypes</span>.</span>c_double</span>]</span> *</span> 3</span></span>
u</span> =</span> v</span> =</span> 1.0</span> +</span> 2.0</span> ** -</span>27</span></span>
p</span> =</span> u</span> *</span> v</span></span>
e2</span> =</span> libm</span>.</span>fma</span>(</span>u</span>,</span> v</span>,</span> -</span>p</span>)</span></span>
assert</span> F</span>(</span>u</span>)</span> *</span> F</span>(</span>v</span>)</span> ==</span> F</span>(</span>p</span>)</span> +</span> F</span>(</span>e2</span>)</span></span>
print</span>(</span>f</span>"p = u * v       = </span>{</span>p</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"e = fma(u,v,-p) = </span>{</span>e2</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>"</span>checked: stored u times stored v == p + e, exactly.</span>"</span>)</span></span>
print</span>(</span>"</span>\n</span>Kahan's compensated sum is fact 1 run in a loop; the</span>"</span>)</span></span>
print</span>(</span>"</span>double-word arithmetic that stretches precision in software</span>"</span>)</span></span>
print</span>(</span>"</span>is facts 1 and 3 kept instead of thrown away.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Keep the order fact in mind. It returns at the end of this page as
the reason two identical training runs on two GPUs never match to
the last bit.</p>
The field guide</a></h2>
The formats in this guide carry nearly all of the world's
floating-point arithmetic, and you now own every idea needed to
read them. Each gets the same treatment: its card, the bit
counts and the numbers that follow from them; its map, where
its points land; where it came from; where it runs today; and
how it fails. Every number on every card is derived from the 2
bit counts alone, with the formulas you used on binary32, and
checked against torch's own tables
(proof</a> runs the check). The guide
covers the 4 wide formats, the ones that stand alone. The 8-,
6- and 4-bit formats enter later, each at the exact place in
the training story that needs it, and a closing sheet gathers
the whole family on one drawing.</p>

  
    </span>
    p16_wide_formats.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the field guide's numbers, derived by hand from each</span></span>
format's two counts (E exponent bits, M mantissa bits), then checked</span></span>
against torch. Nothing on the cards is typed from memory:</span></span>
  bias        = 2^(E-1) - 1</span></span>
  ceiling     = (2 - 2^-M) * 2^bias</span></span>
  floor       = 2^(1 - bias)          (smallest normal)</span></span>
  ramp floor  = 2^(1 - bias - M)      (smallest subnormal)</span></span>
  step at 1   = 2^-M                  (the ulp of 1.0; finfo calls it eps)</span></span>
  cells       = 2^E - 2               (power-of-two intervals)</span></span>
  points/cell = 2^M</span></span>
Also prints the torch defaults the float32 section quotes.</span>"""</span></span>
import</span> math</span></span>
import</span> torch</span></span>
</span>
FORMATS</span> = [</span></span>
    (</span>"</span>float64</span>"</span>,</span> 11</span>,</span> 52</span>,</span> torch</span>.</span>float64</span>,</span> torch</span>.</span>int64</span>),</span></span>
    (</span>"</span>float32</span>"</span>,</span> 8</span>,</span> 23</span>,</span> torch</span>.</span>float32</span>,</span> torch</span>.</span>int32</span>),</span></span>
    (</span>"</span>float16</span>"</span>,</span> 5</span>,</span> 10</span>,</span> torch</span>.</span>float16</span>,</span> torch</span>.</span>int16</span>),</span></span>
    (</span>"</span>bfloat16</span>"</span>,</span> 8</span>,</span> 7</span>,</span> torch</span>.</span>bfloat16</span>,</span> torch</span>.</span>int16</span>),</span></span>
]</span></span>
</span>
for</span> name</span>,</span> E</span>,</span> M</span>,</span> dt</span>,</span> it</span> in</span> FORMATS</span>:</span></span>
    bias</span> =</span> 2</span> **</span> (</span>E</span> -</span> 1</span>)</span> -</span> 1</span></span>
    ceiling</span> = (</span>2</span> -</span> 2.0</span> ** -</span>M</span>)</span> *</span> 2.0</span> **</span> bias</span></span>
    floor</span> =</span> 2.0</span> **</span> (</span>1</span> -</span> bias</span>)</span></span>
    ramp_floor</span> =</span> 2.0</span> **</span> (</span>1</span> -</span> bias</span> -</span> M</span>)</span></span>
    step_at_1</span> =</span> 2.0</span> ** -</span>M</span></span>
    cells</span> =</span> 2</span> **</span> E</span> -</span> 2</span></span>
    digits</span> = (</span>M</span> +</span> 1</span>)</span> *</span> math</span>.</span>log10</span>(</span>2</span>)</span></span>
</span>
    fi</span> =</span> torch</span>.</span>finfo</span>(</span>dt</span>)</span></span>
    # smallest subnormal: bit pattern 0...01 viewed as this dtype</span></span>
    sub</span> =</span> torch</span>.</span>tensor</span>([</span>1</span>],</span> dtype</span>=</span>it</span>).</span>view</span>(</span>dt</span>).</span>double</span>().</span>item</span>()</span></span>
</span>
    assert</span> ceiling</span> ==</span> fi</span>.</span>max</span>, (</span>name</span>,</span> ceiling</span>,</span> fi</span>.</span>max</span>)</span></span>
    assert</span> floor</span> ==</span> fi</span>.</span>tiny</span>, (</span>name</span>,</span> floor</span>,</span> fi</span>.</span>tiny</span>)</span></span>
    assert</span> step_at_1</span> ==</span> fi</span>.</span>eps</span>, (</span>name</span>,</span> step_at_1</span>,</span> fi</span>.</span>eps</span>)</span></span>
    assert</span> ramp_floor</span> ==</span> sub</span>, (</span>name</span>,</span> ramp_floor</span>,</span> sub</span>)</span></span>
</span>
    print</span>(</span>f</span>"</span>{</span>name</span>:9s</span>}</span> E=</span>{</span>E</span>:2d</span>}</span> M=</span>{</span>M</span>:2d</span>}</span>  bias </span>{</span>bias</span>:4d</span>}</span>   "</span></span>
          f</span>"ceiling </span>{</span>ceiling</span>:.4g</span>}</span>"</span>)</span></span>
    print</span>(</span>f</span>"          floor </span>{</span>floor</span>:.4g</span>}</span>   ramp floor </span>{</span>ramp_floor</span>:.4g</span>}</span>   "</span></span>
          f</span>"step at 1 = </span>{</span>step_at_1</span>:.4g</span>}</span>"</span>)</span></span>
    print</span>(</span>f</span>"</span>          {</span>cells</span>}</span> cells of </span>{</span>2</span>**</span>M</span>:,</span>}</span> points; "</span></span>
          f</span>"about </span>{</span>digits</span>:.1f</span>}</span> decimal digits   "</span></span>
          f</span>"(torch.finfo agrees on all four numbers)"</span>)</span></span>
</span>
# absorption never leaves; it only moves. float64's step at 1e16 is 2.</span></span>
x</span> =</span> torch</span>.</span>tensor</span>(</span>1e16</span>,</span> dtype</span>=</span>torch</span>.</span>float64</span>)</span></span>
print</span>(</span>f</span>"</span>\n</span>float64: (1e16 + 1) - 1e16 = </span>{</span>((</span>x</span> +</span> 1</span>)</span> -</span> x</span>).</span>item</span>()</span>:g</span>}</span>"</span></span>
      f</span>"   (the step at 1e16 is 2; the 1 is below half of it)"</span>)</span></span>
</span>
# the defaults the float32 section quotes, read from this install</span></span>
print</span>(</span>f</span>"</span>\n</span>torch </span>{</span>torch</span>.</span>__version__</span>}</span> defaults:"</span>)</span></span>
print</span>(</span>f</span>"  default dtype                  </span>{</span>torch</span>.</span>get_default_dtype</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"  float32_matmul_precision       "</span></span>
      f</span>"</span>{</span>torch</span>.</span>get_float32_matmul_precision</span>()</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"  backends.cuda.matmul.allow_tf32  "</span></span>
      f</span>"</span>{</span>torch</span>.</span>backends</span>.</span>cuda</span>.</span>matmul</span>.</span>allow_tf32</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"  backends.cudnn.allow_tf32        "</span></span>
      f</span>"</span>{</span>torch</span>.</span>backends</span>.</span>cudnn</span>.</span>allow_tf32</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"  autocast dtype on cpu          "</span></span>
      f</span>"</span>{</span>torch</span>.</span>get_autocast_dtype</span>(</span>'</span>cpu</span>'</span>)</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
float64</a></h2>
Double binary32's byte count and you get the format the 1985
standard called double precision:</p>

  
    
    
  </div>
  Figure 20.</span> float64's card. 11 bits of interval, 52 of position; every row follows from those two counts (proof below the guide's intro).</figcaption>
</figure>
The bias is 1023, so the intervals run from 2.2e-308 up to 1.8e308:
2,046 of them, each holding 4,503,599,627,370,496 points. The
ulp at 1.0 is 2.22e-16, which is about 16 decimal digits of
position. A map that wide can only be drawn with breaks:</p>

  
    
    
  </div>
  Figure 21.</span> the float64 map: the E4M3 picture with 2,046 intervals instead of 15, and 2^52 points per interval instead of 2^3.</figcaption>
</figure>
Where it came from: IEEE 754 began as the arithmetic William
Kahan designed with Intel for the 8087, the floating-point
coprocessor sold beside the 8086; the standards committee turned
that chip's arithmetic into everyone's law [3</a>][18</a>]</span>. float64 was
its wide grade, sized so ordinary science could chain millions
of operations and still trust the leading digits.</p>
Where it runs today: everywhere you did not choose a dtype. A
Python float is a float64. NumPy builds float64 arrays unless
told otherwise. The digits of 0.1 printed in the rounding
section were float64 digits.</p>
Its proof: p16</a> derives the whole card
from E and M and checks every row against torch.finfo</code>. One
extra line in it repeats the absorption experiment at this
format's scale: in float64, (1e16 + 1) - 1e16 = 0. The ulp at
1e16 is 2, and the added 1 fell under half of it. 16 digits
push the cliff out of sight; no digit count removes it.</p>
How it fails in a training loop: float64 rarely gives training
a wrong answer. Its problem is cost. 8 bytes per number is 2
times float32 and 4 times bfloat16, spent on digits training
cannot use: the sections ahead show training running where the
step at 1.0 is 0.0078, so digits 4 through 16 add nothing. On
GPUs there is a second cost: most chips carry few float64 units,
and NVIDIA's own throughput tables list float64 operations far
below float32 on most of its hardware [19</a>]</span>. torch's default is
float32, and float64 appears only when you ask
(proof</a> prints the default).</p>
float32, and the format hiding inside it</a></h2>
binary32 you have already decoded by hand; the card collects its
numbers:</p>

  
    
    
  </div>
  Figure 22.</span> float32's card. the orange row, 1.19e-7 at 1.0, is the machine epsilon from the rounding section.</figcaption>
</figure>
254 intervals from 1.2e-38 to 3.4e38, 8,388,608 points in each,
about 7 digits:</p>

  
    
    
  </div>
  Figure 23.</span> the float32 map. the absorption step of 8 at 1e8 is one of these intervals, far to the right of 1.</figcaption>
</figure>
Where it came from: the same 1985 standard's single precision [3</a>]</span>.
Where it runs today: it is the default dtype of torch;
torch.tensor(1.0)</code> is a float32 unless you say otherwise
(proof</a> prints the default from this
install). Even when matmuls run narrow, the master weights of
mixed-precision training stay float32 and the tensor cores
accumulate in float32, so the training half of this page
keeps this format at its center.</p>
Now the format hiding inside it: TensorFloat-32. Since the A100,
NVIDIA's tensor cores can take an ordinary float32 matmul, round
each factor to 10 mantissa bits, multiply, and accumulate in
full float32 [20</a>]</span>. The tensors going in and out are ordinary
float32; only the multiply is narrow, reading 19 of the 32 bits
(1 + 8 + 10). TF32 is that mode, not a storage type you can give
a tensor. NVIDIA measured large speedups at matching accuracy on
deep-learning workloads [20</a>]</span>; for code that needs all seven
digits it is a silent cut, and that is why torch ships with
matmul TF32 off since version 1.12 [21</a>]</span>. On this install,
torch.backends.cuda.matmul.allow_tf32</code> is False, the cudnn
convolution flag is True, and
torch.set_float32_matmul_precision("high")</code> is the one-line
opt-in (proof</a> prints all three). None
of this can execute on the Apple machine this page is measured
on; the flags are read here, the behavior is cited [20</a>][21</a>]</span>.</p>

  
    
    
  </div>
  Figure 24.</span> TF32's card: float32's reach, float16's digits, 19 bits, no storage form. it exists only in the middle of a tensor-core multiply.</figcaption>
</figure>
How it fails: you have watched it fail all page. The step of 8
at 1e8 is float32's. The 7 digits that turned a subtraction of
near-equal inputs into a 19% error are float32's. In
training, its failure is price: 4 bytes per number on the
memory bus that a later section shows is the bottleneck.</p>
float16</a></h2>
Cut 32 bits in half and something must go. There are two ways to
choose, and the choice split the 16-bit world in two:</p>

  
    
    
  </div>
  Figure 25.</span> the fork. float16 kept digits and lost reach; bfloat16 kept float32's whole exponent and pays in digits. training mostly chose reach.</figcaption>
</figure>
float16 is the left fork: keep digits, pay with reach.</p>

  
    
    
  </div>
  Figure 26.</span> float16's card. the orange row is the ceiling, 65504: the number to remember about this format.</figcaption>
</figure>
5 exponent bits give bias 15 and only 30 intervals, from 6.1e-5 to
65504. 10 mantissa bits give 1,024 points per interval and about 3
digits. 30 intervals fit in one drawing, so this is the one wide
format whose map you can see whole:</p>

  
    
    
  </div>
  Figure 27.</span> all of float16, no breaks needed: 30 intervals end to end. compare the reach with the maps above it.</figcaption>
</figure>
Where it came from: film and games, not a numerics committee.
NVIDIA and Microsoft made half</code> a type in the Cg shading
language in 2002, and Industrial Light & Magic built its OpenEXR
film format on the same 1+5+10 layout, in production from 2000
and released as open source in 2003 [22</a>]</span>. IEEE 754 adopted the
layout as binary16 in its 2008 revision [3</a>]</span>. The film industry
shipped the format years before the standard named it.</p>
Where it runs today: graphics APIs and image pipelines still,
inference engines, and mixed-precision training on hardware from
before bfloat16 spread. In torch it is the half in .half()</code>.</p>
Its proof and its failure are the same three numbers. The
ceiling: 60000 times 1.2 is inf in float16
(proof</a>). The digits: 1.001 stores as
1.000977. The floor: normal numbers end at 6.1e-5, and the small
half of a gradient histogram lives below that, which is why the
loss-scaling section exists. When float16 dies in a run, it dies
at one of these three numbers.</p>

  
    </span>
    p7_sixteen_bits.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the 16-bit fork, from finfo, nothing typed from memory.</span></span>
float16 spent its bits on precision and dies at 65504; bfloat16 kept</span></span>
float32's exponent and survives, with two decimal digits left.</span>"""</span></span>
import</span> torch</span></span>
</span>
for</span> dt</span> in</span> (</span>torch</span>.</span>float32</span>,</span> torch</span>.</span>float16</span>,</span> torch</span>.</span>bfloat16</span>):</span></span>
    fi</span> =</span> torch</span>.</span>finfo</span>(</span>dt</span>)</span></span>
    print</span>(</span>f</span>"</span>{</span>str</span>(</span>dt</span>)</span>:16s</span>}</span> max </span>{</span>fi</span>.</span>max</span>:>12.4g</span>}</span>   tiny </span>{</span>fi</span>.</span>tiny</span>:.3g</span>}</span>   "</span></span>
          f</span>"eps </span>{</span>fi</span>.</span>eps</span>:.3g</span>}</span>"</span>)</span></span>
x</span> =</span> torch</span>.</span>tensor</span>(</span>60000.0</span>)</span></span>
print</span>(</span>f</span>"</span>\n</span>60000 * 1.2 in float16 : </span>{</span>(</span>x</span>.</span>half</span>()</span> *</span> 1.2</span>).</span>item</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"60000 * 1.2 in bfloat16: </span>{</span>(</span>x</span>.</span>bfloat16</span>()</span> *</span> torch</span>.</span>tensor</span>(</span>1.2</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)).</span>item</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"1.001 in float16       : </span>{</span>torch</span>.</span>tensor</span>(</span>1.001</span>,</span> dtype</span>=</span>torch</span>.</span>float16</span>).</span>item</span>()</span>:.6f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"1.001 in bfloat16      : </span>{</span>torch</span>.</span>tensor</span>(</span>1.001</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>).</span>item</span>()</span>:.6f</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
bfloat16</a></h2>
The right fork: keep reach, pay with digits. Google's brain
float keeps all 8 of float32's exponent bits and 7 of mantissa:</p>

  
    
    
  </div>
  Figure 28.</span> bfloat16's card: float32's bias and reach, 7 bits of position. the orange row is the step that rounds weight updates away.</figcaption>
</figure>
The reach is float32's (the ceiling prints 3.39e38 rather than
3.4e38 only because the coarser last step lands the top point
lower), the intervals are the same 254 as float32's, and each holds 128
points: about 2 decimal digits.</p>

  
    
    
  </div>
  Figure 29.</span> the bfloat16 map next to float32's: the same intervals, 65,536 times fewer points in each. that one comparison is the entire format.</figcaption>
</figure>
Where it came from: Google built it into the TPU's matrix
units, which multiply in bfloat16 and accumulate in float32, and
chose the layout so that range would never be the problem and
conversion from float32 would be nearly free [23</a>]</span>. The name is
Brain Floating Point, after the Google Brain team.</p>
Nearly free is provable, and on this machine it is proven at the
bit level: bfloat16 is float32's top half. Take any of the
65,536 possible bfloat16 bit patterns, cast it to float32, and
the result is the same 16 bits with 16 zeros appended;
proof</a> checks all 65,536, bit for
bit. The downward cast is the same move with rounding: keep the top
16 bits, round by the low 16. The proof checks
torch's cast against that hand rule on 1,000,010 values (a
million random ones plus the edge cases), and they agree on
every one. Here is 1.7014 making the trip:</p>
float32:  0 01111111 1011001 1100011101111010</span></span>
bfloat16: 0 01111111 1011010   (the top half, rounded up by the rest)</span></span>
stored:   1.703125</span></span></code></pre>
  
    </span>
    p17_bf16_truncation.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: bfloat16 is the top 16 bits of float32, at the bit level.</span></span>
Two directions, both exhaustive where exhaustion is possible:</span></span>
  up:   every one of the 65,536 bfloat16 bit patterns, cast to</span></span>
        float32, must produce exactly (pattern << 16): the same 16</span></span>
        bits on top, sixteen zeros appended below.</span></span>
  down: casting float32 to bfloat16 must equal keeping the top 16</span></span>
        bits and rounding by the low 16 (nearest, ties to even),</span></span>
        checked on a million random values plus the edge cases.</span></span>
"""</span></span>
import</span> torch</span></span>
</span>
# ---- up: bf16 -> fp32 is "append 16 zero bits", all 65,536 patterns</span></span>
pats</span> =</span> torch</span>.</span>arange</span>(</span>65536</span>,</span> dtype</span>=</span>torch</span>.</span>int32</span>).</span>to</span>(</span>torch</span>.</span>int16</span>)</span></span>
as_bf16</span> =</span> pats</span>.</span>view</span>(</span>torch</span>.</span>bfloat16</span>)</span></span>
up_bits</span> =</span> as_bf16</span>.</span>float</span>().</span>view</span>(</span>torch</span>.</span>int32</span>)</span></span>
want</span> =</span> pats</span>.</span>to</span>(</span>torch</span>.</span>int32</span>)</span> <<</span> 16</span></span>
nan_mask</span> =</span> torch</span>.</span>isnan</span>(</span>as_bf16</span>)</span></span>
ok</span> = (</span>up_bits</span> ==</span> want</span>)</span> |</span> nan_mask</span></span>
print</span>(</span>f</span>"up-cast bit check: </span>{</span>ok</span>.</span>sum</span>().</span>item</span>()</span>}</span> / 65536 patterns match "</span></span>
      f</span>"(pattern << 16)"</span>)</span></span>
print</span>(</span>f</span>"  of the 65536, </span>{</span>nan_mask</span>.</span>sum</span>().</span>item</span>()</span>}</span> are NaN codes, "</span></span>
      f</span>"checked separately:"</span>)</span></span>
nan_ok</span> =</span> torch</span>.</span>isnan</span>(</span>as_bf16</span>.</span>float</span>()[</span>nan_mask</span>]).</span>all</span>().</span>item</span>()</span></span>
print</span>(</span>f</span>"  every NaN pattern up-casts to a float32 NaN: </span>{</span>nan_ok</span>}</span>"</span>)</span></span>
exact</span> = (</span>up_bits</span> ==</span> want</span>).</span>sum</span>().</span>item</span>()</span></span>
print</span>(</span>f</span>"  bit-exact including NaN payloads: </span>{</span>exact</span>}</span> / 65536"</span>)</span></span>
</span>
# ---- down: fp32 -> bf16 is "keep the top 16 bits, round by the rest"</span></span>
def</span> truncate_rne</span>(</span>f32</span>):</span></span>
    """</span>The hand rule: add the rounding bias, then keep the top half.</span></span>
    bias = 0x7FFF + (bit 16), the classic nearest-even truncation.</span>"""</span></span>
    u</span> =</span> f32</span>.</span>view</span>(</span>torch</span>.</span>int32</span>)</span></span>
    bias</span> =</span> 0x</span>7FFF</span> +</span> ((</span>u</span> >></span> 16</span>)</span> &</span> 1</span>)</span></span>
    out</span> = ((</span>u</span> +</span> bias</span>)</span> >></span> 16</span>).</span>to</span>(</span>torch</span>.</span>int16</span>).</span>view</span>(</span>torch</span>.</span>bfloat16</span>)</span></span>
    return</span> torch</span>.</span>where</span>(</span>torch</span>.</span>isnan</span>(</span>f32</span>),</span> torch</span>.</span>nan</span>,</span> out</span>.</span>float</span>())</span></span>
</span>
torch</span>.</span>manual_seed</span>(</span>0</span>)</span></span>
x</span> =</span> torch</span>.</span>randn</span>(</span>1_000_000</span>)</span> *</span> torch</span>.</span>logspace</span>(</span>-</span>38</span>,</span> 38</span>,</span> 1_000_000</span>)</span></span>
edges</span> =</span> torch</span>.</span>tensor</span>([</span>0.0</span>,</span> -</span>0.0</span>,</span> 1.0</span>,</span> 1.0078125</span>,</span> 65504.0</span>,</span> 3.4e38</span>,</span></span>
                      1e-40</span>,</span> float</span>(</span>"</span>inf</span>"</span>),</span> float</span>(</span>"</span>-inf</span>"</span>),</span> float</span>(</span>"</span>nan</span>"</span>)])</span></span>
x</span> =</span> torch</span>.</span>cat</span>([</span>x</span>,</span> edges</span>])</span></span>
ours</span> =</span> truncate_rne</span>(</span>x</span>)</span></span>
torchs</span> =</span> x</span>.</span>bfloat16</span>().</span>float</span>()</span></span>
agree</span> = (</span>ours</span> ==</span> torchs</span>)</span> |</span> (</span>torch</span>.</span>isnan</span>(</span>ours</span>)</span> &</span> torch</span>.</span>isnan</span>(</span>torchs</span>))</span></span>
print</span>(</span>f</span>"</span>\n</span>down-cast rule check: </span>{</span>agree</span>.</span>sum</span>().</span>item</span>()</span>}</span> / </span>{</span>len</span>(</span>x</span>)</span>}</span> "</span></span>
      f</span>"values agree with torch's cast"</span>)</span></span>
</span>
# ---- the picture, on one number</span></span>
v</span> =</span> torch</span>.</span>tensor</span>(</span>1.7014</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
u32</span> =</span> v</span>.</span>view</span>(</span>torch</span>.</span>int32</span>).</span>item</span>()</span> & 0x</span>FFFFFFFF</span></span>
b16</span> =</span> v</span>.</span>bfloat16</span>().</span>view</span>(</span>torch</span>.</span>int16</span>).</span>item</span>()</span> & 0x</span>FFFF</span></span>
print</span>(</span>f</span>"</span>\n</span>1.7014 in float32 bits : </span>{</span>u32</span>:032b</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"1.7014 in bfloat16 bits: </span>{</span>b16</span>:016b</span>}</span> (the top half, rounded)"</span>)</span></span>
print</span>(</span>f</span>"bfloat16 value stored  : </span>{</span>v</span>.</span>bfloat16</span>().</span>item</span>()</span>:.6f</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Where it runs today: TPUs since their second generation [23</a>]</span>,
NVIDIA GPUs since the A100 [20</a>]</span>, and this machine: torch's CPU
autocast picks bfloat16 by default
(proof</a> prints it). In large-model
training it is the usual compute half of the mixed-precision
loop, with FP8 taking over the matmuls on the newest chips (the
FP8 section returns to this).</p>
How it fails: 2 digits. 1.001 stores as exactly 1.0
(proof</a>). The step at 1.0 is 0.0078, so
a healthy update of 0.001 sits under half a step and rounds
away: the disaster that opens the training sections ahead, and
the reason master weights exist. bfloat16 did not make training precise. It
made training's failures the quiet, repairable kind, and the
machinery that repairs them is most of the rest of this page.</p>
The guide pauses here, because at 8 bits and below no format
stands alone. Each narrow format arrives later on this page, at
the moment the training story needs it, and a closing sheet
gathers the whole family in one drawing. First, a payoff: with
the wide formats on the table, the small constants in every
model's source code finally have their explanation.</p>
The little epsilons, explained</a></h2>
Open any model's source code and small constants appear: an
eps=1e-5</code> in the layer norm, an eps=1e-8</code> in Adam, a
logits - logits.max()</code> in every attention implementation. None of them is superstition; each one is a patch over a place
where a formula's intermediate value leaves the map.</p>
The largest number float32 can hold is about 3.4e38, and exp
reaches it at 88.73 (measured below). Softmax, the function that
turns a model's raw output scores, its logits, into
probabilities, applies exp to every logit. Logits above 88.73
exist in every large model, so a naive softmax returns inf, then
inf divided by inf, and your loss prints nan. The repair
costs nothing: softmax only sees differences, so subtract the
maximum first and the largest input becomes 0
(proof</a>):</p>

  
    
    
  </div>
  Figure 30.</span> the cliff is at 88.7 in float32 and at 11.09 in float16. every stable softmax you have ever used is the same one-line slide.</figcaption>
</figure>

  
    </span>
    p6_stability.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the stability patterns. exp overflows at 88.73 in float32</span></span>
and 11.09 in float16, so a naive softmax dies on logits your model</span></span>
produces every day; subtracting the max moves nothing and fixes it</span></span>
(softmax is shift-invariant).</span>"""</span></span>
import</span> torch</span></span>
</span>
print</span>(</span>f</span>"float32 exp cliff: exp(88.7)=</span>{</span>torch</span>.</span>tensor</span>(</span>88.7</span>).</span>exp</span>().</span>item</span>()</span>:.3e</span>}</span>, "</span></span>
      f</span>"exp(88.8)=</span>{</span>torch</span>.</span>tensor</span>(</span>88.8</span>).</span>exp</span>().</span>item</span>()</span>}</span>"</span>)</span></span>
h</span> =</span> torch</span>.</span>tensor</span>([</span>11.0</span>,</span> 11.1</span>],</span> dtype</span>=</span>torch</span>.</span>float16</span>)</span></span>
print</span>(</span>f</span>"float16 exp cliff: exp(11.0)=</span>{</span>h</span>.</span>exp</span>()[</span>0</span>].</span>item</span>()</span>}</span>, exp(11.1)=</span>{</span>h</span>.</span>exp</span>()[</span>1</span>].</span>item</span>()</span>}</span>"</span>)</span></span>
</span>
logits</span> =</span> torch</span>.</span>tensor</span>([</span>120.0</span>,</span> 119.0</span>,</span> 115.0</span>])</span></span>
naive</span> =</span> logits</span>.</span>exp</span>()</span> /</span> logits</span>.</span>exp</span>().</span>sum</span>()</span></span>
stable</span> = (</span>logits</span> -</span> logits</span>.</span>max</span>()).</span>exp</span>()</span> /</span> (</span>logits</span> -</span> logits</span>.</span>max</span>()).</span>exp</span>().</span>sum</span>()</span></span>
print</span>(</span>f</span>"naive  softmax([120,119,115]) = </span>{</span>naive</span>.</span>tolist</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"stable softmax([120,119,115]) = </span>{</span>[</span>round</span>(</span>v</span>,</span>4</span>)</span> for</span> v</span> in</span> stable</span>.</span>tolist</span>()]</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"torch.softmax agrees with stable: "</span></span>
      f</span>"</span>{</span>torch</span>.</span>allclose</span>(</span>torch</span>.</span>softmax</span>(</span>logits</span>,</span> 0</span>),</span> stable</span>)</span>}</span>"</span>)</span></span>
</span>
# logsumexp, same illness, same cure</span></span>
print</span>(</span>f</span>"naive  log(sum(exp)) = </span>{</span>logits</span>.</span>exp</span>().</span>sum</span>().</span>log</span>().</span>item</span>()</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"stable logsumexp     = </span>{</span>torch</span>.</span>logsumexp</span>(</span>logits</span>,</span> 0</span>).</span>item</span>()</span>:.4f</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
The same pattern explains the rest of the family. logsumexp</code>
slides by the max for the same reason. log1p(x)</code> and expm1(x)</code>
exist because near zero, 1 + x absorbs x (the absorption figure
again, at 1.0 instead of 1e8). The layer-norm epsilon keeps a
near-zero variance from turning rsqrt</code> into inf; Adam's epsilon
does the same for its denominator; and an attention mask uses a
large negative number so that after the slide it underflows to a
clean zero instead of poisoning the row with nan. One picture,
many patches: keep the intermediate values where the points are.</p>
Why smaller floats</a></h2>
The guide paused at 16 bits. Here is why the family keeps
going down anyway.</p>
Training a network is mostly moving numbers, not multiplying
them: weights travel from memory to the arithmetic and gradients
travel back, billions per step, and the wire is slower than the
multiplier. Halve the bits per number and the same wire carries
twice the numbers, the same memory holds twice the model, and the
tensor cores, built to process the narrower type, double their
arithmetic too. Here is the wire, drawn to scale:</p>

  
    
    
  </div>
  Figure 31.</span> the pipes, to one scale (an H100's numbers [30]). the thin line is how the weights arrive; the wide pipe is what every step and every token pays. the byte ladders ahead shrink what crosses it.</figcaption>
</figure>
The only question is whether the surviving points still cover
the numbers the network produces with small enough error.
Networks tolerate rounding noise unusually well: they are
trained on noisy batches and judged over many outputs, so the
answer stays yes far below 32 bits if the format is chosen
carefully. The fork figure already showed the first halving's
two options, and training mostly picked bfloat16: reach first.
The cost is a step of 0.0078 at 1.0, and the next section shows
what that step does to learning.</p>
The update that vanished</a></h2>
At 16 bits the map's steps are wide enough to stop training
itself. A bfloat16 weight sitting at 1.0 has neighbors
0.0078 away. A healthy update of 0.001 is an arrow one eighth of
that step:</p>

  
    
    
  </div>
  Figure 32.</span> the update that vanished, and the repair. deterministic rounding silently stops training for that weight; stochastic rounding keeps it moving on average.</figcaption>
</figure>
The proof runs it: 1,000 consecutive updates of 0.001
applied to a bfloat16 weight leave it at exactly 1.0
(proof</a>). This is why every mixed-precision
recipe since 2017 keeps a float32 master copy of the weights: the
forward and backward passes run narrow, but the optimizer adds the
update into a copy whose local step is 65536 times finer, where
0.001 lands easily [5</a>]</span>. The other repair is
stochastic rounding, which we will need again at 4 bits: round
up with probability proportional to how far you got. In the same
proof, the stochastically rounded weight reaches 2.0 alongside the
master copy.</p>

  
    </span>
    p8_lost_update.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the update that vanished. In bf16, a weight near 1.0 has a</span></span>
local step of about 0.0078; an update smaller than half that step</span></span>
rounds away, and training silently stops for that weight. The fp32</span></span>
master copy is the fix; stochastic rounding is the other fix.</span>"""</span></span>
import</span> torch</span></span>
</span>
w</span> =</span> torch</span>.</span>tensor</span>(</span>1.0</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)</span></span>
u</span> =</span> 1e-3</span>                                   # lr * grad, a normal size</span></span>
print</span>(</span>f</span>"bf16 next after 1.0 : </span>{</span>torch</span>.</span>nextafter</span>(</span>torch</span>.</span>tensor</span>(</span>1.0</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>),</span> torch</span>.</span>tensor</span>(</span>2.0</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)).</span>item</span>()</span>:.6f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"w + </span>{</span>u</span>}</span> in bf16     : </span>{</span>(</span>w</span> +</span> torch</span>.</span>tensor</span>(</span>u</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)).</span>item</span>()</span>:.6f</span>}</span>  (unchanged: </span>{</span>(</span>w</span> +</span> torch</span>.</span>tensor</span>(</span>u</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>))</span> ==</span> w</span>}</span>)"</span>)</span></span>
</span>
# 1000 such updates, three ways</span></span>
steps</span> =</span> 1000</span></span>
wb</span> =</span> torch</span>.</span>tensor</span>(</span>1.0</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)</span></span>
for</span> _</span> in</span> range</span>(</span>steps</span>):</span></span>
    wb</span> =</span> wb</span> +</span> torch</span>.</span>tensor</span>(</span>u</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)</span></span>
wm</span> =</span> torch</span>.</span>tensor</span>(</span>1.0</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span>     # master copy</span></span>
for</span> _</span> in</span> range</span>(</span>steps</span>):</span></span>
    wm</span> =</span> wm</span> +</span> u</span></span>
g</span> =</span> torch</span>.</span>Generator</span>().</span>manual_seed</span>(</span>0</span>)</span>            # stochastic rounding, emulated</span></span>
ws</span> =</span> torch</span>.</span>tensor</span>(</span>1.0</span>,</span> dtype</span>=</span>torch</span>.</span>float32</span>)</span></span>
for</span> _</span> in</span> range</span>(</span>steps</span>):</span></span>
    hi</span> =</span> torch</span>.</span>nextafter</span>(</span>ws</span>.</span>to</span>(</span>torch</span>.</span>bfloat16</span>),</span> torch</span>.</span>tensor</span>(</span>2.0</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)).</span>float</span>()</span></span>
    lo</span> =</span> ws</span>.</span>to</span>(</span>torch</span>.</span>bfloat16</span>).</span>float</span>()</span></span>
    x</span> =</span> ws</span> +</span> u</span></span>
    p</span> = ((</span>x</span> -</span> lo</span>)</span> /</span> (</span>hi</span> -</span> lo</span>)).</span>clamp</span>(</span>0</span>,</span> 1</span>)</span></span>
    ws</span> =</span> hi</span> if</span> torch</span>.</span>rand</span>((),</span> generator</span>=</span>g</span>)</span> <</span> p</span> else</span> lo</span></span>
print</span>(</span>f</span>"after </span>{</span>steps</span>}</span> updates of </span>{</span>u</span>}</span>:"</span>)</span></span>
print</span>(</span>f</span>"  bf16 accumulate    : </span>{</span>wb</span>.</span>item</span>()</span>:.4f</span>}</span>   (true answer 2.0)"</span>)</span></span>
print</span>(</span>f</span>"  fp32 master        : </span>{</span>wm</span>.</span>item</span>()</span>:.4f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"  bf16 + stochastic  : </span>{</span>ws</span>.</span>item</span>()</span>:.4f</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
So the loop that trains every model you use runs like this:</p>

  
    
    
  </div>
  Figure 33.</span> one training step as it actually runs, all 10 stations. narrow bits where the volume is; float32 where errors accumulate; the S dial and the red inf check guard the float16 path.</figcaption>
</figure>
The products accumulate in float32 inside the tensor cores: that
is the fused multiply-add from the rounding section doing its
job. Precision goes exactly where the numbers need it and
nowhere else. Stations 6 and 8, the S dial and the inf check,
are the subject of the next section.</p>
Loss scaling</a></h2>
One group of numbers still fails inside this loop. Gradients
are the smallest numbers in training, and float16's floor is
high: its
subnormals end near 6e-8 and its normal range starts at 6e-5.
The left tail of the gradient distribution simply falls below
every point of the format:</p>

  
    
    
  </div>
  Figure 34.</span> the histogram meets the map. multiplying the loss by one number slides every gradient into the band; dividing afterward puts them back.</figcaption>
</figure>
The rescue is one multiplication. Scale the loss by S before the
backward pass and, because differentiation is linear, every
gradient arrives multiplied by S; unscale after casting and
nothing has changed except which gradients survived. In practice S adapts:
overflow detected, halve it and skip the step; a run of clean
steps, double it. That is the whole of GradScaler</code>
(proof</a> measures the cast; on our toy
net only 0.1% of gradients die, and the fraction grows with depth
and training time, which is why the recipe exists
[5</a>]</span>). bfloat16 mostly
retired this machinery: it keeps float32's reach, so its normal
floor sits at 1.2e-38 against float16's 6.1e-5, about 34 powers
of 10 lower, and the scaler became optional.</p>

  
    </span>
    p9_loss_scaling.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: loss scaling, measured. Take real float32 gradients from a</span></span>
small net, cast them to float16 as mixed precision must, and count</span></span>
the ones that flush to zero; then scale by 1024 first, cast, unscale.</span></span>
The cast is where gradients die; the scale is the rescue.</span>"""</span></span>
import</span> torch</span>,</span> torch</span>.</span>nn</span> as</span> nn</span></span>
torch</span>.</span>manual_seed</span>(</span>0</span>)</span></span>
net</span> =</span> nn</span>.</span>Sequential</span>(</span>nn</span>.</span>Linear</span>(</span>64</span>,</span> 256</span>),</span> nn</span>.</span>ReLU</span>(),</span></span>
                    nn</span>.</span>Linear</span>(</span>256</span>,</span> 256</span>),</span> nn</span>.</span>ReLU</span>(),</span> nn</span>.</span>Linear</span>(</span>256</span>,</span> 1</span>))</span></span>
X</span> =</span> torch</span>.</span>randn</span>(</span>2048</span>,</span> 64</span>)</span>; y</span> =</span> X</span>[:, :</span>3</span>].</span>prod</span>(</span>1</span>,</span> keepdim</span>=</span>True</span>)</span></span>
loss</span> = ((</span>net</span>(</span>X</span>)</span> -</span> y</span>)</span> **</span> 2</span>).</span>mean</span>()</span></span>
loss</span>.</span>backward</span>()</span></span>
g</span> =</span> torch</span>.</span>cat</span>([</span>p</span>.</span>grad</span>.</span>flatten</span>()</span> for</span> p</span> in</span> net</span>.</span>parameters</span>()])</span></span>
nz</span> =</span> g</span>[</span>g</span> !=</span> 0</span>]</span></span>
plain</span> = (</span>nz</span>.</span>half</span>()</span> ==</span> 0</span>).</span>float</span>().</span>mean</span>()</span></span>
scaled</span> = ((</span>nz</span> *</span> 1024</span>).</span>half</span>().</span>float</span>()</span> /</span> 1024</span> ==</span> 0</span>).</span>float</span>().</span>mean</span>()</span></span>
print</span>(</span>f</span>"nonzero float32 gradients            : </span>{</span>len</span>(</span>nz</span>)</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"flushed to zero by the float16 cast  : </span>{</span>plain</span>:.1%</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"same cast after scaling by 1024      : </span>{</span>scaled</span>:.1%</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"smallest surviving gradient          : </span>{</span>nz</span>.</span>abs</span>().</span>min</span>().</span>item</span>()</span>:.2e</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"float16's smallest subnormal         : </span>{</span>torch</span>.</span>finfo</span>(</span>torch</span>.</span>float16</span>).</span>smallest_normal</span> *</span> 2</span>**-</span>10</span>:.2e</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
FP8</a></h2>
Halve again and there are not enough bits left for one format
to serve both directions of training. So at 8 bits the fork
appears one more time, and this time both forks ship, as a
pair [6</a>]</span>: E4M3 keeps digits, E5M2 keeps reach.</p>

  
    
    
  </div>
  Figure 35.</span> E4M3's card. the map section already drew all 256 of its patterns; this is the same format as a card.</figcaption>
</figure>
E4M3 you have known since the map section: its 15 intervals are
this page's teaching map, and its 256 patterns are decoded in
proof</a>. The card records its one non-IEEE
choice: no infinity. Those codes were used for one more
doubling of range, overflow stops at 448, and a single pattern
means NaN.</p>

  
    
    
  </div>
  Figure 36.</span> E5M2's card: float16's 5 exponent bits kept, 8 of its 10 mantissa bits cut.</figcaption>
</figure>
E5M2 is float16 with the bottom byte cut, and that is a
bit-level fact, provable the same way bfloat16 was: all 256
E5M2 patterns cast to float16 as the same 8 bits with 8 zeros
appended (proof</a> checks every one).</p>

  
    
    
  </div>
  Figure 37.</span> the E5M2 map: float16's 30 intervals with 4 points in each. reach kept, digits nearly gone.</figcaption>
</figure>

  
    </span>
    p20_e5m2_top_byte.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: FP8 E5M2 is float16's top byte, at the bit level. The</span></span>
same fact as bfloat16 being float32's top half, one floor down:</span></span>
E5M2 keeps float16's 5 exponent bits and the top 2 of its 10</span></span>
mantissa bits. Cast any of the 256 E5M2 bit patterns to float16</span></span>
and the result is the same 8 bits with 8 zeros appended.</span></span>
"""</span></span>
import</span> torch</span></span>
</span>
pats</span> =</span> torch</span>.</span>arange</span>(</span>256</span>,</span> dtype</span>=</span>torch</span>.</span>uint8</span>)</span></span>
as_e5m2</span> =</span> pats</span>.</span>view</span>(</span>torch</span>.</span>float8_e5m2</span>)</span></span>
up_bits</span> =</span> as_e5m2</span>.</span>half</span>().</span>view</span>(</span>torch</span>.</span>int16</span>)</span> & 0x</span>FFFF</span></span>
want</span> =</span> pats</span>.</span>to</span>(</span>torch</span>.</span>int16</span>)</span> <<</span> 8</span></span>
nan_mask</span> =</span> torch</span>.</span>isnan</span>(</span>as_e5m2</span>)</span></span>
ok</span> = (</span>up_bits</span> ==</span> want</span>)</span> |</span> nan_mask</span></span>
print</span>(</span>f</span>"up-cast bit check: </span>{</span>ok</span>.</span>sum</span>().</span>item</span>()</span>}</span> / 256 patterns match "</span></span>
      f</span>"(pattern << 8)"</span>)</span></span>
print</span>(</span>f</span>"</span>  {</span>nan_mask</span>.</span>sum</span>().</span>item</span>()</span>}</span> NaN codes, all up-cast to "</span></span>
      f</span>"float16 NaN: </span>{</span>torch</span>.</span>isnan</span>(</span>as_e5m2</span>.</span>half</span>()[</span>nan_mask</span>]).</span>all</span>().</span>item</span>()</span>}</span>"</span>)</span></span>
exact</span> = (</span>up_bits</span> ==</span> want</span>).</span>sum</span>().</span>item</span>()</span></span>
print</span>(</span>f</span>"  bit-exact including NaN payloads: </span>{</span>exact</span>}</span> / 256"</span>)</span></span>
</span>
v</span> =</span> torch</span>.</span>tensor</span>(</span>1.7014</span>,</span> dtype</span>=</span>torch</span>.</span>float16</span>)</span></span>
b8</span> =</span> v</span>.</span>to</span>(</span>torch</span>.</span>float8_e5m2</span>).</span>view</span>(</span>torch</span>.</span>uint8</span>).</span>item</span>()</span></span>
u16</span> =</span> v</span>.</span>view</span>(</span>torch</span>.</span>int16</span>).</span>item</span>()</span> & 0x</span>FFFF</span></span>
print</span>(</span>f</span>"</span>\n</span>1.7014 in float16 bits: </span>{</span>u16</span>:016b</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"1.7014 in E5M2 bits   : </span>{</span>b8</span>:08b</span>}</span> (the top byte, rounded)"</span>)</span></span>
print</span>(</span>f</span>"E5M2 value stored     : "</span></span>
      f</span>"</span>{</span>v</span>.</span>to</span>(</span>torch</span>.</span>float8_e5m2</span>).</span>float</span>().</span>item</span>()</span>:.4f</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
NVIDIA, Arm and Intel proposed the pair for deep learning in
2022 [6</a>]</span>, and hardware arrived with the Hopper chips: E4M3
carries the forward pass, E5M2 carries the gradients, whose
tails need reach more than digits. Each tensor also carries a
float32 scale, chosen from the tensor's recent maximum so that
its histogram slides into the band: exactly the loss-scaling
picture, applied per tensor. Recipes differ only in whether the
maximum is tracked from history (delayed scaling) or measured
on the spot:</p>

  
    
    
  </div>
  Figure 38.</span> where a tensor's FP8 scale comes from: the loss-scaling dial, automated per tensor.</figcaption>
</figure>
And not everything is quantized: attention's and
the MLP's big matmuls run in FP8, while softmax, the
normalizations, the first embedding and the final projection
stay wide [11</a>]</span>.</p>
Wired into a transformer through NVIDIA's Transformer Engine,
the published result is training runs 30 to 40 percent faster
at matching loss curves [6</a>]</span>, and DeepSeek-V3 trained a
671-billion-parameter model with FP8 matmuls by scaling
128-element tiles instead of whole tensors [7</a>]</span>, a step toward
what comes next. One caution before the next halving: E4M3
carries about 1.2 decimal digits and E5M2 about 0.9, so at 8
bits no single stored number is trustworthy; only averages over
many are.</p>
The outlier problem</a></h2>
Before the next halving, meet the format waiting at the bottom,
because its 2 numbers explain everything that follows. 4 bits:</p>

  
    
    
  </div>
  Figure 39.</span> E2M1's card. 16 codes; the next drawing shows every one of them.</figcaption>
</figure>

  
    
    
  </div>
  Figure 40.</span> every value of a 4-bit E2M1 float: 15 numbers plus a second zero. you have now seen all of them.</figcaption>
</figure>
16 codes, 15 values (0 appears twice, once with each sign), no
inf, no NaN, and a total width of 6 / 0.5 = 12x, about 3.6
doublings. The 2023 OCP standard defined E2M1 as the 4-bit
element of the block family these sections are building [9</a>]</span>, and the
newest NVIDIA chips run it at twice the FP8 rate [24</a>]</span>. Alone it
is unusable, and even a per-tensor scale cannot save it,
because per-tensor scaling has one enemy. Transformer
activations grow a few channels whose values run hundreds of
times larger than the rest, systematically, past about 6
billion parameters [8</a>]</span>. One scale must now serve the spike and
the bell at once:</p>

  
    
    
  </div>
  Figure 41.</span> one spike, three formats, measured. FP8 loses 1%; FP4 with one scale loses the entire bell; blocks are the rescue.</figcaption>
</figure>
Read the measured numbers. FP8 with a single scale loses 1% of
the ordinary values next to a 500x outlier, because a float grid
keeps its relative error at any scale until values fall below
its smallest positive number, and FP8's smallest is 229,376x
under its largest (448 / 0.00195, from the map section). FP4's
smallest is only 12x under its largest (6 / 0.5, computed
above). One spike takes the whole 12x, and
100% of the ordinary values quantize to zero
(proof</a>). At 4 bits, blocks are not an
improvement; nothing works without them.</p>

  
    </span>
    p10_fp4_outlier.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: why blocks had to be invented. A float grid keeps its</span></span>
relative precision at any scale, so FP8 with one per-tensor scale</span></span>
survives an outlier surprisingly well. FP4 does not: its largest</span></span>
positive value is 6 and its smallest is 0.5, only 12x apart, so one</span></span>
spike pushes every ordinary value below the smallest, and the</span></span>
tensor quantizes to zeros. A scale per 32 elements</span></span>
rescues it.</span>"""</span></span>
import</span> torch</span></span>
</span>
E2M1</span> =</span> torch</span>.</span>tensor</span>([</span>0.0</span>,</span> 0.5</span>,</span> 1.0</span>,</span> 1.5</span>,</span> 2.0</span>,</span> 3.0</span>,</span> 4.0</span>,</span> 6.0</span>])</span></span>
</span>
def</span> round_e2m1</span>(</span>v</span>):</span></span>
    """</span>Round |v| to the nearest E2M1 magnitude, keep the sign.</span>"""</span></span>
    idx</span> = (</span>v</span>.</span>abs</span>().</span>unsqueeze</span>(</span>-</span>1</span>)</span> -</span> E2M1</span>).</span>abs</span>().</span>argmin</span>(</span>-</span>1</span>)</span></span>
    return</span> E2M1</span>[</span>idx</span>]</span> *</span> v</span>.</span>sign</span>()</span></span>
</span>
def</span> quant</span>(</span>x</span>,</span> block</span>):</span></span>
    q</span> =</span> torch</span>.</span>empty_like</span>(</span>x</span>)</span></span>
    for</span> i</span> in</span> range</span>(</span>0</span>,</span> len</span>(</span>x</span>),</span> block</span>):</span></span>
        b</span> =</span> x</span>[</span>i</span>:</span>i</span>+</span>block</span>]</span></span>
        s</span> =</span> b</span>.</span>abs</span>().</span>max</span>()</span> /</span> 6.0</span></span>
        s</span> =</span> s</span> if</span> s</span> ></span> 0</span> else</span> torch</span>.</span>tensor</span>(</span>1.0</span>)</span></span>
        q</span>[</span>i</span>:</span>i</span>+</span>block</span>] =</span> round_e2m1</span>(</span>b</span> /</span> s</span>)</span> *</span> s</span></span>
    return</span> q</span></span>
</span>
torch</span>.</span>manual_seed</span>(</span>0</span>)</span></span>
x</span> =</span> torch</span>.</span>randn</span>(</span>128</span>)</span> *</span> 0.1</span></span>
x</span>[</span>40</span>] =</span> 50.0</span></span>
mask</span> =</span> torch</span>.</span>ones</span>(</span>128</span>,</span> dtype</span>=</span>torch</span>.</span>bool</span>)</span>; mask</span>[</span>40</span>] =</span> False</span></span>
</span>
for</span> name</span>,</span> block</span> in</span> ((</span>"</span>per-tensor</span>"</span>,</span> 128</span>), (</span>"</span>per-block-32</span>"</span>,</span> 32</span>)):</span></span>
    q</span> =</span> quant</span>(</span>x</span>,</span> block</span>)</span></span>
    zeros</span> = (</span>q</span>[</span>mask</span>]</span> ==</span> 0</span>).</span>float</span>().</span>mean</span>()</span></span>
    rel</span> = ((</span>q</span> -</span> x</span>)[</span>mask</span>].</span>abs</span>()</span> /</span> x</span>[</span>mask</span>].</span>abs</span>()).</span>median</span>()</span></span>
    print</span>(</span>f</span>"</span>{</span>name</span>:14s</span>}</span> ordinary values quantized to zero: </span>{</span>zeros</span>:.0%</span>}</span>   "</span></span>
          f</span>"median rel err of the rest: </span>{</span>rel</span>:.1%</span>}</span>"</span>)</span></span>
q</span> =</span> quant</span>(</span>x</span>,</span> 128</span>)</span></span>
print</span>(</span>f</span>"the outlier itself survives either way: </span>{</span>q</span>[</span>40</span>]</span>:.1f</span>}</span>"</span>)</span></span>
print</span>()</span></span>
print</span>(</span>"</span>the same tensor in FP8 E4M3, one per-tensor scale, for honesty:</span>"</span>)</span></span>
s</span> =</span> x</span>.</span>abs</span>().</span>max</span>()</span> /</span> 448.0</span></span>
q8</span> = (</span>x</span> /</span> s</span>).</span>to</span>(</span>torch</span>.</span>float8_e4m3fn</span>).</span>float</span>()</span> *</span> s</span></span>
print</span>(</span>f</span>"per-tensor FP8    zeros: </span>{</span>(</span>q8</span>[</span>mask</span>]</span>==</span>0</span>).</span>float</span>().</span>mean</span>()</span>:.0%</span>}</span>   "</span></span>
      f</span>"median rel err: </span>{</span>((</span>q8</span>-</span>x</span>)[</span>mask</span>].</span>abs</span>()</span>/</span>x</span>[</span>mask</span>].</span>abs</span>()).</span>median</span>()</span>:.1%</span>}</span>"</span>)</span></span>
print</span>(</span>"</span>a float grid holds its relative error at any scale, until its</span>"</span>)</span></span>
print</span>(</span>"</span>smallest positive value; FP4's smallest (0.5) is only 12x below</span>"</span>)</span></span>
print</span>(</span>"</span>its largest (6), and that is the whole reason microscaling exists.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Microscaling</a></h2>
So the scale moved into the data type. The Open Compute Project's
MX formats, standardized in 2023 by AMD, Arm, Intel, Meta,
Microsoft, NVIDIA and Qualcomm [9</a>]</span>, cut every tensor
into blocks of 32 and give each block one 8-bit scale:</p>

  
    
    
  </div>
  Figure 42.</span> the MX block. 32 elements, one shared scale, consumed whole by the tensor cores. the scale is an exponent with no mantissa: an interval-picker and nothing else.</figcaption>
</figure>
Every idea on this page meets here. The E8M0 scale is a float
with no mantissa at all, the interval choice from the floating-dot
figure with the position deleted, so applying it is exponent
addition, free in silicon. The 32-element block is the histogram
cut fine enough that an outlier only damages its own block.
The accounting is 136 bits per 32 numbers: 4.25 bits each,
136 / 128 = 1.06, so 6% more than raw 4-bit storage. And the block size is a measured compromise: smaller blocks
mean less error and more storage spent on scales, and our own
sweep across sizes 8, 16, 32, 64, 128 shows both moving (proof</a>). The same
chassis carries 8-, 6- and 4-bit elements as MXFP8, MXFP6,
MXFP4.</p>
What the tensor core does with 2 of these blocks is 1 more
drawing:</p>

  
    
    
  </div>
  Figure 43.</span> 2 MX blocks meet in the tensor core: the 32 element pairs multiply and sum in float32; the 2 scale bytes add as exponents and touch the sum once per block.</figcaption>
</figure>
The block chassis also changed who does what inside FP8: with a
scale for every 32 values, a block rarely spans more than E4M3's
18 doublings, so E4M3, with its 8 points per doubling against
E5M2's 4, measured better for the gradients too, and the largest
published MXFP8 pretraining run (8 billion parameters, 15
trillion tokens) quantizes every tensor as E4M3 [11</a>]</span>.</p>
The 6-bit elements deserve their own cards, because they are the
family's open seats: E2M3 keeps digits, E3M2 keeps reach, one
more run of the fork.</p>

  
    
    
  </div>
  Figure 44.</span> FP6 E2M3's card: 3 mantissa bits, reach of only 5.9 doublings.</figcaption>
</figure>

  
    
    
  </div>
  Figure 45.</span> all of E2M3: 3 intervals of 8 points, 7.5 down to 0.125.</figcaption>
</figure>

  
    
    
  </div>
  Figure 46.</span> FP6 E3M2's card: 1 more exponent bit, reach of 8.8 doublings.</figcaption>
</figure>

  
    
    
  </div>
  Figure 47.</span> all of E3M2: 7 intervals of 4 points, 28 down to 0.0625.</figcaption>
</figure>
Their maps are small enough to draw complete: 3 intervals of 8
points, 7 intervals of 4. Blackwell-class chips execute them at
full speed as MXFP6 [9</a>][24</a>]</span>; almost no published recipe uses
them yet. A format survives by having a job nobody else does.
FP8 is safer, FP4 is faster on paper, and FP6 is still looking
for its job.</p>
NVIDIA's NVFP4 variant tightens it further: 16-element blocks, a
fractional E4M3 scale instead of a power-of-two one, plus one
float32 scale per tensor, buying a finer fit for a quarter bit
more [10</a>]</span>. The bytes carry the quarter bit, and the deeper
difference sits in the scale byte itself:</p>

  
    
    
  </div>
  Figure 48.</span> the 2 block families as bytes. 4.25 against 4.5 bits per value, and the deeper difference: an E4M3 scale has mantissa bits, so it can be fractional.</figcaption>
</figure>
Here is one NVFP4 block quantized by hand, every scale shown:</p>

  
    
    
  </div>
  Figure 49.</span> one NVFP4 block, every scale shown. the largest value sets the scales and survives exactly; the smallest fall to 0; the middle carries the percent-level error.</figcaption>
</figure>
The largest of the 16 values, 15.011, sets the scales and comes
back exactly; 0.25 and 0.5 fall below half of the smallest step
at that scale and come back as 0; 3.2002 comes back as 3.7528,
about 17% off (proof</a> runs every
number). A single 4-bit value is coarse. The bet of 4-bit
arithmetic is that billions of such errors, kept unbiased,
cancel on average.</p>

  
    </span>
    p19_nvfp4_worked.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: one NVFP4 block, quantized by hand, every scale shown.</span></span>
Re-derives the worked 1 x 16 example from NVIDIA's recipe (as</span></span>
walked through by Radical Numerics): a global FP32 scale brings</span></span>
the tensor's largest value into range, each 16-value block gets an</span></span>
FP8 E4M3 scale, and the elements land on the 16 values of E2M1.</span></span>
"""</span></span>
import</span> torch</span></span>
</span>
E2M1_MAX</span> =</span> 6.0</span></span>
E4M3_MAX</span> =</span> 448.0</span></span>
BLOCK</span> =</span> 16</span></span>
E2M1_GRID</span> =</span> torch</span>.</span>tensor</span>([</span>0.0</span>,</span> 0.5</span>,</span> 1.0</span>,</span> 1.5</span>,</span> 2.0</span>,</span> 3.0</span>,</span> 4.0</span>,</span> 6.0</span>])</span></span>
</span>
def</span> rne_e2m1</span>(</span>x</span>):</span></span>
    """</span>Round each value to the nearest E2M1 value, ties to even.</span>"""</span></span>
    grid</span> =</span> torch</span>.</span>cat</span>([</span>E2M1_GRID</span>,</span> -</span>E2M1_GRID</span>.</span>flip</span>(</span>0</span>)])</span></span>
    d</span> = (</span>x</span>.</span>unsqueeze</span>(</span>-</span>1</span>)</span> -</span> grid</span>).</span>abs</span>()</span></span>
    return</span> grid</span>[</span>d</span>.</span>argmin</span>(</span>dim</span>=</span>-</span>1</span>)]</span></span>
</span>
x</span> =</span> torch</span>.</span>tensor</span>([</span>0.0</span>,</span> 0.25</span>,</span> 0.5</span>,</span> 0.75356</span>,</span> 1.251245</span>,</span> 3.2002</span>,</span></span>
                  4.5032</span>,</span> 15.011</span>,</span> 0.012</span>,</span> -</span>0.312</span>,</span> -</span>5.50055</span>,</span> 10.06</span>,</span></span>
                  -</span>1.2526</span>,</span> 3.025</span>,</span> 2.5114</span>,</span> 7.0162</span>])</span></span>
</span>
# level 1: one FP32 scale for the whole tensor</span></span>
global_amax</span> =</span> x</span>.</span>abs</span>().</span>max</span>()</span></span>
s_enc</span> = (</span>E4M3_MAX</span> *</span> E2M1_MAX</span>)</span> /</span> global_amax</span></span>
s_dec</span> =</span> 1.0</span> /</span> s_enc</span></span>
</span>
# level 2: one E4M3 scale for each block of 16</span></span>
block_amax</span> =</span> x</span>.</span>abs</span>().</span>max</span>()</span>                    # one block here</span></span>
dec_scale</span> = (</span>block_amax</span> /</span> E2M1_MAX</span>)</span> *</span> s_enc</span></span>
dec_scale_e4m3</span> =</span> dec_scale</span>.</span>to</span>(</span>torch</span>.</span>float8_e4m3fn</span>)</span></span>
enc_scale</span> =</span> 1.0</span> /</span> (</span>dec_scale_e4m3</span>.</span>float</span>()</span> *</span> s_dec</span>)</span></span>
print</span>(</span>f</span>"level 1, the tensor's float32 scale: 448 x 6 / "</span></span>
      f</span>"</span>{</span>global_amax</span>:.4f</span>}</span> = </span>{</span>s_enc</span>:.2f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"level 2, this block's E4M3 scale: "</span></span>
      f</span>"</span>{</span>dec_scale_e4m3</span>.</span>float</span>()</span>:g</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"together: </span>{</span>s_enc</span>:.2f</span>}</span> / </span>{</span>dec_scale_e4m3</span>.</span>float</span>()</span>:g</span>}</span> = "</span></span>
      f</span>"</span>{</span>enc_scale</span>:.4f</span>}</span>, the factor every value is multiplied by"</span>)</span></span>
</span>
# quantize: scale, clamp to E2M1's range, round onto its 16 values</span></span>
scaled</span> = (</span>x</span> *</span> enc_scale</span>).</span>clamp</span>(</span>-</span>E2M1_MAX</span>,</span> E2M1_MAX</span>)</span></span>
q</span> =</span> rne_e2m1</span>(</span>scaled</span>)</span></span>
</span>
# dequantize: multiply back by both scales</span></span>
dq</span> =</span> q</span> *</span> dec_scale_e4m3</span>.</span>float</span>()</span> *</span> s_dec</span></span>
</span>
print</span>(</span>f</span>"global amax </span>{</span>global_amax</span>:.4f</span>}</span>   block encode scale "</span></span>
      f</span>"</span>{</span>enc_scale</span>:.4f</span>}\n</span>"</span>)</span></span>
print</span>(</span>f</span>"</span>{</span>'</span>input</span>'</span>:>9s</span>} {</span>'</span>scaled</span>'</span>:>8s</span>} {</span>'</span>E2M1</span>'</span>:>6s</span>} {</span>'</span>back</span>'</span>:>9s</span>}</span>"</span>)</span></span>
for</span> a</span>,</span> b</span>,</span> c</span>,</span> d</span> in</span> zip</span>(</span>x</span>,</span> scaled</span>,</span> q</span>,</span> dq</span>):</span></span>
    print</span>(</span>f</span>"</span>{</span>a</span>:>9.4f</span>} {</span>b</span>:>8.4f</span>} {</span>c</span>:>6.1f</span>} {</span>d</span>:>9.4f</span>}</span>"</span>)</span></span>
</span>
print</span>(</span>"</span>\n</span>read the two ends: 15.011 comes back as 15.011 exactly</span>"</span>)</span></span>
print</span>(</span>"</span>(it set the scales), 0.25 and 0.5 come back as 0 (they</span>"</span>)</span></span>
print</span>(</span>"</span>fell below half of E2M1's smallest step at this scale),</span>"</span>)</span></span>
print</span>(</span>"</span>3.2002 comes back as 3.7528: about 17</span>% o</span>ff. one 4-bit</span>"</span>)</span></span>
print</span>(</span>"</span>number is coarse; the training recipe works because these</span>"</span>)</span></span>
print</span>(</span>"</span>errors average out over billions of them.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
One subtlety earned a hardware rule. The E8M0 scale can only be
a power of 2, so a block whose largest value is 5.9 must round
its scale one way or the other, and the 2 ways are not close:</p>

  
    
    
  </div>
  Figure 50.</span> the scale-rounding rule: round the block scale up. the first spec said down, and the difference is a failed training run.</figcaption>
</figure>
NVIDIA measured the difference as the gap between divergence and
parity at trillion-token scale [11</a>]</span>; proof</a>
shows the mechanism in 2 lines.</p>
The whole scaling story now fits on one ladder, and every recipe
of the last 5 sections is 1 rung of it:</p>

  
    
    
  </div>
  Figure 51.</span> the granularity ladder. down it, an outlier damages less; up it, fewer bytes go to scales. every scaling recipe on this page is 1 rung.</figcaption>
</figure>
Training at 4 bits</a></h2>
The destination format, E2M1, was drawn whole 2 sections back:
15 values, 12x wide. Training inside those 16 codes
needs 3 rescues beyond blocks,
and each is an idea from earlier on this page pushed one step:</p>
The first is stochastic rounding, the lost-update repair now
doing the main work. At 4 bits nearly every update is below half the
local step, so deterministic rounding freezes everything; rounding
up with probability proportional to progress is unbiased in
expectation. Our proof runs 10,000 micro-updates: nearest
rounds to a frozen 1.0, stochastic reaches a mean of 3.0113
against a true 3.0, noisy per run and honest on average
(proof</a>).</p>

  
    </span>
    p12_sr_drift.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: round-to-nearest is biased when every step is smaller than</span></span>
half the local spacing; stochastic rounding is unbiased in</span></span>
expectation. The same fact that freezes bf16 weights, isolated.</span>"""</span></span>
import</span> torch</span></span>
N</span>,</span> u</span>,</span> TRIALS</span> =</span> 10000</span>,</span> 2e-4</span>,</span> 50</span>   # each step ~1/39 of bf16's step at 1.0</span></span>
</span>
def</span> sr_walk</span>(</span>seed</span>):</span></span>
    g</span> =</span> torch</span>.</span>Generator</span>().</span>manual_seed</span>(</span>seed</span>)</span></span>
    val</span> =</span> 1.0</span></span>
    for</span> _</span> in</span> range</span>(</span>N</span>):</span></span>
        x</span> =</span> val</span> +</span> u</span></span>
        lo</span> =</span> torch</span>.</span>tensor</span>(</span>x</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)</span></span>
        lo_f</span> =</span> lo</span>.</span>float</span>().</span>item</span>()</span></span>
        hi_f</span> =</span> torch</span>.</span>nextafter</span>(</span>lo</span>,</span> torch</span>.</span>tensor</span>(</span>float</span>(</span>"</span>inf</span>"</span>),</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)).</span>float</span>().</span>item</span>()</span></span>
        if</span> lo_f</span> ></span> x</span>:</span></span>
            hi_f</span>,</span> lo_f</span> =</span> lo_f</span>,</span> torch</span>.</span>nextafter</span>(</span>lo</span>,</span> torch</span>.</span>tensor</span>(</span>float</span>(</span>"</span>-inf</span>"</span>),</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)).</span>float</span>().</span>item</span>()</span></span>
        p</span> =</span> 0.0</span> if</span> hi_f</span> ==</span> lo_f</span> else</span> (</span>x</span> -</span> lo_f</span>)</span> /</span> (</span>hi_f</span> -</span> lo_f</span>)</span></span>
        val</span> =</span> hi_f</span> if</span> torch</span>.</span>rand</span>((),</span> generator</span>=</span>g</span>).</span>item</span>()</span> <</span> p</span> else</span> lo_f</span></span>
    return</span> val</span></span>
</span>
rne</span> =</span> torch</span>.</span>tensor</span>(</span>1.0</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)</span></span>
for</span> _</span> in</span> range</span>(</span>N</span>):</span></span>
    rne</span> =</span> rne</span> +</span> torch</span>.</span>tensor</span>(</span>u</span>,</span> dtype</span>=</span>torch</span>.</span>bfloat16</span>)</span></span>
finals</span> =</span> torch</span>.</span>tensor</span>([</span>sr_walk</span>(</span>s</span>)</span> for</span> s</span> in</span> range</span>(</span>TRIALS</span>)])</span></span>
true</span> =</span> 1.0</span> +</span> N</span> *</span> u</span></span>
print</span>(</span>f</span>"true value                     : </span>{</span>true</span>:.4f</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"round-to-nearest, </span>{</span>N</span>}</span> steps  : </span>{</span>rne</span>.</span>item</span>()</span>:.4f</span>}</span>  (frozen)"</span>)</span></span>
print</span>(</span>f</span>"stochastic rounding, </span>{TRIALS}</span> runs  : "</span></span>
      f</span>"mean </span>{</span>finals</span>.</span>mean</span>().</span>item</span>()</span>:.4f</span>}</span>, spread +-</span>{</span>finals</span>.</span>std</span>().</span>item</span>()</span>:.4f</span>}</span>"</span>)</span></span>
print</span>(</span>"</span>unbiased in expectation, noisy per run: that is the trade.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
The second is the random Hadamard transform: rotate the tensor by
an orthogonal matrix before quantizing and undo it after, which
changes no matmul (the rotation cancels) but spreads an outlier's
energy across every coordinate:</p>

  
    
    
  </div>
  Figure 52.</span> the rotation rescue, measured: max/mean falls from 11.8 to 2.6 and nothing about the mathematics moved.</figcaption>
</figure>
Our 64-value proof takes the max-to-mean ratio from 11.8 to 2.6
(proof</a>). The third is selective precision:
the first and last layers, where quantization error is measured
to hurt most, stay in higher precision. With all three plus
NVFP4, a 12-billion-parameter model has been pretrained on 10
trillion tokens at nearly the same loss as its FP8 twin
[10</a>]</span>; MXFP4 needs more tokens for the same loss
[12</a>]</span>. Where each rescue is applied is itself measured
engineering [10</a>][24</a>]</span>: stochastic rounding only on the gradients
(on the forward pass it adds noise for nothing, so weights and
activations round to nearest); the Hadamard rotation only on
the inputs of the weight-gradient matmul; and weights are
scaled in 16 x 16 squares rather than 1 x 16 rows, for a reason
worth drawing:</p>

  
    
    
  </div>
  Figure 53.</span> the transpose problem. blocks run along the reduction direction, so W and W-transposed disagree; squares are the truce, paid only for the weights.</figcaption>
</figure>
A row-scaled weight matrix and its column-scaled transpose are 2
different quantized matrices, and the forward and backward
passes must see the same weights. The last roughly 15% of
layers stay in higher precision, and if a gap to the wide
baseline remains at the end, switching the forward pass to
higher precision for the final phase of training closes most
of it [10</a>][24</a>]</span>. 4-bit training is real and it is young; treat
recipes as versioned, not settled.</p>

  
    </span>
    p13_hadamard.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: an orthogonal rotation spreads one outlier's energy across</span></span>
every coordinate, without changing the vector's length or the matmul</span></span>
it feeds (H is orthogonal). The max shrinks; the format's points get</span></span>
used again.</span>"""</span></span>
import</span> torch</span></span>
</span>
def</span> hadamard</span>(</span>n</span>):</span></span>
    H</span> =</span> torch</span>.</span>tensor</span>([[</span>1.0</span>]])</span></span>
    while</span> H</span>.</span>shape</span>[</span>0</span>]</span> <</span> n</span>:</span></span>
        H</span> =</span> torch</span>.</span>cat</span>([</span>torch</span>.</span>cat</span>([</span>H</span>,</span> H</span>],</span> 1</span>),</span> torch</span>.</span>cat</span>([</span>H</span>,</span> -</span>H</span>],</span> 1</span>)],</span> 0</span>)</span> /</span> (</span>2</span> **</span> 0.5</span>)</span></span>
    return</span> H</span></span>
</span>
torch</span>.</span>manual_seed</span>(</span>3</span>)</span></span>
n</span> =</span> 64</span></span>
x</span> =</span> torch</span>.</span>randn</span>(</span>n</span>)</span> *</span> 0.5</span></span>
x</span>[</span>7</span>],</span> x</span>[</span>23</span>],</span> x</span>[</span>51</span>] =</span> 8.0</span>,</span> -</span>7.0</span>,</span> 6.0</span></span>
H</span> =</span> hadamard</span>(</span>n</span>)</span></span>
y</span> =</span> x</span> @</span> H</span></span>
print</span>(</span>f</span>"norm preserved: </span>{</span>torch</span>.</span>allclose</span>(</span>x</span>.</span>norm</span>(),</span> y</span>.</span>norm</span>())</span>}</span>"</span>)</span></span>
for</span> name</span>,</span> v</span> in</span> ((</span>"</span>before</span>"</span>,</span> x</span>), (</span>"</span>after H</span>"</span>,</span> y</span>)):</span></span>
    print</span>(</span>f</span>"</span>{</span>name</span>:8s</span>}</span> max </span>{</span>v</span>.</span>abs</span>().</span>max</span>()</span>:.3f</span>}</span>   mean </span>{</span>v</span>.</span>abs</span>().</span>mean</span>()</span>:.3f</span>}</span>   "</span></span>
          f</span>"max/mean </span>{</span>(</span>v</span>.</span>abs</span>().</span>max</span>()</span>/</span>v</span>.</span>abs</span>().</span>mean</span>()).</span>item</span>()</span>:.2f</span>}</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
The other grids</a></h2>
Inference relaxed one constraint: if only the weights are
quantized and arithmetic runs wider, the 16 points need not
be a float at all. 3 designs compete on the same bell-shaped distribution of
trained weights:</p>

  
    
    
  </div>
  Figure 54.</span> 16 points, 3 designs, one weight distribution, measured. equal spacing wastes the tails; the float grid crowds zero; the quantile grid follows the weights.</figcaption>
</figure>
INT4 spaces its points equally and wastes them where weights
are rare. FP4 doubles its steps and crowds zero. NF4, from the QLoRA work,
places its 16 values at the quantiles of a gaussian, which is
approximately what trained weights are, and wins the tail
(proof</a>, [13</a>]</span>). Around these grids
grew the post-training toolkit: GPTQ chooses each weight's
rounding to compensate the error of the previous ones
[14</a>]</span>; AWQ rescales channels so the few weights that
matter most land on finer points [15</a>]</span>; and the same
how-many-bits question now runs through the KV cache and even optimizer
states, where 8-bit Adam keeps its moments in blocks with
per-block scales [16</a>]</span>, the MX idea arriving from the
other direction.</p>

  
    </span>
    p14_grids.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: three ways to spend sixteen points. INT4's uniform grid,</span></span>
FP4 E2M1's doubling grid, and NF4's quantile grid (the sixteen</span></span>
values from the QLoRA paper), each quantizing the same gaussian</span></span>
weights with one absmax scale.</span>"""</span></span>
import</span> torch</span></span>
torch</span>.</span>manual_seed</span>(</span>0</span>)</span></span>
w</span> =</span> torch</span>.</span>randn</span>(</span>100_000</span>)</span></span>
</span>
INT4</span> =</span> torch</span>.</span>linspace</span>(</span>-</span>7</span>,</span> 7</span>,</span> 15</span>)</span> /</span> 7.0</span>            # symmetric int4</span></span>
E2M1</span> =</span> torch</span>.</span>tensor</span>([</span>0.0</span>,</span> 0.5</span>,</span> 1.0</span>,</span> 1.5</span>,</span> 2.0</span>,</span> 3.0</span>,</span> 4.0</span>,</span> 6.0</span>])</span> /</span> 6.0</span></span>
E2M1</span> =</span> torch</span>.</span>cat</span>([</span>-</span>E2M1</span>.</span>flip</span>(</span>0</span>),</span> E2M1</span>]).</span>unique</span>()</span></span>
NF4</span> =</span> torch</span>.</span>tensor</span>([</span>-</span>1.0</span>,</span> -</span>0.6961928009986877</span>,</span> -</span>0.5250730514526367</span>,</span></span>
    -</span>0.39491748809814453</span>,</span> -</span>0.28444138169288635</span>,</span> -</span>0.18477343022823334</span>,</span></span>
    -</span>0.09105003625154495</span>,</span> 0.0</span>,</span> 0.07958029955625534</span>,</span> 0.16093020141124725</span>,</span></span>
    0.24611230194568634</span>,</span> 0.33791524171829224</span>,</span> 0.44070982933044434</span>,</span></span>
    0.5626170039176941</span>,</span> 0.7229568362236023</span>,</span> 1.0</span>])</span>  # Dettmers et al. 2023</span></span>
</span>
s</span> =</span> w</span>.</span>abs</span>().</span>max</span>()</span></span>
for</span> name</span>,</span> grid</span> in</span> ((</span>"</span>INT4 uniform</span>"</span>,</span> INT4</span>), (</span>"</span>FP4 E2M1</span>"</span>,</span> E2M1</span>),</span></span>
                   (</span>"</span>NF4 quantile</span>"</span>,</span> NF4</span>)):</span></span>
    q</span> =</span> grid</span>[( (</span>w</span> /</span> s</span>).</span>unsqueeze</span>(</span>-</span>1</span>)</span> -</span> grid</span> ).</span>abs</span>().</span>argmin</span>(</span>-</span>1</span>)]</span> *</span> s</span></span>
    err</span> = (</span>q</span> -</span> w</span>).</span>abs</span>()</span></span>
    print</span>(</span>f</span>"</span>{</span>name</span>:14s</span>}</span> mean abs err </span>{</span>err</span>.</span>mean</span>()</span>:.4f</span>}</span>   "</span></span>
          f</span>"p99 </span>{</span>err</span>.</span>quantile</span>(</span>0.99</span>)</span>:.4f</span>}</span>"</span>)</span></span>
print</span>(</span>"</span>\n</span>the bell wants points where the bell is; NF4 puts them there.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
The whole family on one sheet</a></h2>
Every format on this page has now been met. Gather them:</p>

  
    
    
  </div>
  Figure 55.</span> the whole family on one sheet, block formats included: orange buys reach, blue buys digits, and a shared scale slides a narrow row anywhere. every scalar number derived from the bit counts and checked against torch where torch carries the format (proof below).</figcaption>
</figure>

  
    </span>
    p18_family_sheet.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: the whole-family sheet, derived from each format's bit</span></span>
counts and checked against torch wherever torch has the type.</span></span>
For every format: ceiling, smallest positive value, the number of</span></span>
doublings between them (log2 of their ratio), step at 1.0, and</span></span>
decimal digits. kind says how the top exponent codes are used:</span></span>
  ieee  top exponent code reserved for inf and NaN</span></span>
  fn    no inf; only the top mantissa pattern of the top code is NaN</span></span>
  all   no inf and no NaN; every code is a number (FP4 E2M1)</span></span>
"""</span></span>
import</span> math</span></span>
import</span> torch</span></span>
</span>
def</span> family</span>(</span>E</span>,</span> M</span>,</span> bias</span>,</span> kind</span>):</span></span>
    if</span> kind</span> ==</span> "</span>ieee</span>"</span>:</span></span>
        emax</span> =</span> 2</span> **</span> E</span> -</span> 2</span> -</span> bias</span></span>
        ceiling</span> = (</span>2</span> -</span> 2.0</span> ** -</span>M</span>)</span> *</span> 2.0</span> **</span> emax</span></span>
    elif</span> kind</span> ==</span> "</span>fn</span>"</span>:</span></span>
        emax</span> =</span> 2</span> **</span> E</span> -</span> 1</span> -</span> bias</span></span>
        ceiling</span> = (</span>2</span> -</span> 2.0</span> **</span> (</span>1</span> -</span> M</span>))</span> *</span> 2.0</span> **</span> emax</span></span>
    else</span>:</span>  # all</span></span>
        emax</span> =</span> 2</span> **</span> E</span> -</span> 1</span> -</span> bias</span></span>
        ceiling</span> = (</span>2</span> -</span> 2.0</span> ** -</span>M</span>)</span> *</span> 2.0</span> **</span> emax</span></span>
    smallest</span> =</span> 2.0</span> **</span> (</span>1</span> -</span> bias</span> -</span> M</span>)</span>          # smallest subnormal</span></span>
    return</span> {</span></span>
        "</span>ceiling</span>"</span>:</span> ceiling</span>,</span></span>
        "</span>smallest</span>"</span>:</span> smallest</span>,</span></span>
        "</span>doublings</span>"</span>:</span> math</span>.</span>log2</span>(</span>ceiling</span>)</span> -</span> math</span>.</span>log2</span>(</span>smallest</span>),</span></span>
        "</span>step1</span>"</span>:</span> 2.0</span> ** -</span>M</span>,</span></span>
        "</span>digits</span>"</span>: (</span>M</span> +</span> 1</span>)</span> *</span> math</span>.</span>log10</span>(</span>2</span>),</span></span>
    }</span></span>
</span>
ROWS</span> = [</span></span>
    (</span>"</span>float64</span>"</span>,</span> 11</span>,</span> 52</span>,</span> 1023</span>,</span> "</span>ieee</span>"</span>,</span> torch</span>.</span>float64</span>),</span></span>
    (</span>"</span>float32</span>"</span>,</span> 8</span>,</span> 23</span>,</span> 127</span>,</span> "</span>ieee</span>"</span>,</span> torch</span>.</span>float32</span>),</span></span>
    (</span>"</span>TF32</span>"</span>,</span> 8</span>,</span> 10</span>,</span> 127</span>,</span> "</span>ieee</span>"</span>,</span> None</span>),</span></span>
    (</span>"</span>bfloat16</span>"</span>,</span> 8</span>,</span> 7</span>,</span> 127</span>,</span> "</span>ieee</span>"</span>,</span> torch</span>.</span>bfloat16</span>),</span></span>
    (</span>"</span>float16</span>"</span>,</span> 5</span>,</span> 10</span>,</span> 15</span>,</span> "</span>ieee</span>"</span>,</span> torch</span>.</span>float16</span>),</span></span>
    (</span>"</span>FP8 E5M2</span>"</span>,</span> 5</span>,</span> 2</span>,</span> 15</span>,</span> "</span>ieee</span>"</span>,</span> torch</span>.</span>float8_e5m2</span>),</span></span>
    (</span>"</span>FP8 E4M3</span>"</span>,</span> 4</span>,</span> 3</span>,</span> 7</span>,</span> "</span>fn</span>"</span>,</span> torch</span>.</span>float8_e4m3fn</span>),</span></span>
    (</span>"</span>FP6 E3M2</span>"</span>,</span> 3</span>,</span> 2</span>,</span> 3</span>,</span> "</span>all</span>"</span>,</span> None</span>),</span></span>
    (</span>"</span>FP6 E2M3</span>"</span>,</span> 2</span>,</span> 3</span>,</span> 1</span>,</span> "</span>all</span>"</span>,</span> None</span>),</span></span>
    (</span>"</span>FP4 E2M1</span>"</span>,</span> 2</span>,</span> 1</span>,</span> 1</span>,</span> "</span>all</span>"</span>,</span> None</span>),</span></span>
]</span></span>
</span>
print</span>(</span>f</span>"</span>{</span>'</span>format</span>'</span>:10s</span>} {</span>'</span>ceiling</span>'</span>:>10s</span>} {</span>'</span>smallest</span>'</span>:>10s</span>}</span> "</span></span>
      f</span>"</span>{</span>'</span>doublings</span>'</span>:>9s</span>} {</span>'</span>step at 1</span>'</span>:>10s</span>} {</span>'</span>digits</span>'</span>:>6s</span>}</span>"</span>)</span></span>
for</span> name</span>,</span> E</span>,</span> M</span>,</span> bias</span>,</span> kind</span>,</span> dt</span> in</span> ROWS</span>:</span></span>
    f</span> =</span> family</span>(</span>E</span>,</span> M</span>,</span> bias</span>,</span> kind</span>)</span></span>
    print</span>(</span>f</span>"</span>{</span>name</span>:10s</span>} {</span>f</span>[</span>'</span>ceiling</span>'</span>]</span>:>10.4g</span>} {</span>f</span>[</span>'</span>smallest</span>'</span>]</span>:>10.3g</span>}</span> "</span></span>
          f</span>"</span>{</span>f</span>[</span>'</span>doublings</span>'</span>]</span>:>9.1f</span>} {</span>f</span>[</span>'</span>step1</span>'</span>]</span>:>10.4g</span>}</span> "</span></span>
          f</span>"</span>{</span>f</span>[</span>'</span>digits</span>'</span>]</span>:>6.1f</span>}</span>"</span>)</span></span>
    if</span> dt</span> is not</span> None</span>:</span></span>
        fi</span> =</span> torch</span>.</span>finfo</span>(</span>dt</span>)</span></span>
        assert</span> f</span>[</span>"</span>ceiling</span>"</span>]</span> ==</span> fi</span>.</span>max</span>, (</span>name</span>,</span> f</span>[</span>"</span>ceiling</span>"</span>],</span> fi</span>.</span>max</span>)</span></span>
        assert</span> f</span>[</span>"</span>step1</span>"</span>]</span> ==</span> fi</span>.</span>eps</span>, (</span>name</span>,</span> f</span>[</span>"</span>step1</span>"</span>],</span> fi</span>.</span>eps</span>)</span></span>
print</span>(</span>"</span>\n</span>torch.finfo agrees on ceiling and step for every format </span>"</span></span>
      "</span>torch carries</span>\n</span>(float64/32/16, bfloat16, both FP8 types).</span>"</span>)</span></span>
print</span>(</span>"</span>FP6 sheets from the same formulas; Blackwell tensor cores </span>"</span></span>
      "</span>run them,</span>\n</span>torch has no storage type for them yet.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
Read the sheet by columns. The bit bars share one scale: you
can see float64 dwarf everything and the narrow rows shrink to
almost nothing. The reach bars collapse from float64's 2,098
doublings to E2M1's 3.6; every format from float16 down fits
its whole reach on the drawing. The digits column falls from 16
to 0.6. The block rows at the bottom are the repair: the same
narrow elements, plus a shared scale whose dashed arrows say
what it does, sliding each block's small reach to wherever its
values live. And the sheet is the page in one drawing:
everything before the field guide taught you what the columns
mean, and everything after it was the story of how the narrow
rows are made usable.</p>
Where the bytes live</a></h2>
All of it, on one shelf:</p>

  
    
    
  </div>
  Figure 56.</span> 7 billion weights 4 ways, the 112 GB training footprint, and the serving arithmetic: with float16 KV one conversation fills an 80 GB GPU; FP8 KV halves it. the bus carries all of it, every step and every token.</figcaption>
</figure>
These panels are the honest reason this whole family of
formats exists. A
7-billion-parameter model is 28 GB of float32 or 3.7 GB of MXFP4
with its scales; a training run holds 8x its bf16 weights; and
in serving, the KV cache decides how many conversations one GPU
carries. gpt-oss ships its 120-billion-parameter
mixture-of-experts in MXFP4 so it fits a single 80 GB GPU
[9</a>]</span>.</p>
Serving adds a loop of its own. A chat model generates 1 token
at a time, and every token's step reads all the weights and the
whole KV cache, the stored attention keys and values of every
token so far; serving engines keep that cache in fixed-size
pages so many conversations can grow side by side without
fragmenting memory [29</a>]</span>. That loop, end to end:</p>

  
    
    
  </div>
  Figure 57.</span> one generated token, end to end. every token streams the full weights and the full cache across the memory bus; halving the bytes halves the time per token [29].</figcaption>
</figure>
Hardware truth</a></h2>
Throughput follows the same halvings on hardware that speaks
the format natively, and only there: on a GPU that must
dequantize in software, a narrow format saves memory but not
time. That warning is measurable.
One published comparison ran the same image-generation model on
a GPU with no MXFP8 hardware: the MXFP8 file was smaller, and
generation was slower than bfloat16 (112.8 seconds against
96.0), because every block had to be unpacked in software [25</a>]</span>.
The same tests ranked output quality FP32, then FP16 and BF16
about equal, then MXFP8, then per-tensor-scaled FP8, then
NVFP4, then plain FP8 last [25</a>]</span>: size and quality move
together, and a scale at any granularity beats no scale. Who
speaks what natively today: NVIDIA's Blackwell chips run
MXFP8/6/4 and NVFP4 in their tensor cores [24</a>]</span>; AMD's newest
Instinct chips run the MX family [26</a>]</span>; Hopper GPUs and
Trainium2 can store MX blocks but unpack them for the math [26</a>]</span>.
The same facts, drawn so a spec sheet can be checked against
them:</p>

  
    
    
  </div>
  Figure 58.</span> who speaks what, from this page's sources [20][24][25][26]. filled: runs it in the tensor cores. half: stores it, unpacks in software. empty: these sources make no claim.</figcaption>
</figure>
And what a native format buys, next to the measured price of one
the hardware cannot speak:</p>

  
    
    
  </div>
  Figure 59.</span> each native halving doubles the matmul rate [19][24]; the bottom bar is the measured cost when the hardware cannot speak the format [25].</figcaption>
</figure>
Read a spec sheet's format row before believing its speedup
column.</p>
The contract</a></h2>
Two runs of the same training script, same seeds, same data, on
two different GPUs, will not match bit for bit, and now every
reason is on this page: sums land in different orders across
different core counts; tensor cores round differently than the
unfused loop; TF32 trims mantissas silently on one machine and
not another. And there is a quieter actor: the compiler. IEEE
754 fixes each instruction's bits, but no language promises
which instructions your line becomes. A compiler may legally
fuse a * b + c into 1 fused multiply-add (1 rounding) or keep it
as 2 instructions (2 roundings), and the choice can flip with an
optimization flag or with something as small as inlining a
function [27</a>]</span>; the rounding section's proof showed the 2 paths
disagreeing. torch.compile is a compiler too: its fused kernels
are allowed to round differently than eager mode [28</a>]</span>. And
rounding in 2 hops is not rounding once: take an exact value to
float64 first and then to float32, and you can land on a
different number than going straight to float32, because the
first rounding can create a tie the second one breaks the other
way (proof</a> builds one). This
double rounding is why the old x87 unit, which computed in
80-bit registers and rounded again on every store to memory,
made a program's bits depend on register allocation [4</a>]</span>. None
of it is a bug; all of it is the map.</p>

  
    </span>
    p23_double_rounding.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: rounding twice can differ from rounding once. Take the</span></span>
exact value 1 + 2^-24 + 2^-60. Rounded straight to float32 it goes</span></span>
UP to 1 + 2^-23, because it sits just above the halfway point.</span></span>
Rounded first to float64, the tiny 2^-60 tail is lost and the</span></span>
float64 result lands exactly ON the halfway point; rounding that</span></span>
to float32 then goes DOWN to 1.0 by ties-to-even. Two legal</span></span>
paths, 2 different answers: double rounding, the failure mode of</span></span>
every system that computes wider first and stores narrower.</span></span>
"""</span></span>
from</span> fractions</span> import</span> Fraction</span> as</span> F</span></span>
import</span> numpy</span> as</span> np</span></span>
</span>
exact</span> =</span> F</span>(</span>1</span>)</span> +</span> F</span>(</span>1</span>,</span> 2</span> **</span> 24</span>)</span> +</span> F</span>(</span>1</span>,</span> 2</span> **</span> 60</span>)</span></span>
</span>
# path 1: round once, straight to float32 (decided exactly)</span></span>
lo</span>,</span> hi</span> =</span> F</span>(</span>1</span>),</span> F</span>(</span>1</span>)</span> +</span> F</span>(</span>1</span>,</span> 2</span> **</span> 23</span>)</span>          # float32 neighbors</span></span>
assert</span> lo</span> <</span> exact</span> <</span> hi</span></span>
once</span> =</span> np</span>.</span>float32</span>(</span>1.0</span>)</span> if</span> (</span>exact</span> -</span> lo</span>)</span> <=</span> (</span>hi</span> -</span> exact</span>)</span> else</span> \</span></span>
       np</span>.</span>float32</span>(</span>1.0</span> +</span> 2.0</span> ** -</span>23</span>)</span></span>
</span>
# path 2: round to float64 first, then to float32</span></span>
via64</span> =</span> float</span>(</span>exact</span>)</span>                          # correctly rounded</span></span>
twice</span> =</span> np</span>.</span>float32</span>(</span>via64</span>)</span></span>
</span>
print</span>(</span>f</span>"exact value          : 1 + 2^-24 + 2^-60"</span>)</span></span>
print</span>(</span>f</span>"rounded once to fp32 : </span>{</span>once</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"fp64 first           : </span>{</span>via64</span>!r</span>}</span>"</span>)</span></span>
print</span>(</span>f</span>"then to fp32         : </span>{</span>twice</span>!r</span>}</span>"</span>)</span></span>
assert</span> once</span> ==</span> np</span>.</span>float32</span>(</span>1.0</span> +</span> 2.0</span> ** -</span>23</span>)</span></span>
assert</span> twice</span> ==</span> np</span>.</span>float32</span>(</span>1.0</span>)</span></span>
print</span>(</span>"</span>\n</span>the 2 paths disagree. the old x87 unit computed in 80-bit</span>"</span>)</span></span>
print</span>(</span>"</span>registers and rounded again on every spill to memory, so a</span>"</span>)</span></span>
print</span>(</span>"</span>program's bits depended on register allocation; SSE and its</span>"</span>)</span></span>
print</span>(</span>"</span>one-width successors retired the problem, and this proof is</span>"</span>)</span></span>
print</span>(</span>"</span>why narrow-then-narrower casts still deserve suspicion.</span>"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
 The
working rules: compare models with tolerances
(torch.testing.assert_close</code>, never ==</code>); flip
torch.use_deterministic_algorithms(True)</code> when you need
repeatability inside one machine and accept the speed cost; and
when a loss goes nan, hunt the overflow upstream (an unmasked
softmax, an unscaled fp16 gradient, a variance of zero) rather
than rerolling the seed. The nan at the top of this page has the same kind of cause as
the 0.1 + 0.2: some number moved past the edge of its map.</p>
Where this came from</a></h2>

  
    
    
  </div>
  Figure 60.</span> 80 years of the same idea: use fewer bits where the numbers allow it, and move the repair (bias, scale, block) somewhere cheaper.</figcaption>
</figure>
Konrad Zuse's Z3 computed in 22-bit floating point in 1941,
special values included. 44 years later IEEE 754, largely
William Kahan's design, ended an era in which every vendor's
arithmetic disagreed [3</a>]</span>. The formats since bfloat16
are the same standard's ideas at new bit counts, and the full table
of every named format that ever shipped, from the Z3 to the
G.711 telephone float of 1972 (an 8-bit float in every phone call
for 50 years) to NVFP4, is longer than this page and belongs
to a follow-up table of its own. The roads not taken, posits and
logarithmic number systems, are worth reading about
[17</a>]</span>.</p>
Build it yourself</a></h2>
You already did, in the proofs. The 15-line decoder in
p2_e4m3_map.py</a> is a complete FP8
implementation that torch agrees with on all 256 patterns; the
block quantizer in p11_mxblock.py</a> is MXFP4
minus the packing; the stochastic rounder lives in
p12_sr_drift.py</a>. Change E4M3's constants
to E5M2's and check yourself against torch again: that is the
exercise that makes every format in this family yours.</p>

  
    </span>
    p11_mxblock.py</code>
    the proof, ready to read or run</span>
  </summary>
  
"""</span>Proof: an MX-style block quantizer built from the spec: 32</span></span>
elements share one power-of-two scale (E8M0), elements are E2M1.</span></span>
The scale exponent is rounded UP, the choice that keeps the block</span></span>
maximum representable (the OCP round-down loses it). Sweep block</span></span>
sizes to see the trade: overhead per element vs error.</span>"""</span></span>
import</span> math</span>,</span> torch</span></span>
</span>
E2M1</span> =</span> torch</span>.</span>tensor</span>([</span>0.0</span>,</span> 0.5</span>,</span> 1.0</span>,</span> 1.5</span>,</span> 2.0</span>,</span> 3.0</span>,</span> 4.0</span>,</span> 6.0</span>])</span></span>
</span>
def</span> round_e2m1</span>(</span>v</span>):</span></span>
    idx</span> = (</span>v</span>.</span>abs</span>().</span>unsqueeze</span>(</span>-</span>1</span>)</span> -</span> E2M1</span>).</span>abs</span>().</span>argmin</span>(</span>-</span>1</span>)</span></span>
    return</span> E2M1</span>[</span>idx</span>]</span> *</span> v</span>.</span>sign</span>()</span></span>
</span>
def</span> mx_quant</span>(</span>x</span>,</span> block</span>=</span>32</span>):</span></span>
    q</span> =</span> torch</span>.</span>empty_like</span>(</span>x</span>)</span></span>
    for</span> i</span> in</span> range</span>(</span>0</span>,</span> len</span>(</span>x</span>),</span> block</span>):</span></span>
        b</span> =</span> x</span>[</span>i</span>:</span>i</span>+</span>block</span>]</span></span>
        amax</span> =</span> b</span>.</span>abs</span>().</span>max</span>()</span></span>
        if</span> amax</span> ==</span> 0</span>:</span></span>
            q</span>[</span>i</span>:</span>i</span>+</span>block</span>] =</span> 0</span>; </span>continue</span></span>
        e</span> =</span> math</span>.</span>ceil</span>(</span>math</span>.</span>log2</span>(</span>amax</span> /</span> 6.0</span>))</span>     # round UP: amax stays <= 6*2^e</span></span>
        e</span> =</span> max</span>(</span>-</span>127</span>,</span> min</span>(</span>127</span>,</span> e</span>))</span></span>
        s</span> =</span> 2.0</span> **</span> e</span></span>
        q</span>[</span>i</span>:</span>i</span>+</span>block</span>] =</span> round_e2m1</span>(</span>b</span> /</span> s</span>)</span> *</span> s</span></span>
    return</span> q</span></span>
</span>
torch</span>.</span>manual_seed</span>(</span>1</span>)</span></span>
x</span> =</span> torch</span>.</span>randn</span>(</span>4096</span>)</span></span>
for</span> block</span> in</span> (</span>8</span>,</span> 16</span>,</span> 32</span>,</span> 64</span>,</span> 128</span>):</span></span>
    q</span> =</span> mx_quant</span>(</span>x</span>,</span> block</span>)</span></span>
    rel</span> = ((</span>q</span> -</span> x</span>).</span>abs</span>()</span> /</span> x</span>.</span>abs</span>().</span>clamp_min</span>(</span>1e-9</span>)).</span>median</span>()</span></span>
    bits</span> =</span> 4</span> +</span> 8</span> /</span> block</span></span>
    print</span>(</span>f</span>"block </span>{</span>block</span>:>3</span>}</span>:  bits/element </span>{</span>bits</span>:.3f</span>}</span>   median rel err </span>{</span>rel</span>:.1%</span>}</span>"</span>)</span></span>
</span>
# and the rounding-direction subtlety, demonstrated</span></span>
b</span> =</span> torch</span>.</span>tensor</span>([</span>5.9</span>]</span> *</span> 32</span>)</span></span>
e_up</span> =</span> math</span>.</span>ceil</span>(</span>math</span>.</span>log2</span>(</span>5.9</span> /</span> 6.0</span>))</span>; e_dn</span> =</span> math</span>.</span>floor</span>(</span>math</span>.</span>log2</span>(</span>5.9</span> /</span> 6.0</span>))</span></span>
up</span> =</span> round_e2m1</span>(</span>b</span> /</span> 2.0</span>**</span>e_up</span>)</span> *</span> 2.0</span>**</span>e_up</span></span>
dn</span> = (</span>round_e2m1</span>((</span>b</span> /</span> 2.0</span>**</span>e_dn</span>).</span>clamp</span>(</span>-</span>6</span>,</span> 6</span>))</span> *</span> 2.0</span>**</span>e_dn</span>)</span></span>
print</span>(</span>f</span>"</span>\n</span>block of 5.9s: scale rounded up -> </span>{</span>up</span>[</span>0</span>].</span>item</span>()</span>:.2f</span>}</span>, "</span></span>
      f</span>"rounded down (clamped) -> </span>{</span>dn</span>[</span>0</span>].</span>item</span>()</span>:.2f</span>}</span>   (true 5.90)"</span>)</span></span>
</span></code></pre>
    download and run it</a>
  </div>
</details>
What you can now say</a></h2>

why 0.1 + 0.2 misses, and which 3 numbers were never stored</li>
how to convert 0.1 to binary by hand, and why its bits repeat</li>
what the exponent and mantissa each buy, what the bias is for,
and why the leading 1 is never stored</li>
how to read any format's map: points per interval, step per interval,
ulp, ceiling, floor, ramp</li>
every format's card: where it came from, where it runs, and
the number at which it fails</li>
the whole family on one sheet: how the bits split between
reach and digits, from float64's 2,098 doublings to FP4's 3.6</li>
why bfloat16 is float32's top 16 bits, at the bit level</li>
what absorption, cancellation and order-dependence do to real
code, and which repair fits which</li>
every epsilon in a transformer, and the one-line softmax slide</li>
why training keeps a float32 master copy, and what loss scaling
actually moves</li>
why FP8 is a pair, what a block scale is, why E8M0 has no
mantissa, and why FP4 cannot live without blocks</li>
what stochastic rounding and Hadamard rotations buy at 4 bits</li>
why INT4, FP4 and NF4 put their 16 points in different
places</li>
why two GPUs never agree to the last bit, why even a compiler's
fusion choice can change a program's bits, and why neither is
a bug</li>
</ul>
Try it yourself</a></h2>
Predict first, then run, then explain the difference:
p0</a> the two opening questions;
p1</a> your own histogram;
p2</a> all 256 of E4M3;
p4</a> absorption and cancellation;
p5</a> order;
p6</a> the cliff;
p7</a> the fork;
p8</a> the vanished update;
p9</a> the slide;
p10</a> the spike;
p11</a> blocks;
p12</a> stochastic rounding;
p13</a> the rotation;
p14</a> the three grids;
p16</a> the four cards from two numbers;
p17</a> all 65,536 bfloat16 patterns;
p18</a> the family sheet;
p19</a> one NVFP4 block by hand;
p20</a> all 256 E5M2 patterns;
p21</a> 1 rounding against 2;
p22</a> the error, caught exactly;
p23</a> rounding twice.
On paper: how many doublings separate E5M2's floor from its
ceiling, and would the p1 gradients fit without a scale? And
before running p16: derive float16's ceiling from E=5, M=10
yourself, the way the binary32 section taught you.</p>
References</a></h2>
[1]</span> Khalilli, proof scripts for this page, measured on an Apple
M3 Max, torch 2.11.0, CPU</em>, 2026. Linked in place above; rerun them
to check me.</p>
[2]</span> Sanglard, Floating Point Visually Explained</em>, 2017. The
interval-and-position way of seeing the fields comes from here.
https://fabiensanglard.net/floating_point_visually_explained/</a></p>
[3]</span> IEEE, 754-2019: Standard for Floating-Point Arithmetic</em>
(first edition 1985).</p>
[4]</span> Goldberg, What Every Computer Scientist Should Know About
Floating-Point Arithmetic</em>, ACM Computing Surveys, 1991; and Muller
et al., Handbook of Floating-Point Arithmetic</em>, 2nd ed., 2018.</p>
[5]</span> Micikevicius et al., Mixed Precision Training</em>, ICLR 2018.
https://arxiv.org/abs/1710.03740</a></p>
[6]</span> Micikevicius et al., FP8 Formats for Deep Learning</em>, 2022.
https://arxiv.org/abs/2209.05433</a></p>
[7]</span> DeepSeek-AI, DeepSeek-V3 Technical Report</em>, 2024.
https://arxiv.org/abs/2412.19437</a></p>
[8]</span> Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication
for Transformers at Scale</em>, 2022 (the emergent-outlier measurement).
https://arxiv.org/abs/2208.07339</a></p>
[9]</span> Open Compute Project, Microscaling Formats (MX)
Specification v1.0</em>, 2023; and Rouhani et al., Microscaling Data
Formats for Deep Learning</em>, 2023.
https://arxiv.org/abs/2310.10537</a></p>
[10]</span> NVIDIA, Pretraining Large Language Models with NVFP4</em>,
2025. https://arxiv.org/abs/2509.25149</a></p>
[11]</span> Mishra et al., Recipes for Pre-training LLMs with MXFP8</em>,
2025 (the scale-rounding result).
https://arxiv.org/abs/2506.08027</a></p>
[12]</span> Tseng et al., Training LLMs with MXFP4</em>, 2025.
https://arxiv.org/abs/2502.20586</a></p>
[13]</span> Dettmers et al., QLoRA: Efficient Finetuning of Quantized
LLMs</em>, 2023 (NF4). https://arxiv.org/abs/2305.14314</a></p>
[14]</span> Frantar et al., GPTQ</em>, 2022.
https://arxiv.org/abs/2210.17323</a></p>
[15]</span> Lin et al., AWQ: Activation-aware Weight Quantization</em>,
2023. https://arxiv.org/abs/2306.00978</a></p>
[16]</span> Dettmers et al., 8-bit Optimizers via Block-wise
Quantization</em>, 2021.
https://arxiv.org/abs/2110.02861</a></p>
[17]</span> Gustafson and Yonemoto, Beating Floating Point at its Own
Game: Posit Arithmetic</em>, 2017; logarithmic number systems survey in
Muller et al. [4].</p>
[18]</span> Severance, An Interview with the Old Man of
Floating-Point</em> (William Kahan on IEEE 754 and the Intel 8087), 1998.
https://people.eecs.berkeley.edu/~wkahan/ieee754status/754story.html</a></p>
[19]</span> NVIDIA, CUDA C++ Programming Guide</em>, the arithmetic
instructions throughput table (per-architecture float64 rates).
https://docs.nvidia.com/cuda/cuda-c-programming-guide/</a></p>
[20]</span> Kharya, TensorFloat-32 in the A100 GPU Accelerates AI
Training, HPC up to 20x</em>, NVIDIA blog, 2020.
https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/</a></p>
[21]</span> PyTorch documentation, CUDA semantics</em> (the TF32 flags and
their defaults since 1.12).
https://docs.pytorch.org/docs/stable/notes/cuda.html</a></p>
[22]</span> Industrial Light & Magic, About OpenEXR</em> (the half type,
2000-2003, and its Cg compatibility); and Bogart, Kainz, Hess, The
OpenEXR Image File Format</em>, GPU Gems, 2004.
https://openexr.com/en/latest/about.html</a></p>
[23]</span> Wang and Kanwar, BFloat16: The secret to high performance
on Cloud TPUs</em>, Google Cloud blog, 2019.
https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus</a></p>
[24]</span> Ku, Poli et al. (Radical Numerics), NVFP4 pretraining:
from theory to implementation, Part 1</em>, 2026. The recipe
walk-through this page's NVFP4 worked block follows.
https://www.radicalnumerics.ai/blog/nvfp4-part1</a></p>
[25]</span> Easygoing, Which is Better: FP8_scaled or MXFP8? A
Thorough Comparison of Image Generation AI Model Accuracy and
Speed</em>, AI Image Journey, 2026. The measured quality ranking and
the measured slowdown of MXFP8 on hardware without MX support.
https://note.com/ai_image_journey/n/n99d0ed2f1c1d</a></p>
[26]</span> ZeroEntropy, MXFP4</em> (concepts), 2026. The per-vendor
native-support summary.
https://zeroentropy.dev/concepts/mxfp4/</a></p>
[27]</span> Boehm, Can Function Inlining Affect Floating Point
Outputs? Exploring FMA and Other Consistency Issues</em>, 2023.
https://siboehm.com/articles/23/Inlining-FMA-FP-consistency</a></p>
[28]</span> PyTorch documentation, Numerical accuracy</em>.
https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html</a></p>
[29]</span> Kwon et al., Efficient Memory Management for Large Language
Model Serving with PagedAttention</em> (the vLLM paper), 2023.
https://arxiv.org/abs/2309.06180</a></p>
[30]</span> NVIDIA, H100 Tensor Core GPU</em> specifications (HBM3 bandwidth
and PCIe generation 5 rates).
https://www.nvidia.com/en-us/data-center/h100/</a></p>
Three good things to read after this page: Goldberg's paper [4]
slowly, with the map in hand; the OCP MX specification [9], which is
short and readable; and Kahan's interview on the making of IEEE 754
[18], where the standard's designer tells the story himself.</p>
tensor - numerics

Floating Point
