attaches at</th>	the pytorch side</th>	who</th>	what they keep, what they bring</th></tr></thead>
nn.Module</td>	`torch/nn/</code></td>`	transformers, diffusers, timm</td>	models are Modules; torch runs them</td></tr>
the training loop</td>	`torch/autograd/</code> torch/optim/</code></td>`	lightning, accelerate</td>	torch stays the engine; they drive it</td></tr>
the distributed engine</td>	`torch/distributed/</code></td>`	deepspeed</td>	swaps the engine, brings ZeRO</td></tr>
the eager runtime</td>	`torch/nn/</code> torch/library.py</code></td>`	vllm, sglang, TensorRT-LLM</td>	keep the weights, replace the runtime, each with a csrc/ of its own kernels</td></tr>
the operation list</td>	`aten/</code> torch/library.py</code></td>`	flash-attention, torchvision ops</td>	new names on the list</td></tr>
two floors at once</td>	`torch/autograd/</code> + transformers</td>`	unsloth</td>	trains through transformers, brings its own Triton kernels</td></tr>
only the weights</td>	none; the weights file</td>	TEI, llama.cpp, MLX</td>	left pytorch, kept the weights; llama.cpp re-encodes them to GGUF</td></tr> </tbody> </table></div> Two rows of the table deserve pictures. The first is the eager runtime, the thing the serving engines replace:</p> </div> Figure 17.</span> two ways to run one model. the engines replace the loop in the middle; the ground is shared.</figcaption> </figure> The second is Triton, the kernel language that appears through the whole table: PyTorch's compiler writes it, and libraries bring their own:</p> </div> Figure 18.</span> triton and pytorch. the compiler writes triton itself; hand-written kernels run on torch tensors and join the list.</figcaption> </figure> Read the table downward and less of PyTorch survives each row. The last row keeps nothing but the weights file. That is the quiet law of the ecosystem: the weights outlive the runtime. Part 10 walks these attachment points one by one.</p> The twelve ideas</a></h2> Most of PyTorch is not thousands of separate decisions. It is a small set of ideas, applied everywhere. These twelve make the rest of the codebase predictable before you read it. Each one returns later as a full chapter or part. Each one comes with runnable evidence now; the small proofs share one script (proof</a>).</p> 1. A tensor is a window over storage</a></h3> full story: Part 1, Tensor</span></div> </aside> A tensor does not hold numbers. It holds a description of where to look: a pointer into one flat block of memory, the sizes of each dimension, and the strides. A stride is the number of steps to move in that flat block to reach the next element of a dimension. Two tensors can look completely different and read the same bytes:</p> >>></span> x</span> =</span> torch</span>.</span>arange</span>(</span>6</span>.)</span>; v</span> =</span> x</span>.</span>view</span>(</span>2</span>,</span> 3</span>)</span></span> >>></span> x</span>.</span>data_ptr</span>()</span> ==</span> v</span>.</span>data_ptr</span>()</span> # same address in memory</span></span> True</span></span> >>></span> v</span>.</span>stride</span>(),</span> v</span>.</span>t</span>().</span>stride</span>()</span> # transpose swapped the strides</span></span> ((</span>3</span>,</span> 1</span>), (</span>1</span>,</span> 3</span>))</span></span></code></pre> The transpose moved no data. It swapped two numbers in the description. Some descriptions are impossible to write down, and that is exactly why v.t().view(-1)</code> raises an error while reshape</code> silently copies the data instead. Part 1 opens with this puzzle and solves it completely.</p> Interactive</span> this instrument needs javascript; the still drawing stands in.</p> </div> </div> Interactive 2.</span> six numbers, one storage. shape and strides decide which slot every cell reads; view(-1) exists only when the walk matches storage order.</p> </div> 2. Autograd records a program you never wrote</a></h3> full story: Part 2, Autograd</span></div> </aside> In your code, y</code> is one name, and you overwrite it freely. Line two destroys the value line one made. The graph cannot afford that: backward will need every step. So autograd writes one record per change, and no record is ever overwritten. Watch the record change as y</code> does:</p> >>></span> a</span> =</span> torch</span>.</span>ones</span>(</span>3</span>,</span> requires_grad</span>=</span>True</span>)</span></span> >>></span> y</span> =</span> a</span> </span> 2</span></span> >>></span> type</span>(</span>y</span>.</span>grad_fn</span>).</span>__name__</span></span> '</span>MulBackward0</span>'</span></span> >>></span> y</span>.</span>add_</span>(</span>1</span>)</span> # change y in place</span></span> >>></span> type</span>(</span>y</span>.</span>grad_fn</span>).</span>__name__</span></span> '</span>AddBackward0</span>'</span></span> >>></span> y</span>[</span>0</span>] =</span> 9</span> # overwrite one slot</span></span> >>></span> type</span>(</span>y</span>.</span>grad_fn</span>).</span>__name__</span></span> '</span>CopySlices</span>'</span></span></code></pre> </div> Figure 19.</span> your code keeps one value and destroys the past. the graph keeps every step: one record per change, nothing overwritten.</figcaption> </figure> Three statements, three records, one chain. y.grad_fn</code> always holds the newest record, and each record points at the one before it, so the whole history stays reachable. That history is the program you never wrote. The machinery that keeps it correct under every kind of in-place change has real depth, and it is one of the best chapters of Part 2.</p> 3. One list of operations is the whole interface</a></h3> full story: Part 5, The Machinery</span></div> </aside> </div> Figure 20.</span> the code on the left never reaches a device. only the list does.</figcaption> </figure> The 3,677 registered names are PyTorch's real interface. Each backend implements its share of them. The compiler rewrites programs made of them: torch.compile</code> reads the list your program became and returns a shorter one, where a matrix multiplication and the add after it can fuse into one addmm</code>, and a chain of small elementwise operations becomes one generated kernel. Export formats store them: torch.export</code> writes the list to disk as a graph of exactly these names, and an ONNX file is the same idea with each name translated into ONNX's vocabulary. Quantization replaces them: the float32 matmul is swapped for an int8 body, the same place on the list, different arithmetic. When you meet a new PyTorch technology, ask one question first: what does it do to the operations? The answer usually explains the whole design.</p> 4. PyTorch writes most of its own code</a></h3> full story: Part 5, The Machinery</span></div> </aside> </div> Figure 21.</span> one yaml file in, thousands of functions out, at every build.</figcaption> </figure> native_functions.yaml</code> declares every operation. derivatives.yaml</code> declares every derivative. At build time, torchgen/</code> reads both and writes the Python bindings, the autograd record classes and the dispatcher tables. This is why searching the repository for a function you just called can find nothing: you searched the input of the build, and the function is in the output. People who work on PyTorch read the yaml first. After Part 5, so will you.</p> 5. Features are layers with a switch</a></h3> full story: Part 5, The Machinery</span></div> </aside> </div> Figure 22.</span> every feature watches the same stream of operations. a context manager puts one layer to sleep.</figcaption> </figure> PyTorch's features combine cleanly because each one is a layer in the dispatcher, watching the same stream of operations:</p> >>></span> with</span> torch</span>.</span>no_grad</span>():</span></span> ...</span> z</span> =</span> a</span> </span> 2</span></span> >>></span> z</span>.</span>grad_fn</span> is</span> None</span> # nothing was recorded</span></span> True</span></span></code></pre> no_grad</code> edited no function. It set a flag that sends operations past the autograd layer, so nothing gets recorded. Mixed precision, tracing and vmap work the same way, and that is why they can be combined without knowing about each other. Part 5 opens the machinery under the flag.</p> 6. Every operation pays a fixed cost first</a></h3> full story: Part 7, The Compiler</span></div> </aside> Half a microsecond of crossing and routing before any math, on every single operation. That was the measurement on the boundary floor. Applied honestly, this one number explains why torch.compile</code> exists, why fused optimizers exist, and why the first question about any slow model is: is it limited by compute, or by the cost of issuing many small operations?</p> Interactive</span> this instrument needs javascript; the still drawing stands in.</p> </div> </div> Interactive 3.</span> the same add at twelve measured sizes. the dots are measured on the author's machine; plant your flag before the curve appears.</p> </div> 7. Memory, not speed, is what kills training runs</a></h3> full story: Part 4, Seeing PyTorch</span></div> </aside> </div> Figure 23.</span> the forward pass borrows memory. backward repays it. running out is the most common way a training run dies.</figcaption> </figure> A slow program still finishes. A program that runs out of GPU memory dies with CUDA out of memory</code>, and that is the most common death in all of PyTorch. The forward pass saves values for backward (the turn, above). The allocator keeps and reuses blocks. Between them they decide how large a model you can train. This series treats memory as a first-class subject in Part 4.</p> 8. Python is why it won, and what it costs</a></h3> full story: Part 7, The Compiler</span></div> </aside> </div> Figure 24.</span> the protection and the price are the same picture: everything must cross one bridge.</figcaption> </figure> PyTorch won because you write it in ordinary Python, with ordinary debuggers and print statements. The price is the border cost from Idea 6, paid on every operation. The history of the framework is a sequence of attempts to keep the first while reducing the second. TorchScript tried to replace Python with its own language; it is now in maintenance mode [4</a>]</span>. The current compiler watches your Python run and translates what it can, and it is winning. The pattern to remember: inside PyTorch, betting against Python has always lost.</p> 9. Forward decides what backward must do</a></h3> full story: Part 9, Distributed</span></div> </aside> </div> Figure 25.</span> backward is the reflection of the graph forward wrote.</figcaption> </figure> Backward can only walk what forward wrote. On one machine this sounds like a detail. At scale it becomes the law of the land: in distributed training, the way a tensor is split across GPUs in the forward pass decides which GPUs must exchange data in the backward pass. One idea, from a laptop to a cluster. It is the spine of Part 9.</p> 10. Shared bytes plus in-place writes cause the hardest problems</a></h3> full story: Part 2, Autograd</span></div> </aside> </div> Figure 26.</span> three windows, one storage, one write. every system that records or rewrites programs must handle this.</figcaption> </figure> Idea 1 lets many tensors read the same bytes. Idea 2 lets you change those bytes in place. Combine them: one write can change the meaning of several tensors at once, and any system that records programs (autograd, the compiler, export) must notice and stay correct. When a corner of PyTorch looks strangely complicated, ask what shared bytes plus an in-place write would do to it. That is usually the answer.</p> 11. The code keeps its history</a></h3> full story: Part 5, The Machinery</span></div> </aside> </div> Figure 27.</span> four systems, four eras, one repository. older layers still show through.</figcaption> </figure> The repository holds the remains of every era: the original C code from 2016, the Caffe2 merge of 2018, TorchScript from 2019, the compiler district growing since 2023. When a file looks strange, the explanation is usually historical: something older lived there first. Part 5 tells this history where it explains the present.</p> 12. Floating point is a contract; read it</a></h3> full story: Part 4, Seeing PyTorch</span></div> </aside> </div> Figure 28.</span> the terms are public. every training run signs them.</figcaption> </figure> >>></span> t</span> =</span> torch</span>.</span>tensor</span>(</span>1e8</span>)</span></span> >>></span> ((</span>t</span> +</span> 1</span>)</span> -</span> t</span>).</span>item</span>()</span></span> 0.0</span></span></code></pre> A float32 number has about 7 decimal digits of precision, so adding 1 to one hundred million changes nothing [5</a>]</span>. This is not a bug; it is the number format doing what it promises. Add the faster, less precise formats used in training, plus the fact that some GPU kernels sum in different orders on different runs, and "why did my loss change between runs" becomes a question with exact answers. Part 4 reads this contract clause by clause.</p> How this series draws</a></h2> Every still figure you just saw is a real Excalidraw scene, and the scene files ship with the series; you can open any drawing and edit it. The four instruments you can operate follow the same language, and every number inside them comes from the proof scripts. All of them speak one visual language, so that by Part 2 you read them without thinking:</p> </div> Figure 29.</span> the whole notation on one sheet. learn it once; it holds for the entire series.</figcaption> </figure> Orange always marks the subject: the one thing moving. Ink is structure. Grey is context. Dashed means recorded, implied, or asleep. The depth meter marks the floor. And the robot appears at most once per part, because a mascot that is everywhere stops being funny.</p> How to read this</a></h2> Twelve parts. Each is one long page like this one. And the series has one quiet goal behind every part: by the end, you should know the machine well enough to build a small PyTorch yourself. Every drawing that shows a mechanism, every formula next to a figure, and every proof script is a piece of that.</p> Part</th> What is behind the door</th></tr></thead> 0. The Map</td> you are here</td></tr> 1. Tensor</td> storage, strides, views, data types, broadcasting</td></tr> 2. Autograd</td> the graph, in-place writes, checkpointing, double backward</td></tr> 3. Daily PyTorch</td> nn, optim, data loading, mixed precision, seen from inside</td></tr> 4. Seeing PyTorch</td> the profiler, memory, floating point, honest measurement</td></tr> 5. The Machinery</td> the dispatcher, aten, torchgen, the history</td></tr> 6. Extending PyTorch</td> subclasses, custom operations, new backends</td></tr> 7. The Compiler</td> dynamo, aot autograd, inductor, dynamic shapes</td></tr> 8. Kernels & Hardware</td> the gpu model, triton, cutlass, what fast means</td></tr> 9. Distributed</td> collectives, ddp, fsdp, dtensor, parallel training</td></tr> 10. Ship It</td> export, quantization, executorch, the ecosystem</td></tr> 11. Working on PyTorch</td> the contributor's field guide</td></tr> </tbody></table> You do not have to read front to back. Three reading lines run through the parts, like lines through stations:</p> Interactive</span> this instrument needs javascript; the still drawing stands in.</p> </div> </div> Interactive 4.</span> pick a line. grey is front to back and stops everywhere; orange fits most readers; green chases speed; blue is the one for a new contributor. a dot means the line stops there.</p> </div> Every chapter inside every part follows the same seven steps, so the rhythm becomes familiar fast:</p> </div> Figure 30.</span> the seven steps of every chapter. the proof step is the spine: no claim without a script.</figcaption> </figure> And the method, stated plainly, because you should know what you are trusting. Every mechanism claim is checked against the source code or shown by a script before it is published. The scripts are linked in place and pinned to one torch version. When PyTorch moves and a claim goes stale, the chapter is corrected and the correction is noted on the page. A series about internals that cannot admit drift would be wrong within a year.</p> What you can now say</a></h2> Test yourself against this list. After one reading you should be able to say, in your own words:</p> what type(torch.randn)</code> returns, and where the compiled body actually lives on your disk</li> what the dispatcher is, and how no_grad</code> stops autograd without editing any function</li> what a kernel is, and what decides which one runs</li> why the CPU and the GPU run on two clocks, and why that makes simple timing code lie</li> what the graph is, who writes it, and why backward can never do anything forward did not write down</li> and the twelve ideas, each in one sentence</li> </ul> If one of these is fuzzy, return to its floor; each one is only a minute long. That is what this page is for.</p> Try it yourself</a></h2> The five proof scripts are the exercises. For each one: predict the output first, then run it, then explain the difference.</p> p0_the_library.py</a>: how big are the compiled libraries in your own torch install?</li> p1_graph_chain.py</a>: what graph remains after a two-layer model runs?</li> p2_dispatch_cost.py</a>: what is the fixed cost per operation on your machine?</li> p3_two_timelines.py</a>: how long is your GPU still working after Python is done asking?</li> p4_micro_proofs.py</a>: the twelve ideas, compressed into five small experiments.</li> </ol> Pick a door</a></h2> Part 1 is the tensor. It opens with the puzzle from Idea 1, and now you have seen the error with your own eyes:</p> >>></span> v</span>.</span>t</span>().</span>view</span>(</span>-</span>1</span>)</span></span> RuntimeError</span>:</span> view size</span> is not</span> compatible</span> with</span> input</span> tensor</span>'</span>s</span></span> size</span> and</span> stride</span> ...</span></span> >>></span> v</span>.</span>t</span>().</span>reshape</span>(</span>-</span>1</span>)</span> # this one works. why?</span></span> tensor</span>([</span>0</span>.,</span> 3</span>.,</span> 1</span>.,</span> 4</span>.,</span> 2</span>.,</span> 5</span>.])</span></span></code></pre> Same tensor. Same request. One line refuses, the other quietly copies the data. The difference between those two lines is the whole first part of this series.</p> See you on the next floor down.</p> References</a></h2> [1]</span> Khalilli, five proof scripts, measured on an Apple M3 Max, torch 2.11.0, CPU and Apple GPU</em>, 2026. Linked in place above; rerun them to check me.</p> [2]</span> PyTorch source, native_functions.yaml</em>, pinned to the v2.11.0 tag. https://github.com/pytorch/pytorch/blob/v2.11.0/aten/src/ATen/native/native_functions.yaml</a></p> [3]</span> PyTorch source, derivatives.yaml</em>, pinned to the v2.11.0 tag. https://github.com/pytorch/pytorch/blob/v2.11.0/tools/autograd/derivatives.yaml</a></p> [4]</span> PyTorch documentation, TorchScript</em>, which states it is in maintenance mode. https://docs.pytorch.org/docs/stable/jit.html</a></p> [5]</span> IEEE, 754 single precision</em>: 24 binary digits of precision, about 7 decimal digits.</p> Three good things to read after this page: Edward Yang's PyTorch internals talk</a>, which maps the C++ side in depth; the PyTorch Developer Podcast</a>, short episodes by the same author; and the repository's own CONTRIBUTING.md</a>, which describes the folder layout in the maintainers' words.</p>

The Map

2026-08-07T00:00:00+00:00

</div>

Figure 1. the program this whole series is about.</figcaption> </figure>

You have typed something like this a thousand times. This series exists so that, by its end, you know everything these lines do. All of it: the Python they touch, the C++ they land in, the graph they record, the kernels they choose, the memory they use, and the two clocks they run on. Each of those words gets a plain meaning on its floor below.

This is Part 0, the map. First we go down through all the layers once, fast. Then we draw the territory. Then twelve ideas that make the rest of the codebase predictable. Then how this series works, and how to read it. Nothing here gets its full story. Everything here gets a place, and every full story has a numbered part waiting for it.

One promise before we start. Every measured number in this series comes from a small script you can run yourself, linked right where the number appears. I measured these on an Apple M3 Max laptop with torch 2.11.0 [1</a>]. Your numbers will differ. The pattern they make will not.

The fall</a></h2>
PyTorch is deep. Between your keyboard and the chip there are eight levels. I will call them floors, and this meter shows all of them. It returns through the whole series, so you always know how deep you are.

</div>
Figure 2. the depth meter. the orange dot marks where you are.</figcaption> </figure>
The fastest way to learn a building is to go down through it once without stopping. That is this section.
Floor one: python</a></h3>

floor 2 of 8 pythonfull story: Part 5, The Machinery </div> </aside>
`torch.randn</code> looks like a Python function. Ask Python what it actually is:`
`>>> type(torch.randn) <class 'builtin_function_or_method'></code></pre> Python gives that type only to functions written in compiled code. Compiled code means: code that was translated to machine instructions before you ever installed it, so there is no Python body inside it to read, and no line for your debugger to stop on.`
So where do those machine instructions live? In shared libraries. A shared library is a file of compiled code that a program loads while it runs. They sit inside the torch package on your disk, and you can look at them (proof</a>):
`torch._C -> _C.cpython-312-darwin.so (49 KB, the loader) libtorch_cpu.dylib 206.5 MB (tensors and kernels) libtorch_python.dylib 28.5 MB (the python side of the border)</code></pre>`

"""Proof: where the compiled part of pytorch actually lives. torch._C is a thin compiled stub; the weight of the framework is in the shared libraries next to it. Prints the files and their sizes. """ import glob import os import torch stub = torch._C.file print(f"torch {torch.version}") print(f"torch._C -> {os.path.basename(stub)} " f"({os.path.getsize(stub)/1024:.0f} KB stub)") libdir = os.path.join(os.path.dirname(stub), "lib") for lib in ["libtorch_cpu.dylib", "libtorch_python.dylib"]: p = os.path.join(libdir, lib) if os.path.exists(p): print(f"{lib:24s} {os.path.getsize(p)/1024/1024:6.1f} MB") </code></pre> download and run it</a> </div> </details> Read the sizes, and then look at them: </div> Figure 3. drawn to scale by file size. the part of pytorch that python can see is the orange dot.</figcaption> </figure> The part of PyTorch you can see from Python is a 49 KB file whose only job is to load the other two. The real body is 235 MB of compiled code. import torch</code> brings it into your process, and after that, calling torch.randn</code> means jumping into that body. Today we only need to know these files exist. This is the first honest surprise of the codebase: the Python you write all day is the smallest layer of it. The boundary</a></h3> floor 3 of 8 the boundaryfull story: Part 5, The Machinery </div> </aside> The call leaves Python at once. Where does it land? In a C++ function named THPVariable_randn</code>, inside that 28.5 MB library from the last floor. And here is a strange fact you can keep: this function does not exist in the PyTorch repository. Clone the repo, search for the name, and you find nothing. A program writes this function during the build, together with thousands of its siblings. Idea 4 below explains why, and Part 5 shows the program that does the writing. </div> Figure 4. the border between the two languages. every tensor operation crosses it.</figcaption> </figure> Crossing this border costs time. To see the cost alone, time the smallest possible operation, where almost no arithmetic hides it (proof</a>): add, 1 element : 0.538 microseconds per call add, 4M elements : 337.264 microseconds per call</code></pre> p2_dispatch_cost.py</code> the proof, ready to read or run </summary> """Proof: the fixed cost of one eager op, and why size hides it. Times the same `a + b` at two sizes. The one-element add is nearly pure machinery (dispatch, wrapping, allocation); the 4M-element add is nearly pure arithmetic. CPU, single process. """ import time import torch def per_op_us(a, b, iters): # warmup for _ in range(2000): a + b t0 = time.perf_counter() for _ in range(iters): a + b return (time.perf_counter() - t0) / iters * 1e6 tiny = per_op_us(torch.ones(1), torch.ones(1), 200_000) big_n = 4_000_000 big = per_op_us(torch.ones(big_n), torch.ones(big_n), 2_000) print(f"torch {torch.version}, cpu") print(f"add, 1 element : {tiny:8.3f} us/op") print(f"add, 4M elements : {big:8.3f} us/op") print(f"machinery share of the tiny op: ~all of it") print(f"ops/sec you can issue from python: {1e6/tiny:,.0f}") # The sweep behind the toll meter: the same add at twelve sizes, # 1 to 4M elements in powers of four. Every dot on the widget's # axis is one line of this output. import json sweep = [] for k in range(12): n = 4 ** k iters = max(1_000, min(200_000, 40_000_000 // max(n, 1))) us = per_op_us(torch.ones(n), torch.ones(n), iters) sweep.append({"n": n, "us": round(us, 3)}) print(f"add, {n:>9,} elements : {us:9.3f} us/op") print("JSON_SWEEP=" + json.dumps(sweep)) </code></pre> download and run it</a> </div> </details> The one-element add does almost no math. So its 0.54 microseconds is almost pure crossing cost: leave Python, check the arguments, build the result object, return. Half a microsecond sounds like nothing. It means Python can issue at most about 1.9 million operations per second, and a single training step contains thousands of operations. Keep this number. It returns in Idea 6. The dispatcher</a></h3> floor 4 of 8 the dispatcherfull story: Part 5, The Machinery </div> </aside> Under the border, the call reaches the strangest machine in PyTorch: the dispatcher. The dispatcher is the router that decides, for every operation, which pieces of code run and in what order. Look at what it must decide. Your three lines never said "record gradients". No if</code> statement in your code turns that on. Yet somewhere, something decided that this matrix multiplication should be remembered for backward()</code>. That something is the dispatcher. Every operation passes down through a fixed stack of layers. Each layer can act on the call, change it, or let it pass unchanged. Autograd, the part of PyTorch that computes gradients, is one such layer. Mixed precision is another. On this run, only autograd is awake. </div> Figure 5. four layers touch your call before any arithmetic starts. only the highlighted one is awake today.</figcaption> </figure> The kernel</a></h3> floor 5 of 8 the kernelfull story: Part 8, Kernels & Hardware </div> </aside> At the bottom of the stack, one concrete function is chosen. Chosen is the right word. This torch build has 3,677 registered operation names (proof</a> prints the count), and a name is not a function body. The operation addmm</code>, the matrix multiplication behind model(x)</code>, has separate bodies for CPU and for each kind of GPU, for each data type, for dense and for sparse tensors. A body like this, written for one device and one data type, is called a kernel. The dispatcher's last job is to pick one: </div> Figure 6. one name, a grid of bodies. the dispatcher picks exactly one cell per call.</figcaption> </figure> p4_micro_proofs.py</code> the proof, ready to read or run </summary> """Micro-proofs quoted in Part 0: storage sharing, view errors, mutation rewriting history, the no_grad layer, float32 absorption.""" import torch print(f"torch {torch.version}\n") # 1. a tensor is a window over storage x = torch.arange(6, dtype=torch.float32) v = x.view(2, 3) print("same bytes under both:", x.data_ptr() == v.data_ptr()) print("v.stride():", v.stride(), " v.t().stride():", v.t().stride()) try: v.t().view(-1) except RuntimeError as e: print("v.t().view(-1) ->", str(e).split(".")[0]) # 2. mutation rewrites the recorded program a = torch.ones(3, requires_grad=True) y = a 2 print("\nbefore add_:", type(y.grad_fn).name) y.add_(1) print("after add_:", type(y.grad_fn).name) # 3. no_grad removes one dispatcher layer with torch.no_grad(): z = a 2 print("\ninside no_grad, grad_fn:", z.grad_fn) # 4. float32 absorbs small numbers t = torch.tensor(1e8) print("\n(1e8 + 1) - 1e8 in float32 =", ((t + 1) - t).item()) # 5. the size of the operation list (idea 3) print("\nregistered operation names:", len(torch._C._dispatch_get_all_op_names())) </code></pre> download and run it</a> </div> </details> The full list of operations lives in one file in the repository: native_functions.yaml</code></a> [2</a>]. Its sibling derivatives.yaml</code></a> [3</a>] lists the derivative of each operation. Everything else grows from these two files. No other file in the repository tells you as much per line. </div> Figure 7. 3,677 names on the left. one function body on the right. the funnel is the dispatcher's last job.</figcaption> </figure> One floor down sits memory. torch.randn(64, 128)</code> needs 32,768 bytes: 64 rows times 128 numbers times 4 bytes per number. On the CPU this is an ordinary allocation. On a GPU it is not. There, PyTorch runs its own allocator, a keeper of memory that asks the GPU driver for large blocks once and then reuses them, because asking the driver every time is slow. This allocator decides when you run out of memory and what the error means. Part 4 examines it. The two clocks</a></h3> floor 7 of 8 the queuefull story: Part 4, Seeing PyTorch </div> </aside> Here the story splits in two. What follows is the single most useful performance fact in PyTorch. On a GPU, your Python line does not do the work. It requests the work, and the request returns at once. The GPU does the work on its own clock, while Python continues. I measured it on this machine's GPU (proof</a>): time to request 50 matrix multiplications : 1.58 ms time until the work was actually done : 73.83 ms python was free during : 72.25 ms (98%)</code></pre> p3_two_timelines.py</code> the proof, ready to read or run </summary> """Proof: the CPU runs ahead of the GPU. Queues 50 large matmuls on the MPS device and measures two times: how long Python took to ask for the work, and how long the work actually took. The difference is the gap the chapter draws. """ import time import torch assert torch.backends.mps.is_available(), "needs an Apple-silicon GPU" dev = torch.device("mps") a = torch.randn(2048, 2048, device=dev) b = torch.randn(2048, 2048, device=dev) for _ in range(5): # warmup (a @ b) torch.mps.synchronize() t0 = time.perf_counter() for _ in range(50): c = a @ b t_queue = time.perf_counter() - t0 torch.mps.synchronize() t_done = time.perf_counter() - t0 print(f"torch {torch.version}, mps") print(f"time to queue 50 matmuls : {t_queue1e3:8.2f} ms") print(f"time until work finished : {t_done1e3:8.2f} ms") print(f"python was free for : {(t_done-t_queue)1e3:8.2f} ms ({(t_done-t_queue)/t_done:.0%} of the wall time)") # The three ways to read the loss, measured, for the two-clocks # widget: never, once at the end, after every step. import json def run_mode(mode, iters=50): for _ in range(5): (a @ b) torch.mps.synchronize() t0 = time.perf_counter() t_free = 0.0 for i in range(iters): c = a @ b if mode == "every": c[0, 0].item() t_q = time.perf_counter() - t0 if mode == "once": c[0, 0].item() torch.mps.synchronize() total = time.perf_counter() - t0 return {"mode": mode, "queue_ms": round(t_q 1e3, 2), "total_ms": round(total 1e3, 2), "free_ms": round((total - t_q) 1e3, 2)} modes = [run_mode(m) for m in ("never", "once", "every")] for m in modes: print(f"read {m['mode']:>5}: total {m['total_ms']:8.2f} ms, " f"python busy {m['queue_ms']:8.2f} ms") print("JSON_MODES=" + json.dumps(modes)) </code></pre> download and run it</a> </div> </details> </div> Figure 8. two clocks, one program. the cpu requested everything in the first two milliseconds; the gpu needed seventy-two more to finish.</figcaption> </figure> Python asked for all fifty multiplications in under two milliseconds, then waited, free, while the GPU computed for another seventy-two. On the CPU there is no such split; the math happens before your line returns. On any accelerator, the split is the normal state of the program. This is why eager PyTorch is fast enough to use: Python runs ahead and the GPU never waits for it. It is also why simple timing code gives wrong answers, and why one loss.item()</code> inside a training loop can slow the whole step. .item()</code> needs the actual number. The number sits at the end of a queue of work the GPU has not finished yet, so Python must stop and wait for the whole queue: </div> Figure 9. the queue between the two clocks. the number python asked for is the last ticket, so every ticket ahead of it must finish first.</figcaption> </figure> Interactive this instrument needs javascript; the still drawing stands in. </div> </div> Interactive 1. the two clocks, measured. choose how often the loop reads the loss; the lanes show who waits, and for how long. </div> Part 4 teaches honest measurement on top of exactly this picture. The turn</a></h3> floor 4 of 8 the autograd layerfull story: Part 2, Autograd </div> </aside> Line three: loss.backward()</code>. Nothing so far explains how this line can work. The forward computation is over. How does PyTorch know what to differentiate? It knows because the forward pass had a second job. Every time an operation passed the autograd layer of the dispatcher, a small record was written: which operation ran, and which recorded steps produced its inputs. Records that point at records form a graph, and that word here always means exactly this recorded structure. By the time loss</code> exists, its graph exists too (proof</a>): loss.grad_fn = SumBackward0 SumBackward0 AddmmBackward0 AccumulateGrad</code></pre> p1_graph_chain.py</code> the proof, ready to read or run </summary> """Proof: loss.backward() walks a graph that forward quietly recorded. Builds the chapter's three-line program and prints the autograd graph that exists before backward is ever called. """ import torch import torch.nn as nn torch.manual_seed(0) model = nn.Sequential(nn.Linear(128, 256), nn.ReLU(), nn.Linear(256, 10)) x = torch.randn(64, 128) loss = model(x).sum() print(f"torch {torch.version}") print(f"x.grad_fn = {x.grad_fn}") print(f"loss.grad_fn = {type(loss.grad_fn).name}") node, depth = loss.grad_fn, 0 while node is not None and depth < 10: print(" " depth + type(node).name) nexts = [n for n, _ in node.next_functions if n is not None] node = nexts[0] if nexts else None depth += 1 </code></pre> download and run it</a> </div> </details> </div> Figure 10. the forward pass goes down and writes the graph. backward climbs exactly what was written, and nothing else.</figcaption> </figure> backward()</code> invents nothing. It walks the graph from the loss back to your inputs, runs each recorded derivative, and stores the results in .grad</code>. The walk ends at AccumulateGrad</code>, the record that does the storing. And it starts nowhere else: x.grad_fn</code> is None</code>, because x</code> was created directly, not computed. One question should bother you here. A derivative needs values. The derivative of a matrix multiplication needs the matrices that were multiplied, and the forward pass is long over. Write the derivative out and the need is visible: </div> Figure 11. the derivative of x @ w, written out. the formulas contain x and w themselves; whatever forward used, backward needs again.</figcaption> </figure> So where are they? They were saved, next to the records, during the forward pass: </div> Figure 12. what each record kept. this is where the memory of a training run actually goes.</figcaption> </figure> So the forward pass silently decides how much memory training costs. Part 2 shows the exact saving rules. Part 4 shows how to watch it happen. And a method called activation checkpointing trades that memory for extra compute; it has its own chapter in Part 2. Carry one sentence out of this section: backward can only walk what forward wrote. It sounds small. In Part 9 it becomes the rule that decides which GPUs in a cluster must talk to each other. That was the whole fall: a name, a border, a stack of layers, a chosen kernel, a keeper of memory, two clocks, and a graph that is walked backward. Now the territory, properly. The territory</a></h2> PyTorch is built in layers, and each layer speaks only to its neighbors. Every box below is at least one part of this series. </div> Figure 13. the whole system on one sheet. the orange line on the left is the path we just took.</figcaption> </figure> The same territory, seen as folders in the repository. If you ever open the codebase, this is the map that stops you from being lost: </div> Figure 14. the repository as two river banks. the river is the boundary from figure 4; torch/csrc/ is its one bridge; and both banks stand on the same ground, c10/, where Tensor and Storage themselves live.</figcaption> </figure> Three facts about this map save you weeks. First: torch/</code> is plain Python, and you can read every file in it today. Second: aten/</code> and c10/</code> are C++; the tensors, the kernels and the dispatcher live there, and torch/csrc/</code> is the single bridge that connects the two languages. Third: torchgen/</code> is the program from the boundary floor, the one that writes code during the build. The repository you read is the input. The library you run is the output. That is why searching the repository for THPVariable_randn</code> finds nothing: </div> Figure 15. the repository is the part above the waterline. the code your process runs is the larger part below it.</figcaption> </figure> Above the core sits the ecosystem. It looks endless, but it has a simple shape: every library attaches to PyTorch at a specific, nameable place. Know the attachment places and you know the ecosystem. </div> Figure 16. every line points at the exact place a library attaches. trl and peft sit on the outer ring: they build on transformers, not on pytorch.</figcaption> </figure> Read the picture from the center out. transformers</code> builds its models as nn.Module</code> classes, so if you understand Part 3, you can read its source. deepspeed</code> replaces the distributed engine, so its home is Part 9. vllm</code> and sglang</code> keep the model weights and replace the runtime around them. And trl</code> and peft</code> do not touch PyTorch directly at all; they build on transformers</code>. The whole ecosystem fits in one table. The second column names the place in the PyTorch repository where each family attaches: attaches at</th> the pytorch side</th> who</th> what they keep, what they bring</th></tr></thead> nn.Module</td> torch/nn/</code></td> transformers, diffusers, timm</td> models are Modules; torch runs them</td></tr> the training loop</td> torch/autograd/</code> torch/optim/</code></td> lightning, accelerate</td> torch stays the engine; they drive it</td></tr> the distributed engine</td> torch/distributed/</code></td> deepspeed</td> swaps the engine, brings ZeRO</td></tr> the eager runtime</td> torch/nn/</code> torch/library.py</code></td> vllm, sglang, TensorRT-LLM</td> keep the weights, replace the runtime, each with a csrc/ of its own kernels</td></tr> the operation list</td> aten/</code> torch/library.py</code></td> flash-attention, torchvision ops</td> new names on the list</td></tr> two floors at once</td> torch/autograd/</code> + transformers</td> unsloth</td> trains through transformers, brings its own Triton kernels</td></tr> only the weights</td> none; the weights file</td> TEI, llama.cpp, MLX</td> left pytorch, kept the weights; llama.cpp re-encodes them to GGUF</td></tr> </tbody> </table></div> Two rows of the table deserve pictures. The first is the eager runtime, the thing the serving engines replace: </div> Figure 17. two ways to run one model. the engines replace the loop in the middle; the ground is shared.</figcaption> </figure> The second is Triton, the kernel language that appears through the whole table: PyTorch's compiler writes it, and libraries bring their own: </div> Figure 18. triton and pytorch. the compiler writes triton itself; hand-written kernels run on torch tensors and join the list.</figcaption> </figure> Read the table downward and less of PyTorch survives each row. The last row keeps nothing but the weights file. That is the quiet law of the ecosystem: the weights outlive the runtime. Part 10 walks these attachment points one by one. The twelve ideas</a></h2> Most of PyTorch is not thousands of separate decisions. It is a small set of ideas, applied everywhere. These twelve make the rest of the codebase predictable before you read it. Each one returns later as a full chapter or part. Each one comes with runnable evidence now; the small proofs share one script (proof</a>). 1. A tensor is a window over storage</a></h3> full story: Part 1, Tensor</div> </aside> A tensor does not hold numbers. It holds a description of where to look: a pointer into one flat block of memory, the sizes of each dimension, and the strides. A stride is the number of steps to move in that flat block to reach the next element of a dimension. Two tensors can look completely different and read the same bytes: >>> x = torch.arange(6.); v = x.view(2, 3) >>> x.data_ptr() == v.data_ptr() # same address in memory True >>> v.stride(), v.t().stride() # transpose swapped the strides ((3, 1), (1, 3))</code></pre> The transpose moved no data. It swapped two numbers in the description. Some descriptions are impossible to write down, and that is exactly why v.t().view(-1)</code> raises an error while reshape</code> silently copies the data instead. Part 1 opens with this puzzle and solves it completely. Interactive this instrument needs javascript; the still drawing stands in. </div> </div> Interactive 2. six numbers, one storage. shape and strides decide which slot every cell reads; view(-1) exists only when the walk matches storage order. </div> 2. Autograd records a program you never wrote</a></h3> full story: Part 2, Autograd</div> </aside> In your code, y</code> is one name, and you overwrite it freely. Line two destroys the value line one made. The graph cannot afford that: backward will need every step. So autograd writes one record per change, and no record is ever overwritten. Watch the record change as y</code> does: >>> a = torch.ones(3, requires_grad=True) >>> y = a 2 >>> type(y.grad_fn).name 'MulBackward0' >>> y.add_(1) # change y in place >>> type(y.grad_fn).name 'AddBackward0' >>> y[0] = 9 # overwrite one slot >>> type(y.grad_fn).name 'CopySlices'</code></pre> </div> Figure 19. your code keeps one value and destroys the past. the graph keeps every step: one record per change, nothing overwritten.</figcaption> </figure> Three statements, three records, one chain. y.grad_fn</code> always holds the newest record, and each record points at the one before it, so the whole history stays reachable. That history is the program you never wrote. The machinery that keeps it correct under every kind of in-place change has real depth, and it is one of the best chapters of Part 2. 3. One list of operations is the whole interface</a></h3> full story: Part 5, The Machinery</div> </aside> </div> Figure 20. the code on the left never reaches a device. only the list does.</figcaption> </figure> The 3,677 registered names are PyTorch's real interface. Each backend implements its share of them. The compiler rewrites programs made of them: torch.compile</code> reads the list your program became and returns a shorter one, where a matrix multiplication and the add after it can fuse into one addmm</code>, and a chain of small elementwise operations becomes one generated kernel. Export formats store them: torch.export</code> writes the list to disk as a graph of exactly these names, and an ONNX file is the same idea with each name translated into ONNX's vocabulary. Quantization replaces them: the float32 matmul is swapped for an int8 body, the same place on the list, different arithmetic. When you meet a new PyTorch technology, ask one question first: what does it do to the operations? The answer usually explains the whole design. 4. PyTorch writes most of its own code</a></h3> full story: Part 5, The Machinery</div> </aside> </div> Figure 21. one yaml file in, thousands of functions out, at every build.</figcaption> </figure> native_functions.yaml</code> declares every operation. derivatives.yaml</code> declares every derivative. At build time, torchgen/</code> reads both and writes the Python bindings, the autograd record classes and the dispatcher tables. This is why searching the repository for a function you just called can find nothing: you searched the input of the build, and the function is in the output. People who work on PyTorch read the yaml first. After Part 5, so will you. 5. Features are layers with a switch</a></h3> full story: Part 5, The Machinery</div> </aside> </div> Figure 22. every feature watches the same stream of operations. a context manager puts one layer to sleep.</figcaption> </figure> PyTorch's features combine cleanly because each one is a layer in the dispatcher, watching the same stream of operations: >>> with torch.no_grad(): ... z = a * 2 >>> z.grad_fn is None # nothing was recorded True</code></pre> no_grad</code> edited no function. It set a flag that sends operations past the autograd layer, so nothing gets recorded. Mixed precision, tracing and vmap work the same way, and that is why they can be combined without knowing about each other. Part 5 opens the machinery under the flag. 6. Every operation pays a fixed cost first</a></h3> full story: Part 7, The Compiler</div> </aside> Half a microsecond of crossing and routing before any math, on every single operation. That was the measurement on the boundary floor. Applied honestly, this one number explains why torch.compile</code> exists, why fused optimizers exist, and why the first question about any slow model is: is it limited by compute, or by the cost of issuing many small operations? Interactive this instrument needs javascript; the still drawing stands in. </div> </div> Interactive 3. the same add at twelve measured sizes. the dots are measured on the author's machine; plant your flag before the curve appears. </div> 7. Memory, not speed, is what kills training runs</a></h3> full story: Part 4, Seeing PyTorch</div> </aside> </div> Figure 23. the forward pass borrows memory. backward repays it. running out is the most common way a training run dies.</figcaption> </figure> A slow program still finishes. A program that runs out of GPU memory dies with CUDA out of memory</code>, and that is the most common death in all of PyTorch. The forward pass saves values for backward (the turn, above). The allocator keeps and reuses blocks. Between them they decide how large a model you can train. This series treats memory as a first-class subject in Part 4. 8. Python is why it won, and what it costs</a></h3> full story: Part 7, The Compiler</div> </aside> </div> Figure 24. the protection and the price are the same picture: everything must cross one bridge.</figcaption> </figure> PyTorch won because you write it in ordinary Python, with ordinary debuggers and print statements. The price is the border cost from Idea 6, paid on every operation. The history of the framework is a sequence of attempts to keep the first while reducing the second. TorchScript tried to replace Python with its own language; it is now in maintenance mode [4</a>]. The current compiler watches your Python run and translates what it can, and it is winning. The pattern to remember: inside PyTorch, betting against Python has always lost. 9. Forward decides what backward must do</a></h3> full story: Part 9, Distributed</div> </aside> </div> Figure 25. backward is the reflection of the graph forward wrote.</figcaption> </figure> Backward can only walk what forward wrote. On one machine this sounds like a detail. At scale it becomes the law of the land: in distributed training, the way a tensor is split across GPUs in the forward pass decides which GPUs must exchange data in the backward pass. One idea, from a laptop to a cluster. It is the spine of Part 9. 10. Shared bytes plus in-place writes cause the hardest problems</a></h3> full story: Part 2, Autograd</div> </aside> </div> Figure 26. three windows, one storage, one write. every system that records or rewrites programs must handle this.</figcaption> </figure> Idea 1 lets many tensors read the same bytes. Idea 2 lets you change those bytes in place. Combine them: one write can change the meaning of several tensors at once, and any system that records programs (autograd, the compiler, export) must notice and stay correct. When a corner of PyTorch looks strangely complicated, ask what shared bytes plus an in-place write would do to it. That is usually the answer. 11. The code keeps its history</a></h3> full story: Part 5, The Machinery</div> </aside> </div> Figure 27. four systems, four eras, one repository. older layers still show through.</figcaption> </figure> The repository holds the remains of every era: the original C code from 2016, the Caffe2 merge of 2018, TorchScript from 2019, the compiler district growing since 2023. When a file looks strange, the explanation is usually historical: something older lived there first. Part 5 tells this history where it explains the present. 12. Floating point is a contract; read it</a></h3> full story: Part 4, Seeing PyTorch</div> </aside> </div> Figure 28. the terms are public. every training run signs them.</figcaption> </figure> >>> t = torch.tensor(1e8) >>> ((t + 1) - t).item() 0.0</code></pre> A float32 number has about 7 decimal digits of precision, so adding 1 to one hundred million changes nothing [5</a>]. This is not a bug; it is the number format doing what it promises. Add the faster, less precise formats used in training, plus the fact that some GPU kernels sum in different orders on different runs, and "why did my loss change between runs" becomes a question with exact answers. Part 4 reads this contract clause by clause. How this series draws</a></h2> Every still figure you just saw is a real Excalidraw scene, and the scene files ship with the series; you can open any drawing and edit it. The four instruments you can operate follow the same language, and every number inside them comes from the proof scripts. All of them speak one visual language, so that by Part 2 you read them without thinking: </div> Figure 29. the whole notation on one sheet. learn it once; it holds for the entire series.</figcaption> </figure> Orange always marks the subject: the one thing moving. Ink is structure. Grey is context. Dashed means recorded, implied, or asleep. The depth meter marks the floor. And the robot appears at most once per part, because a mascot that is everywhere stops being funny. How to read this</a></h2> Twelve parts. Each is one long page like this one. And the series has one quiet goal behind every part: by the end, you should know the machine well enough to build a small PyTorch yourself. Every drawing that shows a mechanism, every formula next to a figure, and every proof script is a piece of that. Part</th> What is behind the door</th></tr></thead> 0. The Map</td> you are here</td></tr> 1. Tensor</td> storage, strides, views, data types, broadcasting</td></tr> 2. Autograd</td> the graph, in-place writes, checkpointing, double backward</td></tr> 3. Daily PyTorch</td> nn, optim, data loading, mixed precision, seen from inside</td></tr> 4. Seeing PyTorch</td> the profiler, memory, floating point, honest measurement</td></tr> 5. The Machinery</td> the dispatcher, aten, torchgen, the history</td></tr> 6. Extending PyTorch</td> subclasses, custom operations, new backends</td></tr> 7. The Compiler</td> dynamo, aot autograd, inductor, dynamic shapes</td></tr> 8. Kernels & Hardware</td> the gpu model, triton, cutlass, what fast means</td></tr> 9. Distributed</td> collectives, ddp, fsdp, dtensor, parallel training</td></tr> 10. Ship It</td> export, quantization, executorch, the ecosystem</td></tr> 11. Working on PyTorch</td> the contributor's field guide</td></tr> </tbody></table> You do not have to read front to back. Three reading lines run through the parts, like lines through stations: Interactive this instrument needs javascript; the still drawing stands in. </div> </div> Interactive 4. pick a line. grey is front to back and stops everywhere; orange fits most readers; green chases speed; blue is the one for a new contributor. a dot means the line stops there. </div> Every chapter inside every part follows the same seven steps, so the rhythm becomes familiar fast: </div> Figure 30. the seven steps of every chapter. the proof step is the spine: no claim without a script.</figcaption> </figure> And the method, stated plainly, because you should know what you are trusting. Every mechanism claim is checked against the source code or shown by a script before it is published. The scripts are linked in place and pinned to one torch version. When PyTorch moves and a claim goes stale, the chapter is corrected and the correction is noted on the page. A series about internals that cannot admit drift would be wrong within a year. What you can now say</a></h2> Test yourself against this list. After one reading you should be able to say, in your own words: what type(torch.randn)</code> returns, and where the compiled body actually lives on your disk</li> what the dispatcher is, and how no_grad</code> stops autograd without editing any function</li> what a kernel is, and what decides which one runs</li> why the CPU and the GPU run on two clocks, and why that makes simple timing code lie</li> what the graph is, who writes it, and why backward can never do anything forward did not write down</li> and the twelve ideas, each in one sentence</li> </ul> If one of these is fuzzy, return to its floor; each one is only a minute long. That is what this page is for. Try it yourself</a></h2> The five proof scripts are the exercises. For each one: predict the output first, then run it, then explain the difference. p0_the_library.py</a>: how big are the compiled libraries in your own torch install?</li> p1_graph_chain.py</a>: what graph remains after a two-layer model runs?</li> p2_dispatch_cost.py</a>: what is the fixed cost per operation on your machine?</li> p3_two_timelines.py</a>: how long is your GPU still working after Python is done asking?</li> p4_micro_proofs.py</a>: the twelve ideas, compressed into five small experiments.</li> </ol> Pick a door</a></h2> Part 1 is the tensor. It opens with the puzzle from Idea 1, and now you have seen the error with your own eyes: >>> v.t().view(-1) RuntimeError: view size is not compatible with input tensor's size and stride ... >>> v.t().reshape(-1) # this one works. why? tensor([0., 3., 1., 4., 2., 5.])</code></pre> Same tensor. Same request. One line refuses, the other quietly copies the data. The difference between those two lines is the whole first part of this series. See you on the next floor down. References</a></h2> [1] Khalilli, five proof scripts, measured on an Apple M3 Max, torch 2.11.0, CPU and Apple GPU, 2026. Linked in place above; rerun them to check me. [2] PyTorch source, native_functions.yaml, pinned to the v2.11.0 tag. https://github.com/pytorch/pytorch/blob/v2.11.0/aten/src/ATen/native/native_functions.yaml</a> [3] PyTorch source, derivatives.yaml, pinned to the v2.11.0 tag. https://github.com/pytorch/pytorch/blob/v2.11.0/tools/autograd/derivatives.yaml</a> [4] PyTorch documentation, TorchScript, which states it is in maintenance mode. https://docs.pytorch.org/docs/stable/jit.html</a> [5] IEEE, 754 single precision: 24 binary digits of precision, about 7 decimal digits. Three good things to read after this page: Edward Yang's PyTorch internals talk</a>, which maps the C++ side in depth; the PyTorch Developer Podcast</a>, short episodes by the same author; and the repository's own CONTRIBUTING.md</a>, which describes the folder layout in the maintainers' words.

tensor - pytorch

The Map