Level U3 · Foundations · Under the hood · runs in your browser

Build your own autograd

How can a computer find every gradient automatically?

Side trip · best after level 6 · Backpropagation

A short test to skip this level

Solve these 4 questions on your own. Answer all of them correctly and the level counts as cleared with three stars, and every part of the page opens. Showing an answer doesn’t count.

Number a = 2, b = 3, c = 1, d = −2. e = a·b, f = e + c, L = f·d. What is ∂L/∂a?

Number y = x · x with x = 3. What is dy/dx?

CodeWrite the backward rule of subtraction, out = self − other: one line for self, one line for other.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

CodeWrite the end of backward, after build(self): set the starting gradient of the output node, then call _backward on every node, in reverse topological order. Several lines.

import math

class Value:
    def __init__(self, data, children=()):
        self.data = data
        self.grad = 0.0
        self._children = children
        self._backward = lambda: None

def __add__(self, other):
        out = Value(self.data + other.data, (self, other))
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))
        def _backward():
            self.grad += (1 - t ** 2) * out.grad
        out._backward = _backward
        return out

def backward(self):
        topo, seen = [], set()
        def build(v):
            if v not in seen:
                seen.add(v)
                for child in v._children:
                    build(child)
                topo.append(v)      # a node goes in after all of its inputs
        build(self)
        ____

def grads(a, b, c, d):
    a, b, c, d = Value(a), Value(b), Value(c), Value(d)
    L = (a * b + c) * d
    L.backward()
    return [a.grad, b.grad, c.grad, d.grad]

print(grads(2, 3, 1, -2))

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

In level 6 you wrote the backward pass by hand. In PyTorch (level 9) one line, loss.backward(), does all of it. This level shows what that one line does inside. By the end you will have written the same thing yourself, in about 50 lines of Python.

It uses two ideas:

Record the work. While computing, remember every result, which operation made it, and from which inputs. That record is a computation graph.
Each operation knows only its own rule. Multiplication knows how its output moves when an input moves. So does addition. Nothing knows the whole network. The chain rule from level 6 combines the pieces: walk the graph backwards and multiply.

1. A graph you can step through

Take $L = (a \cdot b + c) \cdot d$ with $a = 2$ , $b = 3$ , $c = 1$ , $d = -2$ . Name the in-between results: $e = a \cdot b$ and $f = e + c$ . Press the main button (it starts as Compute e = a·b) to compute one node at a time. After $L$ , the same button runs the backward pass.

Backward starts by setting $\partial L / \partial L = 1$ . That only says “if L grows by a little bit, L grows by that same little bit”. It is the starting number; every other gradient is a product that starts from it. Then backward visits nodes from the end to the start. Visiting a node means one thing: push the node’s gradient into its inputs, using that node’s own rule.

the product $L = f \cdot d$ sends $d \times$ (L’s grad) to $f$ , and $f \times$ (L’s grad) to $d$ : each input gets the other input times the product’s grad
the sum $f = e + c$ sends f’s grad unchanged to both $e$ and $c$

Autograd, one node at a time: L = (a·b + c)·d

Press the main button to compute forward, one node at a time, then send gradients back. Change an input to try other numbers.

Forward ▸ Backward

Inputs a = b = c = d =

Nothing computed yet. Press “Compute e = a·b” to compute the first node.

values flow forward gradients flow back, times the factor on each edge

Number a = 2, b = 3, c = 1, d = −2. e = a·b, f = e + c, L = f·d. What is ∂L/∂d?

🔒 Answer the question above to unlock

$a$ is farther away. Its gradient travels $L \to f \to e \to a$ , multiplying by one local factor at each node: 1 at the sum, and $b$ at the product $e = a \cdot b$ .

Number a = 2, b = 3, c = 1, d = −2. e = a·b, f = e + c, L = f·d. What is ∂L/∂a?

Try it

Change $c$ to 10 and press Run all. $L$ changes, but every gradient stays the same, because $c$ is only ever added. Now change $b$ . Which gradients move, and why does $\partial L / \partial b$ not depend on $b$ itself?

🔒 Answer the question above to unlock

2. Each value remembers where it came from

Here is the core of an autograd engine. A Value holds a number (data), its gradient (grad), the values it was made from (_children), and a small function _backward that knows the local rule. Addition is done. Multiplication is your turn: the rule is the one from the lab.

Three pieces of Python that may be new to you:

def __add__(self, other) runs when you write a + b with two Values; __mul__ runs for a * b. self is the left one, other the right one.
lambda: None is a tiny function that does nothing. It is the _backward of a value that was not made by an operation.
_backward is defined inside __mul__, so it can still use self, other and out later, when backward calls it. The line out._backward = _backward stores that function on the result, for later.

CodeWrite the gradient that a product sends to its first input.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

I got stuck here Why does _backward use out.grad? Where does that come from?

When _backward runs, out has already received its own gradient from the nodes after it. That is what visiting from the end guarantees. out.grad is “how much L moves per unit of out”. The local rule turns it into “how much L moves per unit of self”. That one multiplication is the chain rule.

🔒 Answer the question above to unlock

3. When a value is used twice

In a real network one weight feeds many places. The simplest case: $y = x \cdot x$ . Here $x$ is both inputs of the same multiplication. Calculus says $dy/dx = 2x$ .

Number y = x · x with x = 3. What is dy/dx?

In x * x, self and other are the same object: both names point to x.

ChooseSomeone writes self.grad = … and other.grad = … (with =, not +=). For y = x · x at x = 3, what does their backward give for x.grad?

So each of the two lines in _backward sends a part of the gradient to the same x: self.grad gets $x \times 1$ , and other.grad gets another $x \times 1$ . Only adding both parts gives $2x$ .

I got stuck here So why += instead of =?

A value that feeds several places changes $L$ through all of them. Its total gradient is the sum of what each path sends back. += collects them. With =, every path overwrites the one before, and only the last path counts.

This is also why loss.backward() in PyTorch adds into .grad, and why you call opt.zero_grad() before every step (level 9). Adding is right inside one backward pass. Across steps, old sums must be cleared.

Now give Value a new operation yourself: subtraction, out = self - other. Its forward part is done. For the backward rule:

If self grows by a little, out grows by the same amount. The local factor is 1, as in a sum.
If other grows by a little, out shrinks by that amount. The local factor is −1.

So self gets 1 × out.grad and other gets −1 × out.grad. Write both lines, and keep adding into .grad: other.grad -= out.grad is short for other.grad = other.grad - out.grad, the same as adding −1 × out.grad.

CodeWrite the backward rule of subtraction, out = self − other: one line for self, one line for other.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

🔒 Answer the question above to unlock

4. The order of visits

A node may only be visited after every node that uses it has been visited. Otherwise its gradient is not complete yet. Listing the nodes so that each comes after its inputs is called a topological order. For the lab’s graph one such list is [a, b, e, c, f, d, L]: every node appears after the nodes it was made from.

The code builds that list with a small recursive function, build: before adding a node, it first builds each of its children (a depth-first walk), so the children always land earlier in the list. Then backward visits the list from the end, starting at L.

One more piece of Python: reversed(xs) walks a list from its last item to its first. For example, list(reversed([1, 2, 3])) is [3, 2, 1].

Write the end of backward, after build(self): give L its starting gradient (section 1), then visit every node in the right order and call its _backward. The tanh method is done too, so the last test is a real neuron.

CodeWrite the end of backward, after build(self): set the starting gradient of the output node, then call _backward on every node, in reverse topological order. Several lines.

import math

class Value:
    def __init__(self, data, children=()):
        self.data = data
        self.grad = 0.0
        self._children = children
        self._backward = lambda: None

def grads(a, b, c, d):
    a, b, c, d = Value(a), Value(b), Value(c), Value(d)
    L = (a * b + c) * d
    L.backward()
    return [a.grad, b.grad, c.grad, d.grad]

print(grads(2, 3, 1, -2))

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

Go deeper This is what loss.backward() does

PyTorch works the same way, with three differences, all only in size:

it works on whole tensors, so one node is a matrix multiplication instead of one product of two numbers;
it knows the local rule for hundreds of operations: @, exp, softmax, layer_norm, …;
the rules are written in fast compiled code, and can run on a GPU.

The order of visits, the +=, the starting gradient $\partial L / \partial L = 1$ : all the same. tanh shows the pattern for any new operation. Forward computes $t = \tanh(x)$ , backward adds $(1 - t^2) \times$ out.grad. Once an operation has those two pieces, the engine can use it anywhere.

Recap

a summary for when you finish the level

The key formulas and common mistakes appear here once you clear the level.

You can now

Find every gradient of a small graph by hand, visiting the nodes from the end.
Write the backward rule of a new operation as a _backward function that adds into .grad.
Put the nodes in topological order and walk it backwards to run the whole backward pass.

Keep in mind

Start with $\partial L / \partial L = 1$ ; each node sends its grad × the local factor to its inputs
Sum: both inputs get out.grad. Product: each input gets the other input’s data × out.grad
Always += into .grad: a value used in several places gets the sum of all its parts
Give the output a gradient of 1, then call every node’s _backward, from the output back to the inputs

Common mistakes

Writing = instead of +=, so a value used twice keeps only the last of its parts.
Visiting the nodes in forward order, before their gradients are complete.

‹ Back to level 6 · BackpropagationU4. Feeding data ›

Press ? for keyboard shortcuts