Matrices and shapes

2. The shape rule

The shape rule comes from that one rule. To multiply a row of A with a column of B number by number, the row and the column must have the same length. In NumPy, and everywhere in this course, @ means matrix multiplication. So:

The two inner numbers must match, and they disappear. The two outer numbers are what is left. When the inner numbers match, we say the shapes line up.

The flip in that answer has a name: the transpose, written B.T. Row i of B becomes column i of B.T, so a (3, 2) matrix turns into (2, 3). You will use it in level 6 and again in attention.

3. Write it yourself

Write the multiplication with three loops, without @ or np.dot. The innermost loop runs over the inner size, which is the dimension that disappears.

4. Shapes you will see a hundred times

A real model does not multiply one sentence at a time. It stacks several sentences into a batch, so the input has three dimensions: (batch, length, features). Multiplying by a weight matrix only touches the last dimension. The rule is the same: the inner numbers must match and disappear.

5. Broadcasting: adding arrays of different shapes

Adding two matrices of the same shape is easy: add cell by cell. But models constantly add things that are smaller: one bias row to every example, one mask row to every word. NumPy has a rule for that, called broadcasting: a dimension of size 1 is stretched (copied) until it matches the other array.

A column (4, 1) plus a row (1, 5): the column is copied across 5 times, the row is copied down 4 times, and then they are added cell by cell.

That case had a 1 in each array. What if one array has no 1 at all?

Pick a case and point at or tap a cell of the result. The pale cells are the stretched copies. NumPy never stores them. The lab uses its own numbers (a = [0, 1, 2, 3]), so its table is not the answer to the question below.

The rule, checked one dimension at a time from the right:

1. If the two sizes are equal, fine.
2. If one of them is 1, stretch it to the other.
3. Otherwise, error.

A missing dimension on the left counts as 1, so a (5,) array behaves like (1, 5).

6. Where you will need it

This section looks ahead. Levels 10–15 explain its words; only the shapes matter here. In levels 14–15 the attention scores of a batch have shape (batch, words, words): for every sentence, every word’s score for every other word. Some sentences are shorter and end in padding words, so each sentence comes with a padding mask: True marks a padding word that must be blocked (its score is set to −∞ before softmax, which you will meet in level 10). The mask is stored once per sentence, as (batch, 1, words), and broadcasting stretches it to every row.

7. Adding up along one axis

Many steps add numbers up along one direction of a table. In NumPy you choose the direction with axis. Take A with shape (3, 2):

A = [[1, 2],
     [3, 4],
     [5, 6]]
A.sum(axis=0)  →  [9, 12]       # go down the rows: one total per column, shape (2,)
A.sum(axis=1)  →  [3, 7, 11]    # go across a row: one total per row, shape (3,)
A.mean(axis=0) →  [3, 4]        # the same, divided by the 3 rows

axis=0 removes the rows. axis=1 removes the columns. The axis you name is the one that disappears.

Now put the two ideas together. X.mean(axis=0) has shape (d,): one mean per column. Subtracting it from X (shape (N, d)) is a broadcast: the row of means is stretched down to all N rows. The result has every column centered at 0. LayerNorm in level 16 centers in the same way, but each row instead of each column: it takes the mean along the last axis, one mean per word, and subtracts that.

# stack 60 rows on top of 60 rows → (120, 2)
pts = np.vstack([np.zeros((60, 2)), np.ones((60, 2))])
# 60 zeros, then 60 ones → (120,)
labels = np.repeat([0, 1], 60)

Optional side trip: level U1 shows how the computer stores an array like these as one row of numbers, and why some reshapes cost nothing. Level 2 continues the main line; you can come back to U1 any time.