Feeding data to a model

A short last batch is normal. Its loss is still an average, so it has the same scale as the others, but it is noisier, because fewer examples go into it. If you do not want it, PyTorch can skip it: drop_last=True leaves 2 batches of 4.

Each batch is one step: one forward pass, one backward pass, one update of the weights. So the number of steps is epochs × batches per epoch.

2. Shuffling each epoch

The 10 examples are stored sorted by class: examples 0–4 are class 0, examples 5–9 are class 1. That happens all the time: one file per class, or data collected one source after another.

The fix is to shuffle: a new random order of the examples every epoch. Each example is still used exactly once per epoch, but each batch now holds a mix, so its gradient is a fair sample of the whole dataset. In the lab, check Shuffle each epoch and press Next epoch a few times.

NumPy gives you a random order of 0, 1, …, n−1 with rng.permutation(n). Here rng = np.random.default_rng(0) is a random generator: an object that makes random numbers, starting from a number called the seed (here 0). For example, np.random.default_rng(0).permutation(10) is [4, 6, 2, 7, 3, 5, 9, 0, 8, 1].

Then you cut that order into batches with slices. idx[4:8] is items 4, 5, 6 and 7 of idx. A slice that runs past the end stops at the end: idx[8:12] of 10 items gives the last 2.

3. A held-out set and seeds

Level 8 kept some examples aside, never trained on, to measure how well the model does on data it hasn’t seen. That held-out set is separated once, before any training, and it never changes.

A teammate suggests a change: draw a fresh random split at the start of every epoch, “so the held-out score is fairer”.

Why two runs give different numbers

Run the same training script twice, and the loss curves are usually a little different. Randomness enters in several places:

1. the starting weights (level 7),
2. the order of the batches,
3. the held-out split,
4. dropout (level 8).

All of them come from random generators, and a generator started from the same seed gives the same numbers every time. np.random.default_rng(0).permutation(10) is [4, 6, 2, 7, 3, 5, 9, 0, 8, 1] today, tomorrow and on your friend’s computer. Seed 1 gives [8, 4, 7, 0, 1, 2, 5, 9, 6, 3]. In PyTorch, torch.manual_seed(0) at the top of a script fixes the seed for weights, dropout and shuffling at once (level 21 uses it).

4. Padding and the collate function

Text examples have different lengths. A batch must be a rectangle, an array of shape (B, L). So the shorter sequences get extra PAD tokens (here the number 0) at the end, until they are as long as the longest one. The function that turns a list of examples into one batch array is called the collate function.

Take three sequences: [5, 3, 8], [2, 9] and [7].

The PAD tokens carry no meaning. Two places must ignore them, and both need to know where they are:

• attention: no word may look at a PAD token. That is the padding mask of level 15.
• the loss: the model is never asked to predict PAD. That is ignore_index in level 17.

So the collate function returns a second array next to the batch: the padding mask, with the same shape, True where a cell is PAD. True means blocked, as everywhere in this course.

How much of a batch is padding depends on which sequences are put in the same batch.

Now write the collate function. NumPy pieces you need:

• np.full((2, 3), 0) is a (2, 3) array of zeros; np.ones((2, 3), dtype=bool) is all True.
• X[i, :k] = s writes the list s (of length k) into the first k cells of row i.
• for i, s in enumerate(seqs) gives each sequence s with its row number i.

import torch
from torch.utils.data import Dataset, DataLoader, random_split

class Pairs(Dataset):                       # __len__ and __getitem__, as in section 1
    def __init__(self, seqs): self.seqs = seqs
    def __len__(self): return len(self.seqs)
    def __getitem__(self, i): return self.seqs[i]

def collate(seqs, pad=0):                   # your function from section 4, in torch
    L = max(len(s) for s in seqs)
    X = torch.full((len(seqs), L), pad)
    for i, s in enumerate(seqs):
        X[i, :len(s)] = torch.tensor(s)
    return X, X == pad                      # the padding mask (if 0 is never a real token)

g = torch.Generator().manual_seed(0)        # a seeded random generator
# split once
train_set, held_set = random_split(Pairs(data), [0.8, 0.2], generator=g)
loader = DataLoader(train_set, batch_size=4, shuffle=True,
                    collate_fn=collate, generator=g)
for epoch in range(3):
    for X, mask in loader:                  # a new order every epoch
        ...                                 # forward, loss, backward, step