PyTorch

---

name: PyTorch description: Avoid common PyTorch mistakes — train/eval mode, gradient leaks, device mismatches, and checkpoint gotchas. metadata: {"clawdbot":{"emoji":"🔥","requires":{"bins":["python3"]},"os":["linux","darwin","win32"]}}

Train vs Eval Mode

• model.train() enables dropout, BatchNorm updates — default after init
• model.eval() disables dropout, uses running stats — MUST call for inference
• Mode is sticky — train/eval persists until explicitly changed
• model.eval() doesn't disable gradients — still need torch.no_grad()

Gradient Control

• torch.no_grad() for inference — reduces memory, speeds up computation
• loss.backward() accumulates gradients — call optimizer.zero_grad() before backward
• zero_grad() placement matters — before forward pass, not after backward
• .detach() to stop gradient flow — prevents memory leak in logging

Device Management

• Model AND data must be on same device — model.to(device) and tensor.to(device)
• .cuda() vs .to('cuda') — both work, .to(device) more flexible
• CUDA tensors can't convert to numpy directly — .cpu().numpy() required
• torch.device('cuda' if torch.cuda.is_available() else 'cpu') — portable code

DataLoader

• num_workers > 0 uses multiprocessing — Windows needs if __name__ == '__main__':
• pin_memory=True with CUDA — faster transfer to GPU
• Workers don't share state — random seeds differ per worker, set in worker_init_fn
• Large num_workers can cause memory issues — start with 2-4, increase if CPU-bound

Saving and Loading

• torch.save(model.state_dict(), path) — recommended, saves only weights
• Loading: create model first, then model.load_state_dict(torch.load(path))
• map_location for cross-device — torch.load(path, map_location='cpu') if saved on GPU
• Saving whole model pickles code path — breaks if code changes

In-place Operations

• In-place ops end with _ — tensor.add_(1) vs tensor.add(1)
• In-place on leaf variable breaks autograd — error about modified leaf
• In-place on intermediate can corrupt gradient — avoid in computation graph
• tensor.data bypasses autograd — legacy, prefer .detach() for safety

Memory Management

• Accumulated tensors leak memory — .detach() logged metrics
• torch.cuda.empty_cache() releases cached memory — but doesn't fix leaks
• Delete references and call gc.collect() — before empty_cache if needed
• with torch.no_grad(): prevents graph storage — crucial for validation loop

Common Mistakes

• BatchNorm with batch_size=1 fails in train mode — use eval mode or track_running_stats=False
• Loss function reduction default is 'mean' — may want 'sum' for gradient accumulation
• cross_entropy expects logits — not softmax output
• .item() to get Python scalar — .numpy() or [0] deprecated/error