Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

Introduces the Skaling law, a neural scaling law that corrects a structural weakness in the standard Chinchilla formulation. Whereas Chinchilla models parameter count and training data as independent additive contributions to loss, Skaling adds one interaction exponent that couples the two. This removes the saddle-shaped prediction bias that appears when models are severely data-limited or overtrained, reducing interpolation and extrapolation mean absolute percentage error by 1.5–3×.

The FAIR team validated the law on an internal SK-Grid of 125 pretraining runs spanning 14 model sizes (134M–4.9B parameters) and 316M–316B tokens, as well as Farseer and original Chinchilla data. On 112 held-out high-compute Farseer runs, Skaling reached 0.60% pooled MAPE, 3.9× lower than Chinchilla. Its sparse L-shaped profiling strategy samples only inexpensive edges of the model-size/data grid yet reconstructs the full loss surface with comparable accuracy using approximately 10× less compute, making compute-allocation decisions possible from substantially cheaper pilot training runs.