Source: the arXiv abstract page for 2610.12444 (cs.LG, 23 pages, v1, submitted 8 October 2026). We summarise the abstract and did not run the code.
The problem: AdamW keeps two running statistics per weight. Storing them in 4 bits saves memory, but rounding error in those states feeds into later moment updates and distorts adaptive step sizes.
The authors say the error should be measured where the quantizer actually chooses between neighbouring levels, which they call rounding space. For the second moment, a small average error in the stored value does not guarantee a small error in the preconditioner used at the next step, especially in the quantization cell nearest zero. A one-dimensional quadratic toy problem shows rounding in state space and in preconditioner space can lead to qualitatively different optimisation behaviour.
ZIP-SR keeps zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space. ZE-EDEN instead excludes zero from the codebook and rescales each quantized block to offset the bias from the positive quantization floor. Both use NF4 for the first moment and stochastic rounding for the language-model head first moment during the last 10% of training.
Results: on pretraining from 130M to 2.7B parameters, both reduce the mean validation-loss gap between TorchAO's 4-bit AdamW and 32-bit AdamW at every model size. The largest reported reduction is 70%. In full-parameter supervised fine-tuning both reach lower validation loss than TorchAO and stay close to 32-bit AdamW on downstream tasks.
Limit: no results above 2.7B parameters are reported in the abstract, so the benefit at frontier scale is untested.