TejasviTomar
October 2026 · Preprint

Tied group sizes make GroupKFold folds depend on the scikit-learn version and the CPU

Tejasvi Singh Tomar

In plain words

Grouped cross-validation splits data into folds and keeps each group, such as one patient's records, in a single fold, so no model is tested on a group it was trained on. Up to version 1.8, scikit-learn's GroupKFold put equal-sized groups into folds in an order that could depend on the numpy version and the processor. Version 1.9 gives the same folds on every machine, but when group sizes tie, its folds differ from those of earlier versions. In one project, the order of the equal-sized groups decided whether a statistical test, registered in advance, passed.

Abstract

GroupKFold, the grouped K-fold splitter of scikit-learn, sorts groups by size and deals them to folds. Up to version 1.8 it used numpy's default argsort, which does not fix the order of equal elements and whose SIMD kernels numpy chooses at run time from the CPU. Whenever two groups had the same size, the folds could therefore depend on the numpy version and the CPU. scikit-learn 1.9.0 (June 2026) sorts stably, which removes the CPU dependence but changes such folds relative to all earlier releases. The problem was reported and fixed upstream. We document its consequences. In one research project, 79 of 80 GroupKFold call sites had tied group sizes, and the non-stable orders moved 16 to 79% of the items to another fold. About 60 of the 188 fold-dependent claims of a paper draft changed at printed precision, one comparison reversed, and a pre-registered decision passed under one tie order (Holm-adjusted p = 0.022) and failed under another (p = 0.120). A self-contained example on the digits data bundled with scikit-learn reproduces the effect. In a pre-registered random sample of public code, 92.8% of GroupKFold uses (95% interval 86.8 to 98.8%) run without shuffling on groups whose sizes can tie. All 8 uses whose data we could rebuild had tied sizes, and in 5 the folds of versions 1.8 and 1.9 differ. We recommend storing folds as data, reporting library versions and checking decisions over several tie orders, and give a helper with a documented tie-break.

Two sets of five columns of squares on a white background, each set standing on a black line. Every column is six squares tall: a block of three squares at the bottom, a block of two above it and a single square on top, each block outlined in black. In the left set, each column is a single colour: from left to right, teal, orange, magenta, yellow and white. The right set has blocks of the same sizes in the same places, but their colours are mixed, so that four of its five columns hold three different colours.
Illustration of how GroupKFold deals groups to five folds, numbered from left to right. Fifteen groups (five with three samples, five with two and five with one) are drawn as blocks of squares, one square per sample. The groups are dealt from the largest to the smallest, each into the fold that holds the fewest samples so far (the lowest-numbered one when several are equally full). In each fold's column, the groups stack up from the bottom. Left: the groups, each coloured by its fold, with those of equal size dealt in one order. Right: the same groups, keeping their colours, with those of equal size dealt in another order. Every fold still holds six samples, but 11 of the 15 groups are in a different fold. Up to scikit-learn 1.8, the order of equal-sized groups could depend on the numpy version and the CPU; from 1.9 on, it follows the group labels. In the paper's example using the digits data bundled with scikit-learn, versions 1.8.0 and 1.9.1 put 73 to 76% of the samples in different folds.