TejasviTomar
October 2026 · Preprint

Copy and relatedness leaks in genomic language model benchmarks are common and can be found and largely removed automatically

Tejasvi Singh Tomar

In plain words

In some public tests for models that learn from DNA or RNA, the answer can be copied from a near-identical training example. The paper introduces seqleak, an automatic checker that finds such leaks. The checker was run on 41 tests that were not used to tune it. It flagged every one in which a manual audit had found copies or close relatives of training examples. For 60 widely used tests, the paper provides versions with fewer leaks.

Abstract

Genomic language model benchmarks rank models, but on some tasks a no-model rule, such as a nearest-training lookup that copies the label of the most similar training item, matches or beats the best published score. Such rules exploit leaks: test items that are copies or relatives of training items, negatives constructed from the positives, and genomic positions that reveal the label. Copies and relatives occur in 8 of the 11 DNA and RNA benchmarks published from 2024 to 2026 that we audited, although on most tasks no leak rule comes close to the published scores. The frozen rules of our linter, seqleak, recover all 24 copy and relatedness flags of that audit on 41 tasks not used to set them. In pre-registered fine-tuning experiments on GUE's yeast histone-mark tasks, however, relatives in training raise test scores only slightly on average. Constructed negatives are rarer, mostly draw only a warning from the frozen rules and survive re-splitting. Yet each of six models fine-tuned on GUE's constructed negatives instead of matched genome windows scores lower on matched genomic negatives, by 0.15 and 0.41 AUROC on average for shuffled and segment-substituted negatives, net of the same comparison on control tasks. On independent G+C-matched test sets the losses are 0.14 and 0.31, whereas an earlier independent test without G+C matching did not meet its registered criterion. About two thirds of the improvement reported on GUE and the original Nucleotide Transformer tasks since their release does not exceed a no-model leak rule's score, so published scores alone cannot tell it apart from leak exploitation; on Genomic Benchmarks, most of it lies above. We release the linter, leak-reduced versions of 60 tasks, of which the re-split ones remove most of the lookup's signal, and PairCtrl, a leaderboard that controls for genomic position.

Three charts stacked. At the top, four small step plots, one each for GUE, the original and the revised Nucleotide Transformer tasks and Genomic Benchmarks, show the best published score from 2023 to 2026 in units of a no-model leak rule, drawn as an orange line at 1: thin grey lines for single tasks and a thick blue median, which stays close to the orange line for GUE and the original Nucleotide Transformer tasks and lies above it for the other two. In the middle, horizontal bars split each benchmark's reported improvement into an orange part at or below the leak ceiling and a blue part above it: 36% above for GUE, 37% for the original Nucleotide Transformer tasks, 100% for the revised ones and 86% for Genomic Benchmarks. At the bottom, dots show how well model rankings on the original and the re-split version of each task agree, with means of 0.83 for tasks with a documented leak and 0.79 for tasks without one.
Figure 2: Reported progress and the no-model leak ceiling, for the primary row set P′ with the registered set P alongside each label. (a) Best published score over time for tasks with a documented leak, in units of the leak rule’s above-chance score on the same test set (1 = the rule): one grey line per task and the median in blue. (b) Share of the reported improvement over the benchmark authors’ best that lies above the leak ceiling (blue) or at or below it (orange), with ceiling-bootstrap intervals. The revised Nucleotide Transformer bar is 100% largely by construction. (c) Kendall’s τ between the rankings of four to eight fine-tuned models on the original and the re-split version of each task, for tasks with and without a documented leak: one point per task, the mean as a diamond and the seed split-half reliability as a bar. 7 of the 13 re-split versions with a documented leak keep their constructed negatives; with fixneg versions the mean is 0.752 compared with 0.878, and at the last checkpoint 0.767 compared with 0.881 (Appendix G).