TejasviTomar
October 2026 · Preprint

Gene position and relatedness confound genomic language model benchmarks

Tejasvi Singh Tomar

In plain words

Language models trained on genomes are ranked on public benchmarks, and high scores are taken to show that the models understand genome organisation. In six such benchmarks, much of a model's score is often explained by information available without running any model: where genes sit and whether a test genome has relatives in the training data. Used alone, none of the language models tested matches a simple distance rule on published tests of finding neighbouring bacterial genes that are switched on together. Some real signal remains, though it often comes from what the genes are rather than where they sit. For each benchmark flaw it finds, the paper proposes a fix.

Abstract

Benchmark scores of genomic language models are read as evidence of learned genome organisation and gene function; we tested, with baselines that use no sequence model, whether gene position and genome relatedness explain them instead. The answer depends on how a benchmark's labels, inputs and splits were built: where these encode position or relatedness, such baselines can explain much of a model's score. Position alone outperforms every language model we compared on bacterial operon benchmarks (as in a companion study), and distance matches or beats most published DNA language models on most eukaryotic long-range tasks. On two label sets we built without genome layout, no model we tested gains more than 0.002 AUROC over position on operons labelled from nanopore reads of expressed genes, and little model signal on experiment-only interaction labels survives strict controls. Relatedness explains much of the models' performance on BacBench's two genome-level tasks, and train–test overlap or negatives built from positives let a rule or lookup exceed the best published score we found on 24 of 86 tasks of widely used DNA benchmarks. Real signal remains on gene-level tasks, where it appears to come from the identity of genes and their neighbours; signal beyond position also remains on STRING interaction labels and in DNA language models fine-tuned on a causal eQTL task, but it is often not specifically genomic. Benchmarks therefore need these baselines, metrics with position held fixed, labels built without layout, and leak-free inputs and splits; we list a fix for each defect and release seqleak, which computes the position baseline and these metrics.

Two rings of small dots on a white background, each around a black circle, with a dashed circle running through each ring. In the right ring almost all dots inside the dashed circle are teal and almost all outside it are orange, with a sharp border between them. In the left ring many teal dots also lie outside the dashed circle among the orange ones, and some orange dots lie inside it.
Illustration of DGEB's operon labels on two bacterial chromosomes drawn as circles: Escherichia coli (left), with labels from the curated EcoCyc database, and Synechococcus elongatus (right), with labels from a BioCyc database whose operons were predicted by the PathoLogic program. Each dot is a pair of neighbouring genes on the same strand; its position around the circle gives the pair's location on the chromosome, and the larger the gap between the two genes, the farther the dot lies from the centre. A dot is teal when DGEB puts its two genes in one transcription unit and orange when it does not. Opposite-strand pairs, all labelled negative, are not shown. A few E. coli pairs lie off the scale: those with the largest overlaps are drawn at the inner edge of the ring of dots, and those with the largest gaps at its outer edge. The dashed circle marks a gap of 40 bp. In S. elongatus, almost all pairs inside the dashed circle are teal and almost all outside it are orange. The paper finds that the rule "same strand and −21 ≤ gap ≤ 40", fitted to the S. elongatus labels, reproduces 98.7% of those labels, including the ones for opposite-strand pairs. In E. coli, by contrast, many teal pairs lie outside the dashed circle.