#10 · ML / AI methods

Omics prediction benchmarks often fail to test the biological property they claim

The common thread is that evaluation design, not predictive accuracy alone, determines what a model is shown to have learned. Perturbation studies compare predictions for unseen gene pairs against summed single-gene effects through additive disagreement, report that predictions can keep shared perturbation effects while collapsing context-specific responses, and reassess whether deep learning perturbation models outperform baselines once a positive-control baseline and calibrated metrics are applied across 14 datasets and 18 metrics. Benchmarks of graph models for toxicity prediction and regulon-based representations for drug combinations similarly test generalization under unseen-cell and split regimes. What is new is explicit attention to the limits of the data and the metric rather than architecture alone. The thread is dominant rather than universal: the feature-stability criterion IQC and the antibacterial synergy model HALO address related validation questions in other settings.