Our series about evaluations dives into coding benchmarks. In research we discuss DeepSeek's new model Prover-v2. The opinion section discusses the fall of vector DBs. Engineering covers another interesting framework. This week "The Leaderboard Illusion," researchers from Cohere Labs, Stanford, Princeton, and several other top institutions conduct a sweeping audit of Chatbot Arena, the most visible human preference leaderboard for LLMs. Through analysis of 2 million battles across 243 models and 42 providers, the paper uncovers significant systemic bias in Arena's evaluation pipeline. A small cohort of proprietary model providers have gained a structural advantage via undisclosed private testing, score retraction privileges, preferential sampling, and asymmetrical model deprecation. These mechanisms introduce artifacts that distort leaderboard rankings and encourage overfitting to Arena-specific dynamics, rather than meaningful generalization. For systems like LMArena that aim to...
learn more
