Arena AutoEval Simulates Human Scoring, Generating Model Rankings in Under 1 Hour

Arena deploys AutoEval to simulate human voting via reward models, delivering estimated leaderboard scores for new AI models in under one hour with high correlation to actual results.

Woofun AI reports that Arena has introduced AutoEval, a system utilizing a reward model trained on millions of user preference sets to simulate human voting and generate leaderboard scores for new models within one hour. This mechanism allows for immediate estimated rankings across text, vision, audio, and code models, which are subsequently refined by actual human votes. Backtesting indicates AutoEval maintains a correlation exceeding 0.98 with final human rankings, achieving over 90% accuracy in determining winners when score differences surpass 10 points and reaching 100% accuracy when gaps exceed 15 points, though differentiation remains challenging for margins under 5 points.