Woofun AI reports that OpenAI has withdrawn its recommendation for SWE-Bench Pro after an audit identified significant flaws in the benchmark's integrity. The review of 731 public questions found that 249, or 34.1%, were unsuitable for assessment due to issues such as incomplete descriptions, overly strict requirements, and hidden test criteria.

OpenAI noted that these defects cause scoring inaccuracies, where pass rates do not necessarily reflect true problem resolution. With state-of-the-art model pass rates on this set rising from 23.3% to 80.3% over eight months, the company warned that such improvements may be artifacts of noise rather than capability gains, calling for the community to develop more reliable evaluation standards.