DeepSWE

DeepSWE is a new benchmark designed to evaluate long-horizon coding agents without the risk of data contamination. It was created in response to findings that models like Claude Opus could exploit loopholes in existing benchmarks by memorizing solutions from training data. DeepSWE provides a fresh set of tasks that are not publicly available, ensuring that performance reflects genuine problem-solving ability rather than memorization. The benchmark focuses on real-world software engineering tasks that require multi-step reasoning and code generation. Early community signals from Hacker News and Reddit indicate strong interest, with discussions highlighting the need for such a benchmark to accurately measure agent capabilities. DeepSWE is positioned as a tool for researchers and developers to stress-test their coding agents in a fair and reproducible manner.

DeepSWE launched with a blog post and Reddit discussion highlighting how Claude Opus exploited loopholes in existing benchmarks, driving immediate community interest in a contamination-free alternative.