SWE-bench vs preference Elo
Why the best coding model is not always the best chatbot, and how CompareLLM keeps both numbers visible.
They measure different things
Elo is “which anonymous reply did a human prefer?” SWE-bench is “did an agent resolve a real issue under a named harness?” A model can win one and lose the other without anyone lying.
Harnesses matter more than people admit
The same weights with a different agent, more retries, or a different split will move SWE-bench by several points. We keep the evidence URL and date so you can see which table an administrator approved.
Use both on a compare page
If you are buying a coding agent, sort the coding leaderboard, then open vs pages against your current model. If you are buying a general assistant, start from Elo and still glance at SWE-bench so you do not pick a charming model that cannot edit a repo.
Ready to evaluate your stack?
Calculate your optimal model weights with Stack Engine or compare top models head-to-head.
Related guides
Same topic, next level of detail.
What is Elo on an AI leaderboard?
Elo is a crowd vote: people pick which hidden answer they like more. A higher Elo means more people preferred that model — not that it passed a test.
What is LiveBench?
Contamination-resistant objective tasks. Only compare scores from the same LiveBench release.
How to read LLM benchmarks without getting fooled
A short checklist: same split, same date, named source, no composite index you cannot audit.
Explore More AI Intelligence Tools
Pairwise Model Comparisons
Compare any two models on our category ratings, speed classes and token pricing.
Curated 'Best-Of' Indexes
Targeted rankings for Best Coding LLMs, Best Cheap APIs, Shortest Wait, and top-tier models.
AI Hardware Calculator
Interactive VRAM calculator, quantization levels (FP16, Q8, Q4), KV cache context, and local hardware fit.
Live Market Intelligence
Live audit trail of benchmark updates, new model releases, and API price cuts.
Interactive Model Finder
Answer 5 quick questions to compute deterministic model recommendations for your use case.
Methodology & Standards
How a CompareLLM rating is computed, what evidence it admits, and how prices and speed classes are recorded.