AHK-Eval Leaderboard
The canonical board for the 36-task, 181-hidden-case AHK v2 suite. Cold = one-shot API call per task; the harness arms re-run the same tasks with a rules card, a repair round, or headless Claude Code. Click a header to sort. The GUI benchmark lives on the clipboard leaderboard.
| # | Model | cold /36 | cases /181 | parse fails | +rules | +repair | CC | out $/M |
|---|
| Model | GUI code /100 | GUI visual /20 | OOP judge /100 | Toolkit cases % | AHK-Eval /36 | Eval +repair /36 |
|---|
| Model | SWE-bench Verified | in $/M | out $/M | AHK-Eval cold |
|---|
SWE-bench Verified scores as published on llm-stats and BenchLM, 2026-07-09; GPT-5.5 is OpenAI-reported (OpenAI stopped submitting Verified in early 2026). xAI does not report Verified — it publishes SWE-bench Pro instead (Grok 4.5: 64.7%), which isn't comparable and so isn't plotted. Likewise GPT-5.6 (launched 2026-07-09) ships without a Verified score — OpenAI reports DeepSWE v1.1 and SWE-bench Pro for it — so the 5.6 family sits in the table unplotted. Pricing from the OpenRouter API, 2026-07-09. AionLabs' Aion-3.0 systems (multi-model storytelling ensembles on GLM / DeepSeek bases) publish no Verified score. Grok Latest is OpenRouter's rolling alias — this run resolved to grok-4.5-20260708 and scored 28/36 against the pinned Grok 4.5 endpoint's 31/36, a caution on reading single-run deltas at temperature 0.2; the alias is skipped in the charts. The AHK-Eval chart plots paid endpoints only — free tiers ($0) can't sit on a log cost axis. In the grid, rows are exact model endpoints: GUI-board and OOP-judge cells fill only where that same endpoint ran the GUI benchmark (its GPT-5.5 Pro and Kimi K2.7 entries are different models, so those cells stay empty). Toolkit is hidden-case % from the 12-function one-shot — the zeros are single parse errors zeroing the whole file. Eval +repair is round 2's rules-card + one-repair arm.