autoresearch vs sia
Karpathy's autoresearch loop as an installable skill for Claude Code, OpenCode and Codex: constraint + mechanical metric + autonomous modify→verify→keep/discard iteration. — versus — Self-improving loop from the SIA paper: Meta, Target and Feedback agents evolve a task agent's harness AND weights against a benchmark — #1 on MLE-Bench Hard, 14x kernel speedups.
Same lineage — the autoresearch loop: metric, modify, verify, keep or discard. The skill hill-climbs your code with a fixed agent; SIA makes the agent itself the thing that improves, up to and including its weights.
| autoresearch | sia | |
|---|---|---|
| Stars | 5.4k | 2.1k |
| Forks | 399 | 244 |
| Language | Shell | Python |
| License | MIT | MIT |
| Last activity | 1 months ago | 21 days ago |
| Topics | skills, coding | agents, training |
| Curated connections | 4 | 2 |
autoresearch — the curator's take
The compounding-gains loop, packaged: pick a metric a machine can check, let the agent mutate-verify-keep against it for hours, and small wins stack — the Karpathy recipe without writing the harness yourself. Works across three agent CLIs. The discipline it demands is the catch: without a truly mechanical metric the loop optimizes noise, and unattended iteration burns real tokens — set budgets before you set it loose.
sia — the curator's take
The most ambitious of the self-improvement loops on the map: where others mutate prompts or code, SIA's Feedback agent rewrites the task agent's harness and updates its weights, with paper-grade receipts (70.1% on LawBench vs 45% prior SOTA, 14x on an AlphaFold-3 Triton kernel, #1 on MLE-Bench Hard). Reach for it when your problem IS a benchmark: a scoreable task the loop can grind against. NOT for tasks without a mechanical metric, and not a polished product — it's the official research implementation, so expect to build the task harness yourself and budget real GPU time for the weight-update path.