JIT vs sia
JIT-Agent: a meta-agent that writes a task-specific harness on the fly — memory, planning, action and capability modules emitted as code — and revises it from traces at test time. — versus — Self-improving loop from the SIA paper: Meta, Target and Feedback agents evolve a task agent's harness AND weights against a benchmark — #1 on MLE-Bench Hard, 14x kernel speedups.
Both come from the 'improve the harness, not just the model' line of work. SIA's Meta/Target/Feedback loop evolves a task agent's harness and weights over time; JIT-Agent generates a fresh harness per task at test time and keeps the generator frozen.
| JIT | sia | |
|---|---|---|
| Stars | 381 | 2.1k |
| Forks | 43 | 255 |
| Language | Python | Python |
| License | NOASSERTION | MIT |
| Last activity | 9 days ago | 10 days ago |
| Topics | agents, training | agents, training |
| Curated connections | 2 | 3 |
JIT — the curator's take
Research code for one sharp idea: the scaffold around an LLM is a trainable, transferable axis of intelligence, separate from the base model. Given a task spec, a protocol and a tool registry, JIT-Agent-27B emits an executable harness (four modules over HarnessFactory interfaces, so it's structured code, not a free-form program), then repairs it as feedback arrives while the generator stays frozen. Reproduce the paper with the runners and benchmark adapters. NOT a framework you build on — no licence file, conda env, benchmark-shaped — and it assumes you serve the meta model yourself (vLLM/SGLang over HTTP).
sia — the curator's take
The most ambitious of the self-improvement loops on the map: where others mutate prompts or code, SIA's Feedback agent rewrites the task agent's harness and updates its weights, with paper-grade receipts (70.1% on LawBench vs 45% prior SOTA, 14x on an AlphaFold-3 Triton kernel, #1 on MLE-Bench Hard). Reach for it when your problem IS a benchmark: a scoreable task the loop can grind against. NOT for tasks without a mechanical metric, and not a polished product — it's the official research implementation, so expect to build the task harness yourself and budget real GPU time for the weight-update path.