Automatic Research Loops Need an Honest Scorer
Karpathy’s autoresearch and DeepMind’s AlphaEvolve improve things with one simple loop: propose a change, score it, keep it only if it wins. I ran the same loop three ways: on trading strategies with a backtest as the scorer, on a spec and a strategy with LLM judges as the scorer, and by hand through 56 LLM review panels. The backtest loop inflated its own results 3 to 4 times. The LLM-judged loops stalled and learned to flatter the judges. Here is what each failure teaches, and how to build a loop that works.