https://evoleinik.com/tags/evaluation/ Recent content in Evaluation on Eugene Oleinik https://evoleinik.com/og-image.png https://evoleinik.com/og-image.png Hugo -- 0.152.2 en-us Tue, 03 Nov 2026 00:00:00 +0000 https://evoleinik.com/posts/automatic-research-loops-honest-scorer/ Tue, 03 Nov 2026 00:00:00 +0000 https://evoleinik.com/posts/automatic-research-loops-honest-scorer/ Karpathy’s autoresearch and DeepMind’s AlphaEvolve improve things with one simple loop: propose a change, score it, keep it only if it wins. I ran the same loop three ways: on trading strategies with a backtest as the scorer, on a spec and a strategy with LLM judges as the scorer, and by hand through 56 LLM review panels. The backtest loop inflated its own results 3 to 4 times. The LLM-judged loops stalled and learned to flatter the judges. Here is what each failure teaches, and how to build a loop that works.