Shrey Shah

We write skills for our coding agents, then we forget about them. Six months later the skill is stale and the agent makes the same mistakes it made in March. Karpathy calls this the agent's amnesia: it doesn't get smarter by default. So don't wait for it to.
I built a loop that trains the skill file itself. The agent runs a held-out task suite and gets scored, with deterministic checks where possible and a pinned LLM judge where not, with the accept bar set above the judge's own noise. A second model reads the failures and proposes one small, bounded edit to the skill. Then a gate decides. Score went up: commit. It didn't: git reset --hard, and the bad edit never happened. Rejected edits go back to the proposer so it stops repeating its own mistakes.
I'll show you a real overnight run, including the ugly parts: one real gain, two edits that tanked the score and got thrown away, and a loop that said 'no progress' and stopped instead of faking a win. I'll also cover the two tests that tell you whether it's learning or just chasing noise, and how to mine your own agent session history for new training tasks.
What you ship is still one skill file. No added cost at inference. The agent just follows better instructions than it had yesterday.