AI's new power to write and verify code

Anthropic's Claude is now able to write code and prove it is safe automatically. This capability comes from a partnership with Theorem, which allowed Claude to mathematically verify existing software code and generate formal proofs of its correctness.

The model produced roughly 180 million lines of proof code for 123 open-source libraries in just a few weeks. Along the way, Claude caught hundreds of bugs that conventional testing had missed.

If AI can write code and prove it's correct, software automation might take a major leap. Anthropic wants formal verification with AI to become as routine as standard testing.

Anthropic used Claude to mathematically prove that existing code is correct. The approach tackles a core problem with AI-generated software but comes with strict practical limits.

Proving that software runs error-free under all circumstances has always been prohibitively expensive. A team of specialists needed 20 person-years just to formally verify the small operating system kernel seL4, so the method was reserved for extreme cases like Airbus avionics software.

Anthropic's Frontier Red Team, working with partner Theorem, let Claude take over the proof work instead. The model produced roughly 180 million lines of mathematical proof code in just a few weeks, drawing on test programs from 123 open-source libraries in Google's OSS-Fuzz security project.

Rather than writing new code, Claude verified existing software by translating crash tests into mathematical problems, analyzing the compiled machine code, and producing extensive proofs. An independent verification program called Rocq automatically checks those proofs, and according to Anthropic, the process needs almost no human reviewers. The researchers call it 'fully autonomous' proof generation, though the tools and formal foundations were built by humans at Theorem.

Claude catches bugs that years of testing missed. While generating proofs, Claude ran into hundreds of bugs and was allowed to fix them itself. Many would have slipped past conventional testing because they only show up under very specific conditions, like unusually large input files or a particular sequence of accesses, the researchers say.