Source: How AI assistance impacts the formation of coding skills — Anthropic Research, January 2025

Today I learned about Anthropic’s randomized controlled trial examining how AI coding assistance affects skill development. The findings are sobering.

The Key Finding

Developers using AI assistance scored 17% lower on comprehension quizzes compared to those who coded by hand—the equivalent of nearly two letter grades. The AI group averaged 50% on the quiz versus 67% for the hand-coding group.

The largest gap was on debugging questions, suggesting that understanding when code is incorrect and why it fails is particularly impaired when relying on AI.

The Nuance

Not all AI usage is equal. Participants who retained stronger mastery used AI differently:

• Asked follow-up questions
• Requested explanations of the code
• Posed conceptual questions while coding independently

They used AI to build comprehension, not just produce code.

The Tension

This creates a workplace dilemma:

• AI can reduce task completion time by up to 80% for existing skills
• But AI may hinder acquisition of new skills
• Junior developers under time pressure may sacrifice learning for speed—then lack the skills to debug issues later

The Takeaway

“Cognitive effort—and even getting painfully stuck—is likely important for fostering mastery.”

When learning something new, consider using AI’s “learning modes” (like Claude Code Learning mode or ChatGPT Study Mode) rather than just copy-pasting solutions. The struggle is part of the skill-building process.

The full paper is available on arXiv.