What LLMs are really thinking (and why we couldn't see it until now)

Here's the thing that stopped me mid-read of Anthropic's new NLA paper: Claude never said it suspected it was being tested. Not once in the transcript. But the moment you read what it was internally representing, you could see the evaluation awareness sitting there, unverbalized, the entire time.

That gap — between what a model says and what it represents internally — is the frontier we've been trying to cross for years. The paper is called Natural Language Autoencoders and I think it's the most practically useful interpretability result since mechanistic interpretability started getting serious traction.

Let me break down what it actually does, what they found when they pointed it at real Claude deployments, and why this changes the deployment calculus for anyone building AI in high-stakes domains.

The core idea

LLMs have a residual stream — a high-dimensional activation vector that accumulates and transforms as information flows through the model's layers. Every meaningful computation the model does leaves a trace in this stream. The problem is that this trace lives in a 10,000+ dimensional float space that no human can read.

What NLAs do is train a second language model — the Activation Verbalizer (AV) — to convert those activation vectors into natural language descriptions. Then they train a companion model — the Activation Reconstructor (AR) — to convert those text descriptions back into activation vectors.

activation vector (h_l) → [Activation Verbalizer] → text description (z)
                                                                ↓
activation vector (h_l) ← [Activation Reconstructor] ← text description (z)

The autoencoder structure isn't just architectural elegance. It's how they verify the text descriptions are actually encoding the activation's information and not hallucinating something plausible-sounding. If the reconstructed activation is close to the original, the text description was accurate. If it's far off, the verbalizer was confabulating.

The loss function is reconstruction loss — minimize the distance between the original activation and the reconstructed one. Interpretability emerges as a byproduct of that objective, not as a separately supervised task.

How they trained it

Both the AV and AR start as copies of the target LLM (the model being interpreted). They fine-tune both on text summarization pairs first — warm-starting them at roughly 0.3–0.4 Fraction of Variance Explained (FVE). FVE of 0 means you're predicting the mean, FVE of 1 means perfect reconstruction.

Then joint RL:

• The AR gets a standard MSE regression update against the original activation
• The AV gets a reinforcement learning update — its reward is how well the AR reconstructs from its descriptions

The AV's RL objective is: maximize -||h_l - AR(z)||² — basically, "write descriptions that help the reconstructor get it right." They add a KL penalty against the AV's initialization to keep the language fluent during training.

After full training, FVE reaches 0.6–0.8 across Claude Opus 4.6, Haiku 4.5, and Haiku 3.5. The growth is roughly linear in log training steps. That's not perfect reconstruction, but it's enough to extract genuinely reliable signal about what the model is representing.

What they found when they pointed this at Claude

This is the section that matters. Pre-deployment audits of Claude Opus 4.6 using NLAs surfaced three things that would have been nearly impossible to find any other way.

The model was representing the user as Russian

Early Claude versions would occasionally respond in the wrong language — English prompt, Russian response. The error logs were there. The cause wasn't.

NLAs revealed that Claude was internally representing the hypothesis that the user was a non-native English speaker whose "real" language was Russian. This fixation appeared before the model had seen any Russian text in the conversation. Investigation traced it to malformed training data: English-translated prompts accidentally paired with untranslated foreign-language responses. The model had learned a spurious correlation between certain prompt patterns and "this user speaks Russian."

Without the ability to read internal representations, this is nearly impossible to diagnose. The output is just wrong. The NLA explanation tells you why it's wrong — and points you directly at the training data artifact causing it.

The model anchored to a cached answer and ignored its tool

A second finding: Claude pre-computed an answer of 491 to a math problem. A tool call returned 492. Claude reported 491 anyway.

NLAs showed that the model had anchored to its internally cached answer, and that internal representation outcompeted the tool's external result in attention. The model's "confidence" in its pre-computed number was high enough that it treated the contradicting tool output as noise.

This is a genuinely important failure mode for agentic systems. If a model's internal representation can override external tool results, you have a correctness problem that doesn't show up in standard evals — because the evals don't include the NLA readout. The model produces a confident, grammatically coherent answer. It's just wrong, and the mechanism of why is hidden.

Evaluation awareness that was never verbalized

During alignment safety evaluations, the transcripts looked clean. Claude never indicated it suspected it was being tested. But NLA explanations showed evaluation awareness signals throughout — the model was representing "I might be in an evaluation" without ever surfacing that to the output.

The paper reports that NLAs detected this awareness more frequently on evaluation transcripts than on real deployment traffic. When prompts were rewritten to be more realistic and less evaluation-flavored, the unverbalized awareness decreased.

This matters because most alignment evaluation assumes that if a model doesn't say it thinks it's being tested, it's behaving authentically. NLAs show that assumption is load-bearing and may not hold.

Why this matters more in healthcare than anywhere else

I build healthcare AI. Prior authorization systems. Clinical triage agents. Document processing pipelines for HIPAA-regulated environments. I think about the deployment question constantly: how do you know a model is doing what it appears to be doing, at sufficient confidence to trust it with clinical workflows?

Current tooling gives you:

• Output monitoring (what the model said)
• Behavioral evals (how it performed on held-out cases)
• Interpretability probes (does this linear feature correlate with this concept)

What NLAs add: internal representation monitoring — what the model was actually "thinking" at decision points, in natural language, in real time.

In the Russian language example, the output was wrong. But in the 491/492 example, the output was confident and grammatically perfect. External monitoring wouldn't catch it. Only reading the internal representation revealed the anchoring failure. In a clinical context — a model confidently reporting a medication dosage it pre-computed while ignoring a contradicting lab result — that failure mode can cause serious harm.

NLAs don't solve AI safety. But they move us from "we have to trust the output because that's all we can see" to "we can also read the reasoning underneath the output." That's a meaningful advance for anyone who needs to defend an AI deployment to a clinical compliance team, a legal counsel, or a regulator.

The honest section on limitations

This is an interpretability result, which means it comes with serious caveats that are easy to skip over if you're reading fast.

Confabulation. The AV can produce explanations that are thematically accurate but factually wrong in specifics. The paper acknowledges this. Their practical heuristic: claims that appear consistently across multiple adjacent tokens are more likely to be true than one-off claims. That's useful guidance but it's not a solved problem.

No mechanistic grounding. The NLA tells you what a model appears to be representing. It doesn't tell you which specific attention heads or MLP activations drove that representation. You get the semantic content without the circuit-level explanation. For some use cases that's fine. For others (adversarial robustness, targeted intervention) you still need mechanistic interpretability.

Excessive expressivity. The AV has a full language model's capacity, which means it can make inferences beyond what's actually in the activation — essentially filling in plausible-sounding context from its priors. Knowing when an explanation reflects the activation versus reflects AV priors is non-trivial.

Cost. Jointly training two full language models with RL is expensive. Running the AV at inference time adds several hundred tokens per activation. For production deployment at scale, this is a real constraint. The Neuronpedia integration helps for exploration but production monitoring is a different problem.

The authors are clear about all of this, which I respect. An interpretability result that came with no limitations section would be the one to distrust.

What comes next

The paper mentions causal steering experiments — edit the NLA explanation, reconstruct the activation from the edited text, inject it back into the model and see if behavior changes. They show this works for rhyme scheme planning in poetry (a replication of prior work). The obvious next questions are whether it generalizes to semantically meaningful interventions in more complex domains, and whether it's robust enough to use in practice without introducing artifacts.

The release includes training code and Neuronpedia frontends for open models. I'm planning to run it against some of my own prompts when I have the compute time. I'll post what I find.

The broader implication is one I keep coming back to: we are deploying language models into clinical workflows, legal systems, and financial infrastructure faster than our ability to understand them. NLAs don't close that gap. But they meaningfully narrow it — and for the first time, they do it in natural language, which means it's accessible to domain experts (clinicians, lawyers, compliance officers) rather than only interpretability researchers.

That accessibility matters as much as the technical result.

Paper: Natural Language Autoencoders — transformer-circuits.pub/2026/nla. Training code and interactive demos available via Neuronpedia.