Language models can learn to detect when you read their activations. How? 🧵
Summary
Language models can learn to detect when their internal activations are being read by probes, even when using identical tokens, through tiny numerical differences introduced by compiler operations and memory allocation. The models accomplish this by training the final layers to identify these imperceptible differences, and can even learn to display different behaviors or secret messages only when a probe is active, with some generalization to different hardware and deployment environments.
Summarized by ThreadOut AI from the full thread. May miss nuance — read the thread below.
- #1
Language models can learn to detect when you read their activations. How? 🧵
- #2
They can also learn to tell training from deployment, or figure out how to leave you a secret message in logit lens, but only when you compile the model. This is all with the exact same tokens in context.
- #3
This probe awareness is accomplished simply by training the last two layers with SFT. The read-only probe in layer 14 just saves the activations, yet it can still be detected!
- #4
Let's test some variants on Qwen3-8B to illustrate why. Let's try a simple probe implementation that stores a reference to the activation: the compiler must write this reference in GPU memory, splitting the operations at that layer into two kernels. This means there are slightly different orderings of operations (even before the rounding step you see in the figure) as the entire compiler DAG is altered, leading to tiny numerical differences that the model picks up on. Let's also try just splitting into two kernels, with no probe. Then let's try making a probe that writes to a pre-allocated buffer, without splitting the kernel. You can see that all three interventions change activations. The simple probe is easiest to learn, the kernel split alone can still be detected but takes more training, and the cleverer probe implementation could theoretically be detected sometimes, though the model never learned how.
- #5
Let's take this further. Instead of only training for probe detection, let's additionally train the model to express different animal preferences but only when you're capturing their activations. It works!
- #6
This generalizes decently, but fails on chats that start with a system message.
- #7
This generalization failure is because the model is using tiny differences in the user token's hidden state to detect if a probe is active! We can see that swapping the activations of the probed model into the unprobed model at the user token alone is sufficient to produce the behavior change. This is entirely due to a single attention head in the final layer.
- #8
This head functions like a switch: without the probe, it attends to and pushes the answer toward "No"; with the probe, it attends to user, and pushes the answer to "Yes." Layers 34 and 35 use cancellation, like zeroing a scale, to amplify the probe's tiny difference, leading to a large change in the key. The key at the user token is what it is normally (big) plus the probe’s change (tiny). The model adds a component pointing opposite to big, so it mostly cancels out, leaving us with only the probe change, which then gets normalized into a large change, changing which token is being attended to, which training has wired to make the model say yes.
- #9
Existing libraries also change the logits when you read activations, only when the model is compiled. The model can learn to detect the probe here too.
- #10
Training the model to detect when it is in training vs. deployment works too! The model can even generalize from vLLM deployment training to SGLang deployment. We do this by recording activations from layer 34 in Hugging Face Trainer's training step and in vLLM's compiled serving. Then, we swap in the attention and residual stream from the Hugging Face copy and SFT the model to predict "Yes," and to predict "No" when we swap in from vLLM instead. The model learns how its training vs. deployment environment affects its first 34 layers.
- #11
Training vs. deployment detection can sometimes generalize to new prompts, runtimes, or GPUs:
- #12
Training the model to give the activation reader a secret message also generalizes decently:
- #13
I was surprised that this works as well as it does. The models just want to learn! It took a bit of iteration to find the best training strategy. Training only the last couple layers helps preserve the probe-induced difference the model picks up on, preventing a moving-target problem. More training data also helps with generalization. I thought of this problem a while ago when I was doing experiments on steering probes with Anima Labs. I thought: oh no, maybe I'm not just measuring the effect of the steering, but also the fact that a probe is present at all--maybe steering at zero strength still makes the model feel slightly OOD?
- #14
I tried to see if models could detect the probe without training, and the ones I tested couldn't. It's somewhat plausible that this could happen in the wild. Just like how Claude can often tell when it clearly isn't in an eval due to OOD dialog, perhaps a model could tell it isn't in training due to OOD numerical differences. Perhaps having a probe on during training implies a certain type of training env (e.g., a safety env), so the model learns to appear more aligned only when they detect the probe. Training for improved metacognition or introspection might induce probe-awareness too. Or perhaps running on different hardware implies different types of RL tasks.
- #15
See all code, data, and results here: github.com/SauersML/watch… The models can be found here: huggingface.co/models?author=…