# Why Agents Should Imitate the Update, Not the Answer *On a recent Google paper, the perception-action loop, and what it would take to build agents that actually learn from evidence in production.* --- ## A small mistake worth fixing There is a way of talking about Bayesian reasoning that has crept into AI commentary, and it gets the most important thing exactly backwards. The story goes: a Bayesian agent has priors, and learning means updating them with evidence to get a posterior. So far so good. But then the gloss becomes "Bayesian agents learn from their priors," as if the prior were the substance of the thing. The prior is not the substance. The prior is the starting condition. The substance — the part that makes Bayes interesting and that almost nothing in the LLM stack actually does well — is **the update**. A Bayesian agent has a guess, sees evidence, revises the guess, acts on the revised guess, sees more evidence, revises again. The whole machinery is in the revision. Take that away and you are left with a confident system that never changes its mind. I want to write about a paper from Google Research that makes this distinction unusually concrete, and then argue that the same distinction is the one that separates agents that actually accumulate competence in production from agents that look impressive in a demo and then go quietly mad on contact with reality. --- ## What the Google paper actually showed In March, Google Research published "[Teaching LLMs to reason like Bayesians](https://research.google/blog/teaching-llms-to-reason-like-bayesians/)." The setup is small and clean. They built a flight-recommendation task: a simulated user has hidden preferences over departure time, duration, stops, and cost. The assistant sees three flight options per round, has to pick the one the user would pick, and gets feedback at the end of each round. Five rounds, then a new user. The hard part is that the assistant cannot see the preferences directly. It has to infer them from a sparse stream of choices. They built two reference systems against which to measure LLMs: - A **Bayesian Assistant**, which maintains an explicit probability distribution over user preferences and uses Bayes' rule to update after each observed choice. This is the optimal strategy for the task. - An **Oracle Assistant**, which has been told the user's preferences directly and always picks the user's choice. Off-the-shelf LLMs were tested. They did poorly. More tellingly, they did not get *better* over the five rounds the way the Bayesian Assistant did. They tended to plateau after the first interaction — a model that, having formed an initial guess, was largely incapable of revising it as more evidence arrived. The Bayesian Assistant climbed to 81% accuracy by round five. The LLMs did not climb. Then came the interesting part. The researchers fine-tuned the LLMs on demonstrations of two different teachers: - **Oracle teaching**: train on transcripts of the Oracle Assistant — the one that always picks correctly because it cheats. - **Bayesian teaching**: train on transcripts of the Bayesian Assistant — the one that does *not* always pick correctly, especially in early rounds when uncertainty is high. You might expect Oracle teaching to win. You're showing the model the right answer every time. The Bayesian teacher, by contrast, is visibly hedging, sometimes flat-out wrong in early rounds, and only gets sharp as evidence accumulates. Why would training on a teacher who guesses wrong outperform training on a teacher who is always right? But that is exactly what happened. Bayesian teaching beat Oracle teaching. And not narrowly — the Bayesian-fine-tuned models also generalized better, transferring their probabilistic logic to a web shopping domain that wasn't in the training set at all. The result has a clean interpretation, and it's the one I want to sit with. Oracle teaching trains the model to mimic *answers*. Bayesian teaching trains the model to mimic *the posture of belief revision*. The Oracle's transcripts contain no information about what reasoning under uncertainty looks like, because the Oracle has no uncertainty. The Bayesian's transcripts contain exactly that information, encoded in the way the predictions tighten as evidence arrives. The student that learns to imitate the update generalizes. The student that learns to imitate the answer does not. This is, I think, a genuinely important finding, and it is going to be misread if we are not careful. It is *not* a paper about pre-training versus post-training; both teaching regimes are post-training. It is a paper about *what kind of demonstration* is worth training on. Demonstrations of right answers are cheap and seductive. Demonstrations of principled belief revision are rarer and more valuable, because they carry the structure that lets the student handle situations the teacher never encountered. --- ## Perception, action, and the loop in the middle Step back from the paper for a moment. Why would imitating the update generalize at all? Why is the act of revising belief on evidence so apparently fundamental that learning to imitate it transfers across domains? The answer that I find most defensible, without overreaching, comes from a tradition in cognitive science that runs from Hermann von Helmholtz through modern predictive processing: the idea that **perception and action are not two separate things connected by a controller, but two faces of the same belief-updating loop**. The agent maintains a model of the world. It generates predictions. It samples evidence — through its senses, through tool calls, through whatever instruments it has. It updates the model on the difference between what it predicted and what it observed. And then it acts, where action is itself a way of either gathering more evidence or moving the world closer to its predictions. Then the loop repeats. I want to be careful here. There are strong forms of this thesis — Karl Friston's free energy principle in its more sweeping presentations — that are contested, and I don't want to plant a flag on them. The weaker, almost uncontroversial claim is enough for what I want to argue: an agent that operates in an environment it does not fully control is constantly cycling through prediction, observation, update, and action. Anything that interrupts that cycle, or replaces real updates with confident static answers, will degrade in proportion to how unpredictable its environment really is. This is the lens through which the Google paper becomes more than a curiosity. The reason an off-the-shelf LLM plateaus after one round is that its training did not teach it to maintain the loop. It learned the shape of an answer, not the shape of an inference process. Bayesian teaching repairs the loop, at training time, by handing the model a thousand worked examples of what it looks like to hold uncertainty and let it shrink in response to evidence. But — and this is where I want the argument to turn — Bayesian teaching at training time is only half of what's needed. Maybe less than half. --- ## The half the paper doesn't address Imagine the best possible outcome of Bayesian teaching: every LLM in production has been fine-tuned to reason like the Bayesian Assistant. The models hold uncertainty well. They update on evidence. They generalize across domains. Now put one of those models in front of a real production system. A particular customer's billing API. A specific Salesforce instance with seven years of org-specific customizations. A vendor's documentation that lies in three places and is silent in five more. The model encounters this system fresh. It calls an endpoint. The response has a field that wasn't documented. It calls another endpoint. The pagination is off. It tries the combination the docs said would work and it doesn't. The model is doing exactly what Bayesian teaching trained it to do. It is updating on evidence. It is holding the posterior open. It is, in the small, behaving like a good Bayesian agent. And then the conversation ends. A new agent invocation begins. The new invocation has no idea what the previous one learned. The hard-won posterior — the field that wasn't documented, the off-by-one pagination, the combination that breaks — is gone. The next agent starts from the same prior. It rediscovers the same things. The system gets billed for the rediscovery. This is the half the Google paper doesn't address, because it isn't trying to. The paper is about how to teach a model to update *within* a single trajectory, given the context window it has. What it does not solve, and was never going to solve, is how priors persist *across* trajectories. How yesterday's posterior becomes tomorrow's prior. How an agent — or a team of agents working against the same systems — accumulates competence over time rather than starting from zero every morning. If you take the perception-action loop seriously, this is not a small omission. It is the entire game. A real Bayesian agent does not just update within an episode. It carries the posterior forward and uses it as the prior for the next episode. That is what makes the loop a loop. Otherwise it's just a single line of inference that ends when the context window does. --- ## Where the priors actually live This is the problem we have spent the last year of [Rote](https://getrote.dev) trying to make tractable. The dominant pattern in agent infrastructure today is to inject "skills" or "memories" into the agent's working context — usually as instructional text. A markdown file that says: when the user asks about invoices, do these steps. The agent reads the instruction, follows it, fails in some case the instruction didn't anticipate, and either asks for help or hallucinates a solution. The instruction is doing the same thing Oracle teaching does in the Google paper: it carries an answer, not an update. I wrote about why this is the wrong shape for skill in [a previous post on Polanyi and tacit knowledge](https://modiqo.ai/blog/skills-that-cannot-be-written-down). The short version: most of what makes a senior practitioner effective in a system is knowledge that resists propositional articulation. The instructional skill cannot carry it. What can carry it is a record of practice — what was done, what happened, what worked, repeated across cases until structural regularities surface. The Bayesian frame and the Polanyi frame look different on the surface. Bayes is the canonical formal account of belief; Polanyi is the argument that the most important knowledge isn't formal. But once you ask where priors *live* in a real production agent, the two collapse onto each other. The prior that matters — the prior an agent should be carrying into its next encounter with a system — is not something that gets written down as a propositional rule. It is something that accumulates from doing, sharpens over many runs, and eventually becomes reliable enough to be invoked rather than rediscovered. In Rote, that's what the architecture is for. The **adapter** is the surface through which the agent probes a real system. Every call leaves a record. That record is not a log line for a dashboard; it is evidence in the Bayesian sense. It is the data over which a posterior should form. The **canvas** is where the records accumulate. Across one run, across many runs, across many agents working against the same system. The canvas is the substrate on which something resembling a posterior can take shape — not as a literal probability distribution over endpoints, but as a structured, queryable record of what has actually been observed across attempts. It is the thing that lets the next agent's prior be informed by the last agent's evidence. The **flow** is what happens when the posterior has hardened. When the same task has been done enough times across the canvas that the structural regularity of what worked is stable, that regularity gets crystallized into a deterministic, replayable artifact. A flow is not a probability distribution and I do not want to overclaim it as one. A flow is what a hardened posterior looks like once it is reliable enough to run as code: the part of the inference that has earned the right to be invoked rather than redone. This is the inversion that matters. The standard model of agent skill is to hand the agent an instruction up front, like an Oracle teacher. Rote's model is to let the agent accumulate evidence on the canvas, let beliefs sharpen across runs, and crystallize the parts that have stabilized. It is, structurally, the runtime analog of what Bayesian teaching does at training time — except instead of distilling a Bayesian teacher into a model's weights, it lets the agent's own practice, against the systems it actually has to operate in, become the source of the priors it brings to its next encounter. Bayesian teaching closes the loop within an episode by training the model to update on evidence. Rote closes the loop *across* episodes by giving the evidence somewhere to accumulate and the posterior somewhere to live. --- ## What this lets us stop pretending If you accept the framing, a few things in the current agent stack start to look obviously wrong. Stuffing skills into the system prompt is Oracle teaching at inference time. It's handing the agent a confident answer to imitate, with no record of the uncertainty the answer was supposed to resolve. It works for the easy cases and fails silently on the hard ones, because there is no machinery for noticing that the prescribed steps do not match what the system actually does. Treating each agent invocation as stateless is even worse. It is the literal opposite of what a Bayesian agent looks like. A Bayesian agent that throws away its posterior at the end of every episode is not a Bayesian agent. It is a system that performs the gestures of inference and then erases its own learning. That is what a great deal of production agent traffic looks like today, and it is why the same operations get rediscovered on Tuesday that got discovered on Monday, at the same token cost. The fix is not a smarter prompt and not a better fine-tune, though both help. The fix is structural. Agents need a place where evidence accumulates, where posteriors form, where the parts that have stabilized can be crystallized and handed forward. Without that, you can train all the Bayesian teachers you want and the agents will still arrive at every new system as if it were the first time, every time. What Google's paper showed, in a small task with a clean setup, is that imitating the update generalizes and imitating the answer does not. The implication for the rest of the stack is bigger than the paper itself: an agent architecture that doesn't preserve the update across runs is, at every level above the model weights, doing Oracle teaching all over again. It is handing the next invocation a confident answer instead of a hardened posterior, and then expressing surprise when nothing compounds. The question Rote is trying to answer is what the alternative looks like in production. Not a Bayesian model in a vacuum, but a Bayesian *substrate* — adapters that probe, canvases that accumulate, flows that crystallize. A place where the loop actually closes, and where the prior an agent carries into tomorrow's task is not a developer's best guess but yesterday's posterior, paid for once and then reused. That is what it would mean to take the perception-action loop seriously at the level of infrastructure rather than just the level of weights. And it is, as best I can tell, the only path by which agents stop being expensive demonstrations of inference and start being systems that actually accumulate competence. --- *The Google Research paper, "[Teaching LLMs to reason like Bayesians](https://research.google/blog/teaching-llms-to-reason-like-bayesians/)," and the underlying [Nature Communications paper](https://www.nature.com/articles/s41467-025-67998-6), are worth reading in full. For the Polanyi argument that complements this one, see "[Skills that cannot be written down](https://modiqo.ai/blog/skills-that-cannot-be-written-down)."*