To Make Sense of the Present, Brains May Predict the Future

By Jordana Cepelewicz

But in spite of the clear role that generative models and expectations play in brain function, scientists have yet to pinpoint exactly how that’s implemented at the level of neural circuits. “The Bayesian brain story is relatively agnostic about what the underlying mechanisms are,” said Mark Sprevak, a professor of philosophy of mind at the University of Edinburgh in Scotland.

Enter predictive coding theory, which offers specific formulations of how brains can be Bayesian. Predictive coding gets its name from a technique for transmitting telecommunications signals more efficiently: Because video files contain a lot of redundancy from one frame to the next, it’s inefficient to encode every pixel in every image when compressing the data. Instead, it makes more sense to encode the differences between adjacent frames and then work backward to interpret the entire video.

In 1982, scientists found that this idea has a neat application in neuroscience — because it appears to explain how neurons in the retina encode information about a visual stimulus and transmit it along the optic nerve. It’s also been cemented as a principle of how the brain’s reward system functions: Dopamine neurons encode the magnitude of the mismatch between an expected reward and the actual reward that’s received. These prediction errors, researchers say, help animals update their future expectations and drive decision-making.

But despite these examples, scientists mostly saw predictive coding as a process specific to certain networks. Functional magnetic resonance imaging tests and other types of experiments have begun to change that.

A Universal Framework

Part of what makes the predictive coding hypothesis so compelling is its incredible explanatory power. “What I find convincing is how so many things all get accounted for under this story,” said Andy Clark, a professor of logic and metaphysics at the University of Edinburgh and an expert on the theory.

First, it unifies perception and motor control under a single computational process. The two are essentially opposite sides of the same coin: In each case, the brain minimizes prediction errors, but in different ways. With perception, it’s the internal model that gets adjusted; with motor control, it’s the actual environment. (For the latter, imagine that you want to raise your hand. If your hand is not already raised, that discrepancy generates a large prediction error — which can be minimized if you simply move your hand.)

Experiments in perception and motor control have so far provided the strongest evidence for predictive coding theory. In a paper published last month in the Journal of Neuroscience, for example, experimenters had subjects read the word “kick” on a screen, then had them listen to a distorted recording of the word “pick” that sounded like a loud whisper. Many heard “kick” instead, and fMRI scans revealed that the brain represented the initial “k” or “p” sound most strongly — the sound that correlated to a prediction error. If the brain were simply representing its perceptual experience, the strongest signal should have corresponded to “ick” instead (because that was represented both on screen and in the audio).

But efforts are also ongoing to widen predictive coding’s relevance beyond perception and motion — to establish it as the common currency of everything going on in the brain. “It’s like having building blocks with which different strategies can be built,” Clark said. Different brain regions simply trade in different kinds of prediction.

Friston, among others, claims this applies to higher cognitive processes including attention and decision-making. Recent computational work on the prefrontal cortex has implicated predictive coding in working memory and goal-directed behaviors. Some researchers theorize that emotions and moods can be formulated in predictive coding terms: Emotions could be states the brain represents to minimize prediction error about internal signals such as body temperature, heart rate or blood pressure. If the brain recognizes that it’s agitated, for instance, then it knows all those factors are going up. Perhaps that’s also how the concept of selfhood can emerge.

Most of the work being done in this vein focuses on how predictive coding might explain neuropsychiatric and developmental disorders. “The idea,” Friston said, “is that if the brain is an inference machine, an organ of statistics, then when it goes wrong, it’ll make the same sorts of mistakes a statistician will make.” That is, it will make the wrong inferences by placing too much or too little emphasis on either predictions or prediction errors.

Aspects of autism, for instance, might be characterized by an inability to ignore prediction errors relating to sensory signals at the lowest levels of the brain’s processing hierarchy. That could lead to a preoccupation with sensations, a need for repetition and predictability, sensitivity to certain illusions, and other effects. The reverse might be true in conditions that are associated with hallucinations, like schizophrenia: The brain may pay too much attention to its own predictions about what is going on and not enough to sensory information that contradicts those predictions. (Experts are quick to caution, however, that autism and schizophrenia are much too complicated to be reduced to a single explanation or mechanism.)

“The most profound part of it is that it shows us how vulnerable our mental function is,” said Philip Corlett, a clinical neuroscientist at the Yale School of Medicine. Experiments in Corlett’s lab set up new “beliefs” in healthy subjects that encourage them to hallucinate stimuli they previously experienced. (For instance, in one experiment, the scientists conditioned participants to associate a tone with a visual pattern. The subjects continued to hear the tone when they saw the pattern, even when there was no sound at all.) The researchers are trying to unravel how those beliefs translate into perception. Through these studies, “we’ve got evidence suggesting that perception and cognition are not that separate,” Corlett said. “New beliefs can be taught and can change what you perceive.”

But that evidence hasn’t come close to offering proof — until now.

Zooming in for a Better Look

“Experimental work often shows a particular result is compatible with predictive processing, but not that it’s the best explanation of that result,” Sprevak said. The theory is widely accepted in the cognitive sciences, but “in the field of systems neuroscience, it’s still a bit of an underdog,” said Georg Keller, a neuroscientist at the Friedrich Miescher Institute for Biomedical Research in Switzerland. His lab is trying to change this with harder evidence.

In a study published last year in Neuron, Keller and his colleagues observed the emergence of neurons in the visual system of mice that became predictive over time. It began with an accident, when they set out to train the mice on a video game, only to find that the virtual world had gotten its directions mixed up. Ordinarily — and up until the time of the experiment — the mice saw their field of vision move to the right whenever they turned to the left, and vice versa. But someone had unintentionally flipped the virtual world the researchers used in the study, inverting left and right so that turning leftward meant the mice also experienced vision leftward. The researchers realized that they could capitalize on the accident. They monitored the brain signals that represented this visual flow and found that the signals changed slowly as the mice learned the rules of the inverted environment. “The signals looked like predictions of visual flow to the left,” Keller said.

If the signals had simply been sensory representations of the mouse’s visual experience, they would have flipped immediately in the virtual world. If they had been motor signals, they wouldn’t have flipped at all. Instead, “it is about identifying prediction,” Keller said. “The prediction of visual flow, given movement.”

“The work provides a kind of evidence that didn’t exist before,” Clark said. “A very local, cell-by-cell, layer-by-layer demonstration that the best-fit model for what’s going on is predictive coding.”

Similar findings in the parts of the brain that macaques use to process faces were reported around the same time. Previous work had already shown that neurons at lower levels in the network code for orientation-based aspects of a face — by firing at, say, any face in profile. At higher levels, neurons represent the face more abstractly, by paying attention to its identity rather than its position. In the macaque study, the researchers trained monkeys on pairs of faces in which one face, appearing first, always predicted something about the second one. Later, the experimenters interfered with those expectations in specific ways, by showing the same face from a different angle, or an entirely different face. They found prediction errors in lower-level areas of the face processing network, but these errors were associated not with predictions about orientation but with predictions about identity. That is, the errors stemmed from what was going on at higher levels of the system — suggesting that lower levels construct the error signal by comparing incoming perceptions with predictions descending from higher levels.

“It was exciting to find prediction errors, and to find the specific content of predictions, in that system,” said the paper’s lead author, Caspar Schwiedrzik, a neuroscientist at the European Neuroscience Institute Göttingen in Germany.

According to Lucia Melloni, a researcher at the Max Planck Institute for Empirical Aesthetics in Frankfurt, Germany, her group is starting to see results compatible with an explanation of prediction error in neuronal data currently being collected from humans.

A Race to Find More Predictive Machines

Not everyone agrees that the case for predictive coding in the brain is strengthening. Some scientists accept that the theory can explain certain aspects of cognition but reject the idea that it could explain everything. Others don’t concede even that much. To David Heeger, a professor of psychology at New York University, it’s important to make a distinction between “predictive coding,” which he says is about transmitting information efficiently, and “predictive processing,” which he defines as prediction-making over time. “There’s a lot of confusion in the literature because these things have been assumed to all be part of the same soup,” he said. “And that’s not necessarily the case, nor is it necessarily the best way to go forward in studying it.” Other types of Bayesian models, for instance, might provide a more accurate description of brain function under certain circumstances.

What many experts in the field do agree on, however, is that this research has the potential for exciting applications in machine learning. At present, the vast majority of artificial intelligence research does not involve predictive coding, instead focusing on other kinds of algorithms.

But formulating predictive coding architecture in a deep-learning context could bring machines closer to intelligence, Friston argues.

DeepMind’s GQN serves as a good example of that potential. And last year, researchers at the University of Sussex even used virtual reality and artificial intelligence technologies that included predictive coding features to create what they called the “Hallucination Machine,” a tool that was able to mimic the altered hallucinatory states typically caused by psychedelic drugs.

Machine learning advances could be used to provide new insights into what’s happening in the brain by comparing how well predictive coding models perform against other techniques. At the very least, introducing predictive coding into artificial systems could significantly improve the intelligence of those machines.

But before that can happen, much work lies ahead. Scientists need to continue the kind of research being done by Keller, Schwiedrzik and others to determine just where, for instance, the brain’s internal representations reside. And it remains to be seen whether similar experiments can substantiate claims for predictive coding in higher cognitive processes.

Predictive coding “is as important to neuroscience as evolution is to biology,” said Lars Muckli, a neurophysiologist at the University of Glasgow who has done extensive work on the theory. But for now, Sprevak noted, “the jury is still out.”