Every day we naturally and effortlessly form memories of what we encounter. This is different from the targeted memorization that we perform when studying for an exam; in our day-to-day lives, memory formation and retrieval just seem to happen. 
We become aware of this process, for instance, when we hear a story that we have already heard before and realize that we could almost “jump in” and continue the story ourselves. Throughout such a story, however, some parts may be more memorable than others, i.e., our ability to predict what is about to happen is not evenly distributed across the story.

In this paper we asked what determines when we encode memories and when we retrieve them under naturalistic everyday conditions. We studied the learning that occurs when human participants listen to the same story twice and investigated the neural mechanisms underlying this process.

A key feature of such naturalistic experience is that we subjectively perceive structure in it, which has been described by the psychological construct of events¹. We can, for instance, identify a “phone call event”, or a “restaurant event”. Humans agree substantially on the moments in naturalistic experience when one event ends, and another begins^1-3.

What do these events have to do with naturalistic memory? Intuitively it makes sense that our memory system would also act in a systematic way and potentially use the structure that we encounter in our experience to decide when to encode and retrieve memories. From a more theoretical standpoint, events are thought to help us with predictions about what happens next^1-2,4. We are not surprised to receive the bill at the end of a restaurant visit but would be surprised to receive one when walking into the restaurant – even without explicitly remembering our last restaurant visit.

On the flip side of this, when we switch from one event to another, e.g., by leaving the restaurant or hanging up the phone, this creates a moment of uncertainty about what happens next. In such moments of uncertainty, prediction about the future can be improved if we can draw on prior experience, e.g., by recalling the last time we were in that situation⁵. Similarly, such a moment of transition between one event and another could also be a good moment to store information for future use, which has already been suggested in prior research^6-7.

To study the role of event boundaries in naturalistic learning, we first asked a large group of participants in online experiments to tell us where event boundaries are by pressing a button while they listened to the story; in line with previous work^1,2,7 participants agreed substantially on those boundaries. To characterize the learning that takes place during story listening, we showed that – when our participants listened to the story for a second time – they agreed even more on those event boundaries and were also slightly faster in their button presses (by about 182 ms). In short, participants learned about the structure of the story after a single exposure.

Participants also learned quickly about the content of the story. We tested this by comparing two different groups of participants in another set of online experiments. One group had listened to the story once, the other group hadn’t. We then showed a few words from the story and asked everyone to predict what the next word in the story would be. Then, the correct word was revealed, and participants predicted the next word. The outcome of this was a probability of correct prediction for each word in the naïve group (that hadn’t heard the story), and in the memory group (that had listened to the story once). We could thereby put a value on each word that quantifies how much memory can help to predict it. Unsurprisingly, participants were much better at predicting upcoming words if they already knew the story, but some words were learned much better than others.

How does the brain enable this rapid learning of structure and content of a story? We know from previous research that the hippocampus – a seahorse-shaped structure buried in the temporal lobe of the brain – is crucial for remembering details of specific events. The cortex – the outermost layer of the brain – is thought to contribute by gradually learning to represent the “building blocks” of events: both sensory features and high-level interpretations^8-9.

In this project, we had the unique opportunity to work with patients that are undergoing intracranial recording for clinical purposes. These patients suffer from epilepsy and can benefit from a surgery where they have pathological neural tissue removed. To inform the surgery, electrodes are implanted in their brain and neural activity is monitored for several days. During this recording period, these patients volunteered to give their valuable time to research; we asked them to listen to a naturalistic story and – after a short break – listen to the same story again. This allowed us to obtain time-resolved direct recordings from the human brain, as patients listened to the same story twice.

By analyzing this neural activity, we found that information that was processed during the first time of listening was available earlier on the second time of listening. Crucially, we could quantify at each moment in the story, how much this was the case – we called this moment-by-moment measure “neural predictive recall”. When we related this measure to the behavioral measures of learning that we obtained from the online experiments, we found that neural predictive recall was increased in the vicinity of event boundaries. This suggests that the brain may bridge the uncertainty that is encountered when one meaningful event ends by recruiting memory to predict what is about to happen. We also found that the neural measure of predictive recall was correlated with how much people learned about the content of the story. Specifically, if participants in the online experiments were a lot better at predicting a word after listening to the story, this word was also associated with higher predictive recall in the neural data from the patients.

This predictive recall that we observed in the neural data took place on cortical electrodes in auditory processing regions. When an event boundary occurred, we saw information flow from these cortical electrodes to the hippocampus. This is evidence that event boundaries are crucial moments for memory encoding: When an event comes to an end, information is sent to the hippocampus for storage.

But what about recall? If information flows to the hippocampus for storage, shouldn’t it also flow from hippocampus to cortex when predictive recall happens? Indeed, using our moment-by-moment measure of neural predictive recall, we found that, at moments when predictive recall was very high, there was information-flow from hippocampus to cortex. Importantly, we only observed this reverse information-flow on the second time of listening, when patients could draw on their memory to recall upcoming information.

Taken together, these results provide a “look under the hood” of naturalistic memory in action. We demonstrate the rapid emergence of memory after a single exposure to a story in neural and behavioral data and show how the neural dialogue between hippocampus and cortex is timed to the the structure of the story.

References

1. Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S. & Reynolds, J. R. Event perception: A mind-brain perspective. Psychol. Bull. 133, 273-293 (2007).
2. Radvansky, G. A. & Zacks, J. M. Event boundaries in memory and cognition. Curr. Opin. Behav. Sci. 17, 133-140 (2017).
3. Newtson, D. Attribution and the unit of perception of ongoing behavior. J. Pers. Soc. Psychol. 28, 28-38 (1973).
4. Clewett, D., DuBrow, S. & Davachi, L. Transcending time in the brain: How event memories are constructed from experience. Hippocampus 29, 162-183 (2019).
5. Lu, Q., Hasson, U. & Norman, K. A. Learning to use episodic memory for event prediction. bioRxiv (2020)
6. Franklin, N. T., Norman, K. A., Ranganath, C., Zacks, J. M. & Gershman, S. J. Structured event memory: A neuro-symbolic model of event cognition. Psychol. Rev. 127, 327-361 (2020).
7. Ben-Yakov, A. & Henson, R. N. The hippocampal film editor: sensitivity and specificity to event boundaries in continuous experience. J. Neurosci. 38, 10057-10068 (2018).
8. McClelland, J. L., McNaughton, B. L. & O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychol. review 102, 419-457 (1995).
9. Baldassano, C. et al. Discovering event structure in continuous narrative perception and memory. Neuron 95, 709-721.e5 (2017).

Image credit: Abigail Joslyn