In early 2019, we published a paper in Nature Communications revealing that drawings can be a meaningful and objective method to measure the information in a memory. Participants studied photographs of real-world scenes and after a delay were surprised with a pen and paper and asked to draw those images from memory. We uploaded these 2,682 drawings to an online experiment platform, and asked 8,596 people to score these drawings for all sorts of metrics: what objects they contained, where these objects were located, and what errors were made. Importantly, separate participants also drew images while perceiving them, and from their schemas of various scene categories--creating estimates of upper and lower bounds with which to compare memory drawing performance. We observed a level of detail in the memory drawings that far outpaced the expectations set by prior work utilizing verbal recall or visual recognition tasks--participants recalled on average 150 objects in pixel-precise locations with few false memories for objects. This method proved to be a promising tool for capturing the visual information in a mental representation or memory.

Since the completion of this study, we have successfully applied this method to several key questions in the domain of visual memory. By analyzing these memory drawings made of a diverse, naturalistic set of scene images, we could revisit classic concepts about scene memory, such as boundary extension--a tendency for people to remember a scene as having information beyond its boundaries, ascribed to an automatic extrapolation of information. After combining evidence from the drawings of our 2019 paper (downloadable here) and several new online experiments, we found striking evidence against boundary extension as a unidirectional phenomenon (Bainbridge & Baker, 2020a); boundary contraction (a tendency to zoom into the scene and crop out information) was equally likely (Fig. 1).

Importantly, we found that the direction of the effect depended highly on several visual properties of the image; central object photos like those classically studied for boundary extension displayed extension, but more scene-like images with multiple objects across a space elicited contraction. This has led to a debate in the field on what may account for these previously observed effects, and whether it is truly a mnemonic phenomenon, or something reflecting perception or attention processes (Intraub, 2020; Bainbridge & Baker, 2020b). This has also opened up conversation about open data and replication, and we make all our data available online.

Highly quantified drawings can also serve as an illuminating technique for understanding unique populations. We recently published a study using drawing to investigate the visual memories of individuals with aphantasia - a complete lack of voluntary visual imagery in spite of intact semantic memory and recognition. Until recently, aphantasia had been relatively under-characterized, with only seven published studies, and some researchers skeptical of whether it might be a largely psychogenic condition. We conducted the first large-scale and open data study of aphantasia, comparing memory drawings between aphantasic individuals to those with typical imagery. We were surprised to observe a specific deficit with object memory, but preserved spatial memory and incredibly low false memories (Bainbridge, Pounder, Eardley, & Baker, 2020). We also found no differences between participant groups when they were copying an image during perception, suggesting that these differences are not merely due to drawing ability, drawing strategies, or effort (Fig 2). This study has opened up many new questions about the nature, representations, and processes of visual imagery, as well as provided important scientific validation of the experience of aphantasia.

Finally, we think these drawing methods can reveal impactful insights into the mechanisms of memory more broadly. We have a recent pre-print showing how drawings can uncover a complex interplay of memory processes as they relate to scene semantics (Bainbridge, Kwok, & Baker, 2020). More specifically, when an object was inconsistent with the semantics of its encompassing scene, there was greater memory for the gist of the scene, but diminished memory for the specific objects within that scene. Given the high level of detail we have observed in drawings across these various studies, we have also been interested in directly comparing the neural patterns representing visual content of a memory during the encoding of an image and its recall. In a ultra-high-field 7T functional magnetic resonance imaging (fMRI) study, we discovered that patterns during recall showed different representational structure, as well as a spatial distinction from patterns during encoding (Fig. 3; Bainbridge, Hall, & Baker, 2020). Specifically, recall traces contained less abstract mid-level information about a stimulus (e.g., whether a scene is natural or manmade), and their representations extended anterior to perceptual representations of the same item. These findings serve as evidence against the idea that recall occurs as a reinstatement of the exact same encoding traces in the brain. Instead, these results add to growing theories on the transformations that perceptual representations undergo in the brain when they become memories (Favila, Lee, & Kuhl, 2020).

We have been very fortunate with several new directions to go from our original 2019 Nature Communications paper. In our plans for the near future, we will be constructing a tutorial that will allow other researchers to easily code and quantify their own drawing experiments to study a wide range of questions. We envision these methods could be applied to topics even beyond memory, such as attention, decision making, emotional processing, and social interaction.

References

1. Bainbridge, W.A., & Baker, C.I. (2020a). Boundaries extend and contract in scene memory depending on image properties. Current Biology, 30, 537-543.
2. Bainbridge, W.A., & Baker, C.I. (2020b). Reply to Intraub. Current Biology, 30, R1465-1466.
3. Bainbridge, W.A., Hall, E.H., & Baker, C.I. (2020). Distinct representational structure and localizataion for visual encoding and recall during visual imagery. Cerebral Cortex, bhaa329.
4. Bainbridge, W.A.*, Kwok, W.Y.*, & Baker, C.I. (2020). Disrupted object-scene semantics boost scene recall but diminish object recall in drawings from memory. bioRxiv, https://doi.org/10.1101/2020.05.12.090910.
5. Bainbridge, W.A., Pounder, Z., Eardley, A.F., & Baker, C.I. (2020). Quantifying aphantasia through drawing: Those without visual imagery show deficits in object but not spatial memory. Cortex, 135, 159-172.
6. Favila, S.E., Lee, H., & Kuhl, B.A. (2020). Transforming the concept of memory reactivation. Trends in Neurosciences, 43, 939-950.
7. Intraub, H. (2020). Searching for boundary extension. Current Biology, 30, R1463-R1464.