Exploring uncertainty can improve future decisions
In our research, we looked at the question of what makes us choose something uncertain above something we already know and the processes that are represented in the brain when making such kinds of decision.
We’ve all been there. You’re standing in front of four shelves’ worth of biscuits in the supermarket. How is anyone expected to make a decision about which ones to buy? Decision are hard because we often have too many options to choose from. When do you know you have made a good choice and can stop looking for possibly better alternatives? Many times, the decision problem we are facing is making a choice between something familiar or something unknown. Sometimes you just go for what you know, sometimes you decide to get something you’ve never had before. The latter might mean you discover the best biscuits you have ever tasted. This decision problem defines a dilemma that is called the exploration/ exploitation dilemma: you have to choose between exploiting the familiar option with known value or exploring the option with uncertain value which might lead to a better outcome in the future.
When I started my PhD, I was curious about all the different beliefs we might hold about options and how they could determine the trade-off between exploitation and exploration. This led me to study what drives exploratory and exploitative choices and how it is represented in the brain.
The first challenge we faced was to determine which type of beliefs you might hold about possible options that matter in the trade-off between exploration/exploitation and, further, how we could quantify them. When choosing amongst biscuits, your goal is probably to pick those that taste the best. It is easy to evaluate the taste of those you had in the past, however, it is more challenging to be certain about the taste of biscuits that you have not tried before. The approach I took to thinking about this problem was to assume that each time we make a choice and try a biscuit we evaluate how tasty the biscuit is, in other words, we evaluate its value. Moreover, we consider how certain we are about this evaluation: how certain we are that they are tasty. In addition to our belief about value, this is the second key parameter in our study: the uncertainty in our belief about the value. We showed that both beliefs – the belief about the value and the uncertainty of the belief about value – are important when making choices and further, they had different types of influence at different points when a series of decisions were taken so that at different points you might go for something new or familiar: exploratory choices were directed towards options where there was uncertainty about their value (‘I am unsure about how tasty the newly flavoured Oreo biscuits are’), while exploitative choices are directed towards options when we are certain of their value (‘I know the original Oreo biscuits are delicious’). However, which kind of choice you will make – an exploratory or exploitative one, depends on how much time you have left to choose amongst the same set of options. If you can choose amongst different biscuits many times, you have the freedom to explore the ones you are uncertain about, but when there are only a few opportunities left, you will probably choose the ones you know best. We showed that having more time will make you try out new things and eventually might led you discover your new favourite biscuits.
Once we had a better understanding of how different beliefs (‘value’ and the ‘uncertainty about the value’) drive our choices, we wanted to find out how these beliefs are reflected in brain activity. Would we find a neural substrate that represents both subjective beliefs in a manner that it is relevant for decision-making? Are there brain areas that represent subjective beliefs when inspecting exploratory and exploitative choices separately?
In line with previous findings, we showed that both medial and lateral frontal cortical areas are active during exploration. In addition, we found that ventromedial prefrontal cortex had a unique role in representing the beliefs that guided choices. This part of the brain is thought to play a role in calculating the decision variables that guide decision behaviour, but many studies have focussed on its activity mainly during exploitation and so the most relevant decision variable is usually the value of the options being considered. We found that this was the case in our study too but additionally, by using a fine-grained computational model, we were able to show that the ventromedial prefrontal cortex plays an important role in representing multiple beliefs that are important to guide choice behaviour not only during exploitation but also exploration. We found that when you are exploring biscuits about which you are relatively uncertain, ventromedial prefrontal cortex represented the relative uncertainty between options while when you are making decisions about biscuits and you are in the exploitative mode and you are relatively certain of their taste, the same ventromedial prefrontal area now represented the relatively certainty of beliefs about the options. These results show that ventromedial prefrontal cortex represents multiple decision variables (‘value’, ‘uncertainty’, ‘certainty’) and which decision variable is represented is determined by whether you are making explorative or exploitative choices.