Evaluating what works (or doesn't) in energy innovation policy
How does public spending impact clean energy innovation? The evidence on what works best, and why, is still surprisingly limited. We develop a framework for energy innovation policy and programme evaluation for generating a wider evidence base.
By Jacquelyn Pless, Cameron Hepburn, and Niall Farrell
We need further advances in energy technologies and services to tackle climate change. Market forces will deliver some of these advances, particularly for technologies that are already commercially competitive, or close to it. However, there is a gap between what the market is likely to deliver, and the pace and scale of the transformation required to reach net zero emissions by 2050. Large amounts of public spending will be required to fill this gap.Yet indiscriminately ramping up support for clean energy R&D does not guarantee success. Twenty-four countries and the European Union committed to double public spending on clean energy research, development, and demonstration (RD&D) by 2021 through “Mission Innovation”. Spending these and other resources wisely is key to ensuring that the intended outcomes are achieved, and in a cost-effective manner.
This paper highlights how modern empirical economics research—when conducted with sufficient rigour—provides valuable insight that can be used to improve energy innovation policy and programme effectiveness. Implementing clever research designs can uncover detailed nuances about what funding programmes and policies work, what doesn’t, and why. But doing so comes with its challenges. Our paper highlights solutions for overcoming these challenges to generate a wider evidence base.
For instance, quantifying how much of innovation activity can be attributed directly to a policy or programme as opposed to other factors (i.e., estimating the “causal effect”) is crucial for understanding the returns to public spending. Randomised control trials are often seen as the gold standard, and for good reason, but expensive trials are not always necessary if one can embed other forms of “randomness” in the policy design. This mimics the nature of an experiment. One possibility for the energy sector is to randomise certain requirements attached to grant funding (once grant recipients are selected), such as the obligation to collaborate with specific types of firms, national laboratories, or universities. Through this randomness, the importance of collaboration for innovation success can be measured. Policymakers and funding agencies, in turn, may consider whether to require or encourage such collaboration based on the observed outcomes.
We are not suggesting funds should be allocated at random per se, which has the potential to undermine domain expertise in the project proposal evaluation process, although some jurisdictions are indeed taking this approach. Rather, some alternative approaches to randomisation are close to “free lunches” in the sense that the randomness is introduced into a variable which is otherwise set arbitrarily, if at all. We can create value from a policy attribute that otherwise had limited utility.
There may be even lower hanging fruit. Some “quasi-experiments” are already embedded in existing policy and programme designs, and while the data to study them often exists, it is not usually readily available to researchers. Enabling researchers to access data, while respecting privacy concerns, may deliver further insight.
In our paper, we identify some tweaks that may provide such insight, and we identify the sort of evidence that could be prioritised. A greater emphasis on understanding what works best for whom would be useful, as what works for one organisation may not work for another. This heterogeneity extends to the types of innovations developed; some policies may be better than others at driving innovation that protects environmental systems.
This is a sample of the ideas explored in our paper. With the help of reviewers and the editors of Nature Energy, we have highlighted a set of research methodologies and priorities that could help us shed more light on what works and why, which can be used to improve energy innovation programme and policy design. Getting this right—as soon as possible—matters. We are at a turning point in one of the greatest challenges to face humanity. Cheap, clean and effective energy technologies are required urgently; the evidence base on which to make sensible investments in R&D was arguably needed yesterday. Today is better than tomorrow.