RIETI's EBPM Research: History and Challenges

MORIKAWA Masayuki
President and CRO, RIETI

1. Long History of Research on Policy Impacts

Providing research and proposals that contribute to EBPM, or Evidence-Based Policy Making, is RIETI’s raison d’etre, as stated on the top page of RIETI’s website, “Make policy proposals based on evidence.” It is a matter of course that governments should adopt policies which are highly effective in achieving their policy objectives with minimal negative side effects, based on theoretical and empirical research. Therefore, providing the tools that are necessary for accomplishing that goal is the fundamental role of not only RIETI but any policy research institution. That has been the fundamental principle since before EBPM became a popular concept, and policy simulations and cost-benefit analyses using quantitative models have been conducted over that time.

It is in terms of methodology that research concerning EBPM in recent years is significantly different from the research that was done previously. Using various causal inference methods developed through the technical advance of econometric analysis has become a global trend in policy research. Since around the 1990s, as a result of increased availability of panel data, causal inferences using randomized controlled trials (RCT) and natural experiments have become widespread, mainly in such fields as labor economics, public economics, and development economics. This new trend in the analytical approach of economics is known as the “credibility revolution” (Angrist and Pischke, 2010).

Indeed, of the research papers on empirical policy evaluation that have been published in leading international economics journals since around the 2000s, around two-thirds used RCT or some other type of causal inference (Morikawa, 2019) (Note 1).

2. Early Years of EBPM Research at RIETI

Under the mid-term plan (from FY2001) that was prepared when RIETI was founded, RIETI’s mission was defined as conducting policy research and proposal activities based on theoretical and analytical frameworks. Although the term EBPM was not used, RIETI’s fundamental principle of research at that time was essentially the same as the present principle.

Research projects that explicitly adopted “evidence-based” policymaking as the research theme started with (1) research on social security policy (Hidehiko Ichimura, Faculty Fellow, from FY2006) and (2) a series of research works concerning development assistance (Yasuyuki Sawada, Faculty Fellow, from FY2006). (1), which contributed to the progress in the development of panel data regarding elderly people (JSTAR), was developed with the aim of promoting evidence-based policy making based on an abundance of micro-level data as an established practice in the field of social security in Japan. (2) was intended to systematically clarify the governance structure of development assistance based on evidence.

In the 2010s, the concept of evidence-based policy research became the underlying thread behind all of RIETI’s activities. The description of RIETI’s mission on the website that was mentioned at the beginning of this article was introduced at around that time. The third mid-term plan (from FY2011) stipulated that conducting “evidence-based policy research” based on objective and neutral analysis should be the first principle of research. At that time, while the term EBPM was not explicitly mentioned, RIETI’s approach to EBPM was several years ahead of the government’s initiative.

3. The Government’s Approach to EBPM

In the 2010s, the term “evidence-based policy” came to be used, albeit sporadically, in governmental policy documents (Note 2). In particular, the Basic Policy on Economic and Fiscal Management and Reform in 2017 stipulated that the government will “promote evidence-based policymaking” and adopted the acronym EBPM as a standard term. In the same year, the “EBPM Promotion Committee” was established within the Cabinet Office. In this way, the government started to promote EBPM in earnest.

In the first half of 2017, prominent Japanese books concerning causal inference—Ito (2017), and Nakamuro and Tsugawa (2017)—were published, and these are presumed to have had an impact on the government’s initiative to EBPM. Moreover, EBPM-related activities gained momentum in the fields of higher education and academic research, as exemplified by the establishment of the Center for Research and Education in Program Evaluation at the University of Tokyo and EBPM research centers at Osaka University and Hitotsubashi University.

4. RIETI’s EBPM Research: Progress and Expansion

At that time, RIETI established the Policy History and Policy Assessment program (program director: Haruhito Takeda) under the fourth mid-term plan (from FY2016) (Note 3). Under this program, the Promoting Evidence-Based Policy in Japan project (Kazuo Yamaguchi, Visiting Fellow) and the Comprehensive Research on Evidence-Based Policy Making (EBPM) project (Yoichi Sekizawa, Senior Fellow) started in 2017 and 2018, respectively. In terms of organization, RIETI established the “EBPM Unit” in FY2018 and started employing “policy economists,” whose main task is to conduct policy evaluation research related to EBPM and advise policymaking authorities with respect to academic knowledge.

The number of papers on policy evaluation, including ones that do not explicitly mention EBPM in their titles, is increasing. Against the backdrop of a change in the global trend in policy research, the share of papers using causal inference methods, including RCT, RDD (regression discontinuity design) and DID (difference-in-difference), is growing (see Figure 1) (Note 4).


Figure 1: Share in RIETI’s policy evaluation papers by research method

Figure 1: Share in RIETI’s policy evaluation papers by research method
Note: The above data, prepared by the author, covers around 300 RIETI papers that are considered to belong to the policy evaluation research category. “Others” include traditional cost-benefit analysis and VAR analysis.

However, unlike in the case of policies mainly targeting households and individuals, such as employment, education, and healthcare policies, applying causal inference in the analysis of industrial policies will involve various technical difficulties (Morikawa, 2020). Specifically, the feasibility of RCT, which is regarded as the “gold standard” of EBPM, is limited for the following reasons: (i) company-to-company difference is larger than individual-to-individual difference; (ii) it is difficult from the viewpoint of cost to conduct a field experiment that would have a visible impact on companies’ productivity or investment; and (iii) policies that would have a macroeconomic impact are susceptible to secondary effects through general equilibrium mechanism. Therefore, while DID and RDD, which are based on natural experiments, may be useful, there are few actual policies for which the variety of sample groups necessary for such empirical analysis can be secured.

Meanwhile, in April 2022, RIETI established the EBPM Center by expanding an existing internal organization. In addition to the abovementioned EBPM Unit, the EBPM Center comprises a unit responsible for conducting deliberations and providing advice on analysis methods with respect to the sorts of policies to which it is difficult to apply standard causal inference methods, such as large-scale projects. As mentioned at the beginning, there is a long history of model simulation and cost-benefit analysis in the field of policy evaluation research. Even so, in order to identify policy impacts with sufficient precision for practical purposes, it is necessary to set highly reliable parameters, which require more precise and detailed data. In addition, when analyzing new policies, it is necessary to explore new areas for appropriate approaches through trial and error.

5. Challenges for EBPM

RIETI’s EBPM research is only a small part of the policy research initiative that contributes to EBPM. Creating an environment that broadly secures easy access to policy research for researchers from universities and research institutions is essential in order to expand the breadth of EBPM research, although this may appear to be a roundabout way of contributing to EBPM. The challenges that should be addressed on the policymaking side in order to create such an environment include developing policy-related data in a way that will be usable by researchers in general and lowering the barriers to use of government statistics and micro-level industry data.

It is a welcome turn of events that awareness of EBPM is growing on the frontlines of policymaking, but it should be kept in mind that there are various limitations to policy evaluation analysis (Note 5). It is impractical to make strong determinations as to whether or not a certain policy has a positive impact based only on the results of a single policy evaluation analysis. In addition, although redistributing policy resources based on evaluation results is necessary for creating a meaningful EBPM cycle, it is essential to avoid the folly of rapidly making “policy-based evidence making” in order to obtain a conclusion.

There is a large body of established knowledge that does not require new analysis. What is practically important to do at the policy-making stage is to make effective use of existing academic evidence. For example, it is important to avoid adopting policies which, in light of the accumulated research results, are highly likely to be ineffective, to bring little benefit relative to cost, or to produce significant negative side effects, and if such policies already exist, they should be abolished or corrected (Note 6).

April 28, 2022
Footnote(s)
  1. ^ Based on the author’s calculation of papers published in three journals—American Economic Review, Quarterly Journal of Economics, and Journal of Political Economy—in 2001-2019.
  2. ^ The first instance of reference to evidence-based policy making in policy documents subject to cabinet-level authorization is probably the phrase “policy making based on objective evidence” in the Fourth Science and Technology Basic Plan (2011).
  3. ^ Under the fifth mid-term plan (from FY2020), a policy evaluation program (program director: Daiji Kawaguchi) was established.
  4. ^ Information on RIETI’s EBPM-related papers, reports, and symposiums is summarized at: https://www.rieti.go.jp/jp/projects/ebpm/index.html (in Japanese).
  5. ^ Ohashi (2020) is useful in that it discusses not only EBPM’s significance but also its limitations and points of attention regarding practical affairs.
  6. ^ For example, Bloom et al. (2019) conducted a comprehensive evaluation of various policies and programs intended to promote innovation based on existing academic evidence. Excellent survey papers are useful for policy making.
Reference(s)
  • Angrist, Joshua D. and Jörn-Steffen Pischke (2010), “The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics,” Journal of Economic Perspectives, 24(2), 3-30.
  • Bloom, Nicholas, John Van Reenen, and Heidi Williams (2019), “A Toolkit of Policies to Promote Innovation,” Journal of Economic Perspectives, 33(3), 163-184.
  • Ito, Koichiro (2017), The Power of Data Analysis: How to approach causality, Kobunsha.
  • Morikawa, Masayuki (2019), “Evidence regarding EBPM,” presentation material used at the RIETI EBPM Symposium (https://www.rieti.go.jp/jp/events/19122501/handout.html).
  • Morikawa, Masayuki (2020), “Verification of the Effects of Industrial Policies: Introduction of Research Conducted in Japan and Overseas [Sangyo Seisaku no Koka Kensho: Naigai no Kenkyu Rei no Shokai],” EBPM Report (https://www.rieti.go.jp/jp/special/ebpm_report/007.html).
  • Nakamuro, Makiko and Tsugawa Yusuke (2017), Causal Inference in Economics, Diamond Inc.
  • Ohashi, Hiroshi (2020), Economics of EBPM, University of Tokyo Press.

June 21, 2022