Momentum is building for Japan to introduce an evidence-based policymaking (EBPM) approach, following the example of the United Kingdom. Evidence-based policymaking examines empirically the effectiveness of policies implemented in the past and possible future policy options, and makes policy plans by taking into consideration the findings obtained. Our discussion in this article will focus on the government's policymaking process, but we believe that corporate decision making has similar problems to those of the government's policymaking.
The Institute for Fiscal Studies (IFS), a UK private-sector think tank, has been playing a significant role in the country's evidence-based policymaking. Since the 1980s, IFS has collaborated with other organizations in publishing an annual report in the run-up to the Chancellor of the Exchequer's announcement of the following year's budget, whereby they examine overall economic trends and identify possible options in each policy area such as tax, social security, labor, and education.
Japan is following suit. In its Basic Policy on Economic and Fiscal Management and Reform 2017, the government has clarified its intent to establish a framework for promoting evidence-based policymaking and ensure that evidence-based discussions and studies be properly reflected in the government's budget plans. In response to this development, the Cabinet Office has started examining the effectiveness of policy measures in the area of education and employment support on a trial basis.
In what follows, we will present four factors that are essential to making evidence-based policymaking deliver its intended effects.
♦ ♦ ♦
First, policy evaluation, which constitutes the backbone of evidence-based policymaking, should be carried out by agencies independent from those responsible for policy planning and implementation. Government ministries and agencies are already evaluating policies under their respective jurisdictions with the Ministry of Internal Affairs and Communications (MIAC) playing an overarching role, as provided for under the Government Policy Evaluation Act brought into force in April 2002. However, the independence and objectivity of the existing system, in which a ministry or agency self-evaluates its own policy measures, are questionable. To be sure, administrative programs are subject to review by a panel of external experts. However, both the programs to be reviewed and the external experts to serve on the review panel are selected by the ministry or agency concerned. This casts doubt on the independence of the panel of "external" experts. Furthermore, those serving on external review panels are mostly certified public accountants and lawyers, and as such, not necessarily well equipped to dig deep into the question of how policy evaluation should be carried out.
To begin with, it takes a coordinated effort by a group of experts to conduct an empirical analysis of policy effects. It is important to establish the following flow of work: 1) multiple independent agencies critically evaluate a specific selected policy, and 2) administrative officers involved in the process convey to legislators any findings or insights derived therefrom regarding the effectiveness of the policy so that such findings or insights will be properly reflected in future policymaking.
Second, in evaluating a certain policy, it is necessary to pay attention to the appropriateness of the intended policy goals as well as to other policy options that could achieve the same goals. For instance, if the government is to pursue a policy to make college education free, it is crucial to carry out both ex-ante and ex-post evaluations to measure its impact. In addition, it is also important to identify broadly defined policy goals—e.g., the correction of disparities, productivity enhancement, or whatever else the government intends to achieve by offering tuition-free college education—and to examine beforehand whether such policy goals are top priorities for Japan today and whether the tuition-free college education is more effective than various other policy options.
♦ ♦ ♦
Third, it is necessary to establish a system that enables external independent agencies to access data needed to evaluate policies. In order to be able to make accurate assessments, external evaluators must have an accurate understanding of the intention and actual operation of the policy in question, and thus they need to communicate closely with those responsible for the implementation of the policy.
Meanwhile, the availability of high-quality data is essential to allowing independent policy evaluation. What is important in policy evaluation is to make available data on individuals who are comparable to those affected by a certain policy in various attributes—academic background, age, work experience, place of residence, etc.—except that they are not subject to the policy. For instance, in evaluating a program designed to help people find jobs, data collected on those individuals who have received support under the program are all that is available so far. This makes it extremely difficult to evaluate the program by using the latest analytical techniques.
Meanwhile, among the data collected by government agencies in the course of conducting their business—including statistical data held by local governments as well as data on pensions, unemployment insurance claims, and tax payments—are quite a few datasets that would be useful in policy evaluation. It is necessary to build an institutional infrastructure to enable effective use of data collected by the central and local governments. While both the United States and Europe have extensive sets of panel data collected over time by surveying the same individuals at multiple occasions, such is not the case in Japan, where there are very few panel surveys that are reliable with fairly high response rates. We must change this situation immediately.
Fourth, the presence of human resources equipped with expertise needed for evidence-based policymaking is an absolute must. Microeconomics, macroeconomics, and econometrics are the core knowledge base for evidence-based policymaking. According to our experience, it takes two 105-minute classes a week over a course of about 13 weeks to teach the basics of econometrics even to students with some background in mathematics and statistics.
Those students who have mastered basic knowledge on econometrics should move on to study economics at the graduate level to acquire more specialized knowledge through econometric courses focused specifically on policy evaluation methods and/or courses on labor economics and the economics of education. In addition, if they are to become capable of evaluating policies, it is essential to develop skills to make research plans, collect and analyze data, and summarize their findings. They also need to become experienced in writing research papers. As a matter of consequence, those equipped with all such skills and expertise are necessarily those who have completed doctorate degrees. However, neither central government agencies nor private-sector think tanks commissioned to evaluate government policies are sufficiently staffed by human resources properly trained for the task, posing an obstacle to promoting policymaking firmly based on empirical findings.
Another equally important requirement concerning human resources is to secure excellent university researchers capable of undertaking research at an internationally recognized level and providing high-quality education firmly supported by their solid knowledge. Failure to do so would mean that Japanese universities will lose their ability to provide such training and become unable to attract excellent students who can compete internationally.
Over the past 15 years, the level of pay for economic researchers has increased in North American and certain Asian countries. Japan is facing a serious brain drain to those countries where universities typically pay two to three times more than that offered by Japanese universities. The researchers who would have been trained by distinguished researchers are not visible, but the number of such "forgone" researchers is increasing.
♦ ♦ ♦
Techniques for policy evaluation have evolved significantly over the past 30 years. Of course, evaluating the effectiveness of policies in today's highly complex reality involves a number of assumptions, and one could take the cynical view that it is possible to draw any conclusion by changing assumptions.
In proceeding with explicit evidence-based policymaking, it is necessary to disclose the assumptions underlying conclusions. In the course of discussing those assumptions, any conclusions derived from wrong assumptions would be filtered out. What Works Centres, a group of organizations jointly formed by the public and private sectors in the United Kingdom to improve the quality of evidence-based policymaking, assign a score to each piece of evidence based on its reliability. The robustness of assumptions used for analysis also affects the score.
With its population aging and the cost of social security rising, Japan is facing strong headwinds. If Japan is to realize—against this adverse backdrop—a society where people can fully demonstrate their abilities, policymaking based on empirical findings is a must. Ensuring the viability of such policymaking inevitably requires long and persistent efforts to design institutional and regulatory systems for policy evaluators, improve the quality of data and create an environment that facilitates effective use of such data, and develop relevant human resources.
The University of Tokyo's Graduate School of Economics has recently established a Center for Research and Education in Program Evaluation (CREPE) in collaboration with the Institute of Social Science and the Graduate Schools of Medicine, Education, and Law and Politics. The center will undertake research on policy evaluation and develop, through practical training, human resources to engage in policy evaluation.
The IFS served as a model for the CREPE. In evaluating various policies, the center will hire not only researchers but also outstanding undergraduate and graduate students and let them learn by practice. Through such experience, those students would determine their future courses and demonstrate their abilities in various sectors of society, not limited to the academic field but including the government, consulting services, journalism, and international organizations. We hope that the new center will serve as a springboard for young talents.
* Translated by RIETI.
October 16, 2017 Nihon Keizai Shimbun
