Evidence of "Evidence-based Policymaking"

Vice Chairman & Vice President, RIETI

The importance of evidence-based policymaking has been highlighted in recent years. International institutions led by the Organisation for Economic Co-operation and Development (OECD) and the World Bank are promoting this policymaking approach vigorously, while some advanced economies, such as the United States and the United Kingdom, are taking concrete steps to put it into practice (Note 1). The approach has been applied to various policy areas including healthcare, social security, labor, and education. Japan has recently begun to take similar steps in some policy areas such as science and technology. In a bid to promote evidence-based policymaking, the Ministry of Economy, Trade and Industry (METI) seems to be planning to mandate the presentation of empirical evidence, such as statistical data and research findings, as a requisite to introducing new policy programs. The move should be welcomed as it contributes to the effective use of limited financial, personnel, and other policy resources.

Roles of policy think tanks

In order to ensure that evidence-based policymaking will deliver its intended effects, not only policymakers but also relevant research institutes have important roles to play. For instance, IZA, a leading think tank in Europe which has a partnership with RIETI, has a website section called "IZA World of Labor: Evidence-based policymaking," which is designed to provide policymakers with findings from academic policy research in the areas of labor economics. Its research findings have also been compiled and published as a book (Zimmermann and Kritikos, 2015). High-quality research findings are presented in a way that is easy to understand for the general public on various issues of great interest to Japan including non-standard employment (part-timers, agency workers, etc.), gender inequality, child care support, foreign labor, and employer-provided education and training.

RIETI also takes it as its important mission to put forward evidence-based policy proposals. Specific examples include Fujita ed. (2016), a recent publication from RIETI, which has a subtitle entitled "Evidence-based Policy Recommendations" and provides a bird's eye view of research findings in each area relevant to putting the Japanese economy on a sustainable growth path.

However, in order to promote evidence-based policymaking, we first need to have evidence as to how such an approach is perceived by Japanese policymakers, to what extent it is being utilized in a practical policymaking setting, and what factors, if any, stand as obstacles to taking such a policymaking approach.

Actual state of evidence-based policymaking

With an aim to link RIETI's research findings to policy planning and formulation, we conducted surveys to find out how policymakers and policy researchers perceive evidence-based policymaking, asking their views on: 1) the necessity of evidence-based policymaking, 2) policymakers' awareness, 3) progress hitherto made in implementing evidence-based policymaking, and 4) factors inhibiting the implementation thereof. In doing so, we thought it would be desirable to be able to identify the perception gap between policy researchers (including think tank researchers and academic scholars engaging in policy research) and policymakers. Thus, we conducted two separate surveys on two different samples—(A) government policymakers and (B) policy researchers (RIETI fellows)—using almost identical questionnaires (Note 2). Since these are simple surveys conducted on small samples, I have considerable reservations about interpreting the survey results. But I would like to introduce some of our preliminary findings.

Aggregate survey results regarding the first three questions are shown in Figure 1. The number beside each bar, which takes the value of one through four, represents the average score rated by respondents in each group using a single-answer, multiple-choice format. The higher the value is, the greater is the tendency for the respondents to believe that "evidence-based policymaking is necessary," "policymakers are aware of evidence-based policymaking," or "evidence-based policymaking is implemented" as applicable, with a value of 2.5 indicating that the positive and negative responses are roughly equivalent (Note 3).

Figure 1: Evidence-based Policymaking
Figure 1: Evidence-based Policymaking
Note: Rated on a scale of one to four. The higher the value is, the higher is the level of necessity, awareness, or implementation as applicable.

Both policymakers and policy researchers showed similar patterns in their responses. Respondents in both groups are quite forthcoming in acknowledging 1) the necessity of evidence-based policymaking. However, they gave rather low scores to 2) policymakers' awareness of evidence-based policymaking, and their evaluations were even lower on 3) the implementation of evidence-based policymaking. Meanwhile, a comparison between the two groups show that policy researchers' evaluations on policymakers' awareness of evidence-based policymaking are lower than policymakers' self-evaluations, and the same tendency is observed on the degree of its implementation (Note 4).

One interpretation of this is that the differences represent perception gaps between the two groups on the quantity and quality of evidence deemed necessary. Whether in making budget requests or amending laws, there are many hurdles that need to be cleared, such as priority setting within each organization, scrutiny in the budget-making process, and deliberations at the Diet, making it impossible to formulate any policy without evidence. Furthermore, in a relatively recent move, government policy programs have been made subject to ex post evaluation, for instance, through the administrative project review system launched several years ago.

At the forefront of policymaking, aggregate statistical data, information obtained from interviews with companies, and overseas case examples tend to be used as evidence showing the need to maintain, alter, or scrap specific policy programs. In contrast, policy researchers are more inclined to look to detailed empirical analysis in assessing the effects of policy measures. For instance, they would try to determine whether an observed correlation is a causal relationship, or they would want to know the size of effects in quantitative terms. Also, there is a growing tendency to put emphasis on the estimation of causal relationships based on evidence from natural experiments and more recently on policy evaluation using randomized controlled trials (RCT).

The quantity and quality of evidence needed in actual policy formulation vary depending on the skill levels of senior decision-makers and examiners. This is a question of whether it should be considered sufficient enough to present specific examples and show the existence of a correlation, or if further evidence is required.

Obstacles to evidence-based policymaking

Figure 2 illustrates responses to the question concerning factors inhibiting evidence-based policymaking (Note 5). It is a multiple-answer question and each percentage value represents the ratio of respondents who selected the answer. "Policy decisions are made politically irrespective of evidence" is the most chosen answer by policymakers, followed by "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings," and "Evidence-based policymaking is neither a customary practice nor in line with the organizational culture" in that order. This compares to "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings," "Policymakers are too busy in daily works and unable to find time for evidence-based policymaking," and "Policy decisions are made politically irrespective of evidence" selected by policy researchers, showing some differences in the patterns of responses between the two groups (Note 6).

Figure 2: Obstacles to Evidence-based Policymaking
Figure 2: Obstacles to Evidence-based Policymaking
Note: Multiple answers are allowed and each percentage value represents the ratio of respondents who selected the answer.

It is interesting that policymakers, who appear very busy in the eyes of policy researchers, do not find their heavy workload as constraints on evidence-based policymaking (Note 7). "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings" was selected by roughly two-thirds of the respondents in each group, suggesting that improving government officials' analytical skills is crucial to evidence-based policymaking. It is expected that not only the use of microdata collected for government statistics but also the applications of big data and artificial intelligence may become feasible in the coming years, and it may require much higher levels of skills to utilize them (Note 8).

Conclusion and some reservations

Few respondents deny the necessity of evidence-based policymaking. However, when asked whether they think evidence-based policymaking is being implemented in Japan, both policymakers and policy researchers are far from positive in responses. In addition to evolving the existing policymaking and ex post evaluation mechanisms, government officials need to improve their skills to utilize academic research findings. At the same time, it is also important to enhance the interest of academic scholars and researchers in real-world government policies, and thereby accumulate and disseminate in an easy-to-understand way research findings useful for policymaking.

As aforementioned, the surveys were conducted on small samples, which are presumably biased in favor of evidence-based policymaking as respondents contained therein are either policymakers having contact with RIETI or researchers engaging in policy research at RIETI. There is no ruling out the possibility that policymakers who have no contact with policy research and scholars who have very few occasions to interact with policymakers may have different views. Also, the ratings provided by respondents on various aspects of evidence-based policymaking are based on their subjective judgments.

Also, even though both policymakers and researchers definitely affirm the importance of evidence, they may differ in their understanding of what constitutes sufficient evidence. Finding ways to bridge such gaps between policymakers and researchers is an important role that policy think tanks, including RIETI, are required to play.

May 27, 2016
  1. ^ Particularly, in the United Kingdom, evidence-based policymaking has been promoted for nearly 20 years since the government of Prime Minister Tony Blair (see Ieko et al., 2016).
  2. ^ The survey of sample (A) was conducted from December 2015 through January 2016 and for sample (B) from February through March 2016. The number of effective responses is 192 for sample (A) and 50 for sample (B). I am grateful to Mr. Junichi Ogawa and Ms. Akemi Mogi of RIETI who helped with conducting the surveys. I would also like to express my appreciation to the policymakers and researchers who took time from their busy schedules to respond to the surveys.
  3. ^ 1)Questions asked in 1) through 3) are as follows: 1) Do you think evidence-based policymaking is necessary?; 2) Are you cognizant of an evidence-based approach in performing your policymaking duties? (for policymakers) / Do you think that policymakers are cognizant of an evidence-based approach in performing their policymaking duties? (for policy researchers); and 3) Do you think that evidence-based policymaking is being implemented in Japan? Answer options are "Yes, definitely," "Yes, to some extent," "Not very much," "Not at all," and "Not sure / Do not know." In Figures 1 and 2, those who selected the last answer option ("Not sure / Do not know") are excluded from aggregation.
  4. ^ There is a statistically significant difference at the 1% level in 2) and 3).
  5. ^ Respondents were asked: "Which of the following factors do you think inhibit evidence-based policymaking? (Choose all that apply.)" Answer options are: "Policymakers are too busy in routine works and unable to find time for evidence-based policymaking," "Evidence-based policymaking is neither a customary practice nor in line with the organizational culture," "Policy decisions are made politically irrespective of evidence," "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings," "Useful data and research studies are very limited in availability," and "Others."
  6. ^ There are statistically significant differences between the two groups at the 5% level in the ratio of respondents who chose each of the following answers: "Policymakers are too busy in routine works and unable to find time for evidence-based policymaking," "Evidence-based policymaking is neither a customary practice nor in line with the organizational culture," and "Policy decisions are made politically irrespective of evidence." Meanwhile, as evident from Figure 2, there is no significant difference in the ratio of those who selected "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings."
  7. ^ Based on her own experience of serving as chief economist at the U.S. Department of Labor by temporarily leaving academia, Adriana Kugler notes that while policy development in Washington is surprisingly grounded on evidence, time is the biggest constraint inhibiting evidence-based policymaking (Kugler, 2014). Based on this observation, she points to the importance of increasing interactions between policy makers and academic researchers.
  8. ^ Uchiyama (2015) calls for reviewing the recruitment and personnel management systems at government agencies in order to enhance Japan's policymaking capabilities, noting that the number of government economists increased in the United Kingdom under the Blair government, which emphasized evidence-based policy making.
  • Fujita, Masahisa ed. (2016) Sustainable Growth of the Japanese Economy: Evidence-based policy recommendations, University of Tokyo Press.
  • Ieko, Naoyuki, Yohei Kobayashi, Natsuko Matsuoka, and Shinji Nishio (2016) "How policy formation changes with evidence: Trends toward evidence-based policy in the UK, as verified by randomized controlled trials, and some implications for Japan," Mitsubishi UFJ Research and Consulting Policy Research Report.
  • Kugler, Adriana (2014), "Labor Market Analysis and Labor Policymaking in the Nation's Capital," Industrial and Labor Relations Review, 67, Supplement, 594-607.
  • Uchiyama, Yu (2015), "Seisaku Ritsuan Noryoku Takameruniwa: Keizaibunseki no senmonka saiyo o [How to Enhance Policymaking Capabilities: Recruit more specialists on economic analysis]," May 29, 2015 Nihon Keizai Shimbun.
  • Zimmermann, Klaus F. and Alexander S. Kritikos (2015), Evidence-based Policy Making in Labor Economics, London and New York: Bloomsbury.

June 20, 2016