As the authors assist the national and local governments in promoting EBPM (Evidence-Based Policy Making), we are frequently asked by officials involved in policy development about how to implement EBPM and what KPIs (key performance indicators) should be set at which level. What lies behind those questions appears to be an understanding that EBPM means setting appropriate KPIs and measuring and evaluating the KPI achievement. However, in many cases, even if KPIs are set and measured, it is still difficult to implement EBPM.
Therefore, in this article, we will discuss specifically how policy management that is based on the setting and measuring of KPIs (KPI-based policy management) is different in approach from EBPM and why it is important to distinguish between these two methods in order to promote EBPM.
Differences in Approaches to Assessing Achievement between KPI-based Policy Management and EBPM
Table 1 outlines differences in the approach to assessing achievement between KPI-based policy management and EBPM.
KPIs are indicators which evaluates performance. The purpose of KPI-based policy management is to identify the progress in policy implementation, the results, and the achievements against targets as simply and as broadly as possible by setting KPIs and monitoring implementation and to use the findings for policy development. In the study of evaluation, this method is known as performance measurement. Under the framework of policy evaluation in Japan, it is called "results-based evaluation" (Note 1). Indicators presented in the Administrative Project Review Sheet are based on this approach, and "administrative evaluation," which has been introduced widely among local governments, is also basically a performance measurement.
On the other hand, EBPM is a policymaking approach that places emphasis on evidence which is provided through analysis of the causal effects of policies. However, it is in no way easy to clarify a causal relationship between policies and results. Therefore, under EBPM, it is essential to focus on examining the causal effects of only highly necessary or important policies, rather than conducting an exhaustive examination of the effects of a broad range of policies. The key distinction between EBPM and KPI-based policy management is that EBPM precisely examines how the target indicators would have moved if the policy under examination had not been implemented. Examining effects based on hypothetical propositions like that, which is called a counterfactual approach, is the central concept of causal inference, which is a method of statistical analysis of causal relationships. It may be said that EBPM is a policy development method based on the causal inference approach. However, unfortunately, it is difficult to clarify the causal effects of policies by merely pursuing KPI-based policy management, as will be explained later.
|KPI-based policy management (performance measurement, results-based evaluation, and administrative evaluation)
|Periodically measuring and evaluating the results and efficiency of policies, measures, projects, etc. (not necessarily paying attention to causal relationships)
|Clarifying a causal relationship between policies and results
|Covering a broad range of policies, measures, and projects
|Focusing on policies of interest
|Source: Prepared in reference to Ono (2018).
Different Requirements for Indicators between KPI-based Management and EBPM
Because of the abovementioned difference in purpose, different sets of requirements should be set for target indicators used under KPI-based management and EBPM. Table 2 outlines the different sets of requirements.
KPI-based policy management characteristically seeks to identify the status of policy implementation as simply and broadly as possible. As a result, target indicators under KPI-based policy management must be ones that can be measured periodically and continuously, and ideally, data-collection costs should be low. Additionally, rigorous examination of causal effects is not required, so it is not necessary to collect data concerning a control group (a group to which the policy under examination is not applied). Finally, the evaluation simply consists of determining whether or not the preset target level has been met.
On the other hand, in the case of target indicators used under EBPM, the main focus of interest is the examination of the causal effects of policies, so data should not necessarily be collected periodically, while a relatively high data-collecting cost may be acceptable. However, in order to precisely examine a causal relationship, it is necessary to clarify the degree to which the target indicator actually improved compared to a counterfactual case in which the policy under examination has not been implemented. As a result, in most cases, it is necessary to collect data concerning a control group (to which the policy under examination is not applied). Under normal policy management, even though data may be collected with respect to the treatment group (to which the policy under examination is applied), it is difficult to collect with respect to a control group in many cases. This point often presents a hurdle to the implementation of EBPM.
|KPI-based policy management (performance measurement, evaluation of results, and administrative evaluation)
|Capable of periodically and continuously identifying performance.
|Identify effects only at a timing that is necessary and adequate for examining the effects is sufficient (effects do not necessarily have to be identified periodically and continuously).
|Data collection cost
|Ideally, data can be collected at a minimum cost from administrative information and other sources.
|If evaluation is conducted just once, a relatively high cost is acceptable.
|Data concerning control group
|It is acceptable to use only data concerning the treatment group (to which the policy under examination is applied).
|Essential for precisely examining a causal relationship
|Whether or not the preset target level has been achieved.
|The degree of improvement of the indicator compared with a control group (to which the policy under examination was applied).
|Source: Prepared by the author.
EBPM and KPI-based Policy Management Should Be Used Alternately for Different Purposes
There are significant differences in purpose and approach between KPI-based policy management and EBPM, and the two methods have their respective advantages and disadvantages. While simplicity is an advantage of KPI-based policy management, inability to collect sufficient information concerning causal effects is a disadvantage. On the other hand, EBPM has the advantage of being able to identify causal effects but it is at a disadvantage in that it inevitably requires more labor and incurs higher costs.
In the frontlines of administrative policy development in Japan, these two methods tend to be confused in the absence of a clear understanding on their differences and respective advantages and disadvantages. In 2021, in order to further promote EBPM in Japan, it is important to use the two methods alternately based on their different purposes and strengths.
This article was prepared by making additions and revisions to Chapter II, Section 4 of "Report on FY2019 Industrial Economic Research Commissioned Project (Assistance and Research Related to Practices of Evidence-Based Policy Making, etc.)," which was compiled by Mitsubishi UFJ Research and Consulting Co., Ltd. We would like to express our appreciation to the Policy Evaluation and Public Relations Division at the Minister's Secretariat of the Ministry of Economy, Trade and Industry for providing consent to our using the report in this way.