1. Introduction
In order to maximize R&D results through early marketization, it is necessary to construct a management system that incorporates standardization into the R&D project from the beginning, rather than starting standardization efforts after the R&D results are obtained. Building an R&D management system that emphasizes the marketability of R&D results is an important policy issue from the perspective of the goal of building an innovation system.
In order to promote such an R&D system, it is also necessary to develop a policy evaluation system for standardization. The development of policy evaluation methods for standardization is an issue that needs to be addressed [1][2]. This article briefly explains the issues and background of the system design of the policy evaluation for standardization activities in the second mid-term plan period of NEDO (New Energy and Industrial Technology Development Organization), for which I worked as a policymaker on the system design through a trial-and-error process (Note 1).
Since the methodology for policy evaluation concerning standardization has not been fully established, it is essential to accumulate knowledge through the study of theoretical aspects as well as through the accumulation of individual cases of implementation. For this reason, the case study presented in this article will be useful for both practitioners and academic researchers when designing evaluation systems for R&D institutions and national standardization activities. Simultaneously, the evaluation of standardization activities is an area where there is a general lack of knowledge worldwide[1][2]. I hope that the information presented here will be useful for innovation system reform around the world.
2. Overview of R&D Policy Evaluation System
2.1. Framework
Although NEDO, as a national research and development organization, has its own evaluation system for project evaluation, the specific system design is based on the evaluation system for national R&D projects (Note 2).
The first guideline for the evaluation of national R&D projects (policy evaluation and program evaluation), "General Guidelines for National R&D Evaluation," was established by the Prime Minister in 1997 [3]. The former Ministry of International Trade and Industry (now the Ministry of Economy, Trade and Industry) determined the evaluation guidelines for R&D projects and national R&D organizations under its jurisdiction at the same time as the primary guidelines were formulated [4] (Note 3).
The evaluation targets are broadly classified into two categories: (1) evaluation of individual R&D projects (ex-post evaluation that looks at the results on an R&D project basis) and (2) evaluation of the entire R&D organization (institutional evaluation that looks at the results of organizational R&D).
There are three necessary elements to policy evaluation: (1) data, (2) indicators, and (3) target values. (1) indicators and (2) target values must be meaningful and feasible in terms of related policy. Only indicators that have the right policy meaning and that are feasible are accepted as prerequisites for policy evaluation. The setting of wrong achievement targets has a negative impact on R&D activities. Therefore, in-depth understanding and insight into the data and policy goals used in the evaluation are essential for setting appropriate targets and target values.
2.2. Current status
NEDO has been promoting efforts to introduce quantitative evaluation indicators for evaluating the results of the standardization of research outcomes [5][6][7][8]. There have been academic discussions on evaluation methods for standardization activities [1][2]. However, there are still many issues to be addressed. In Japan's Fifth Science and Technology Basic Plan (FY2016-2020), only qualitative goals for standardization are stipulated, and no quantitative numerical goals are stipulated (based on research in 2019)[9].
3. Case Study of New Energy and Industrial Technology Development Organization (NEDO)
3.1. Background
The setting of targets for NEDO's R&D evaluation is based on its mid-term plan, which is formulated every five years. At this point, plans for the first (2003-2008), second (2008-2013), third (2013-2018), and fourth (2018-2023) terms have been formulated [5][6][7][8]. Before NEDO's second term (during the first mid-term plan), only "qualitative targets" were set to evaluate the results of standardization activities. This was because the design of a quantitative evaluation method for standardization was still a trial-and-error process.
3.2 Issues to be solved
NEDO needed to design a mechanism to evaluate the degree of achievement of standardization results only for R&D results that are "suitable for standardization." In other words, it was necessary to prepare a system that does not forcibly require the standardization of R&D results that are not suitable for standardization.
If NEDO research projects only focused on the ICT field, evaluating almost all research results of the achievement of standardization might have been appropriate, but NEDO's research projects cover a wide range of fields (e.g., biotechnology and energy in addition to ICT). Not all R&D results necessarily require standardization for commercialization. In some cases, standardization may be an obstacle to the commercialization of research results.
Moreover, a point of concern was whether the number of successfully established standards should be used as a numerical target in evaluating the policy results. In the case of a standard, the standard development organization's consensus is usually required before the proposal is accepted as a standard. It is difficult to predict whether the consensus will be obtained or not. In addition, there is a good chance that the content will be changed through deliberation. For this reason, it was pointed out that using the number of approved standards as a numerical indicator would mean setting a target with a high degree of uncertainty.
3.3. Adopted evaluation design
In order to overcome the challenges, a quantitative evaluation system based on numerical targets separated into two distinct groups of plans was introduced (Note 4):
(1) The five-year medium-term plan does not specify numerical targets but only evaluation methods (evaluation indicators).
(2) Annual plans set numerical targets for evaluation indicators.
One feature of this system is that standardization activities can be monitored both on an annual basis and on a medium-term basis. This enables NEDO to set reasonable numerical targets only for research appropriate for standardization, taking into account the content of the R&D results before the start of each year.
In addition, as a target indicator, it was decided that the numerical target would be the "number of standards proposed" rather than the "number of standards established." This is because it is necessary to set a predictable and feasible target in order to realize the policy goal. The policy evaluation here is an assessment of achievement. Therefore, a goal without foreseeability is not an appropriate goal because it is difficult to estimate in advance the necessary amount of policy resources (funds and human resources) for achieving it (i.e., both can be estimated to require an infinite amount), making a comparative cost-benefit analysis impossible (Note 5).
3.4. Sharing of case studies within relevant ministries and agencies to promote standardization activities
In order to accumulate knowledge and best practices such as policy evaluation related to standardization, the "Relevant Governmental Agencies Liaison Meeting on International Standardization" (2007) was established to provide an opportunity to exchange information on the standardization policy initiatives of each ministry and agency. The secretariat of this liaison meeting was the Japanese Industrial Standards Committee (JISC) [10] (Note 6). Since standardization covers a wide range of products and services, multiple ministries and agencies from the Japanese government are involved. For this reason, efforts to both share and solve issues are beneficial. However, at the time of implementation, no appropriate mechanism for sharing information among ministries and agencies existed.
Like Japan, there are many countries where different ministries hold jurisdiction over different industries. Thus, Japan's efforts are considered to be a good international reference example.
4. Summary
In the evaluation of R&D policies and measures, it is essential to develop theories for the sophistication of the methods and also to develop concrete methods that can be implemented. In this sense, in order to ensure the implementation of a system that is appropriate to its purpose and allows for future improvements, background information on the issues and challenges behind evaluation systems that have been introduced is essential. For this reason, I introduced the issues and background behind the first introduction of numerical targets for standardization in the NEDO case. I hope this article will help researchers and policymakers in tackling related issues in the future.
Acknowledgments:
This study was supported by JSPS Grants-in-Aid for Scientific Research (19K01827 PI: Suguru TAMURA).
