R&D policy evaluation system at NEDO, a national R&D agency (An example of the introduction of quantitative evaluation indicators for technology standards)

Senior Fellow, RIETI

1. Introduction

In order to maximize R&D results through early marketization, it is necessary to construct a management system that incorporates standardization into the R&D project from the beginning, rather than starting standardization efforts after the R&D results are obtained. Building an R&D management system that emphasizes the marketability of R&D results is an important policy issue from the perspective of the goal of building an innovation system.

In order to promote such an R&D system, it is also necessary to develop a policy evaluation system for standardization. The development of policy evaluation methods for standardization is an issue that needs to be addressed [1][2]. This article briefly explains the issues and background of the system design of the policy evaluation for standardization activities in the second mid-term plan period of NEDO (New Energy and Industrial Technology Development Organization), for which I worked as a policymaker on the system design through a trial-and-error process (Note 1).

Since the methodology for policy evaluation concerning standardization has not been fully established, it is essential to accumulate knowledge through the study of theoretical aspects as well as through the accumulation of individual cases of implementation. For this reason, the case study presented in this article will be useful for both practitioners and academic researchers when designing evaluation systems for R&D institutions and national standardization activities. Simultaneously, the evaluation of standardization activities is an area where there is a general lack of knowledge worldwide[1][2]. I hope that the information presented here will be useful for innovation system reform around the world.

2. Overview of R&D Policy Evaluation System

2.1. Framework

Although NEDO, as a national research and development organization, has its own evaluation system for project evaluation, the specific system design is based on the evaluation system for national R&D projects (Note 2).

The first guideline for the evaluation of national R&D projects (policy evaluation and program evaluation), "General Guidelines for National R&D Evaluation," was established by the Prime Minister in 1997 [3]. The former Ministry of International Trade and Industry (now the Ministry of Economy, Trade and Industry) determined the evaluation guidelines for R&D projects and national R&D organizations under its jurisdiction at the same time as the primary guidelines were formulated [4] (Note 3).

The evaluation targets are broadly classified into two categories: (1) evaluation of individual R&D projects (ex-post evaluation that looks at the results on an R&D project basis) and (2) evaluation of the entire R&D organization (institutional evaluation that looks at the results of organizational R&D).

There are three necessary elements to policy evaluation: (1) data, (2) indicators, and (3) target values. (1) indicators and (2) target values must be meaningful and feasible in terms of related policy. Only indicators that have the right policy meaning and that are feasible are accepted as prerequisites for policy evaluation. The setting of wrong achievement targets has a negative impact on R&D activities. Therefore, in-depth understanding and insight into the data and policy goals used in the evaluation are essential for setting appropriate targets and target values.

2.2. Current status

NEDO has been promoting efforts to introduce quantitative evaluation indicators for evaluating the results of the standardization of research outcomes [5][6][7][8]. There have been academic discussions on evaluation methods for standardization activities [1][2]. However, there are still many issues to be addressed. In Japan's Fifth Science and Technology Basic Plan (FY2016-2020), only qualitative goals for standardization are stipulated, and no quantitative numerical goals are stipulated (based on research in 2019)[9].

3. Case Study of New Energy and Industrial Technology Development Organization (NEDO)

3.1. Background

The setting of targets for NEDO's R&D evaluation is based on its mid-term plan, which is formulated every five years. At this point, plans for the first (2003-2008), second (2008-2013), third (2013-2018), and fourth (2018-2023) terms have been formulated [5][6][7][8]. Before NEDO's second term (during the first mid-term plan), only "qualitative targets" were set to evaluate the results of standardization activities. This was because the design of a quantitative evaluation method for standardization was still a trial-and-error process.

Setting of Evaluation Indicators and Target Values for Standardization Activities at NEDO
Setting of Evaluation Indicators and Target Values for Standardization Activities at NEDO
Source: [5][6][7][8]
Note: The description of quantitative indicators is a concise explanation of the contents of the reference plan.

3.2 Issues to be solved

NEDO needed to design a mechanism to evaluate the degree of achievement of standardization results only for R&D results that are "suitable for standardization." In other words, it was necessary to prepare a system that does not forcibly require the standardization of R&D results that are not suitable for standardization.

If NEDO research projects only focused on the ICT field, evaluating almost all research results of the achievement of standardization might have been appropriate, but NEDO's research projects cover a wide range of fields (e.g., biotechnology and energy in addition to ICT). Not all R&D results necessarily require standardization for commercialization. In some cases, standardization may be an obstacle to the commercialization of research results.

Moreover, a point of concern was whether the number of successfully established standards should be used as a numerical target in evaluating the policy results. In the case of a standard, the standard development organization's consensus is usually required before the proposal is accepted as a standard. It is difficult to predict whether the consensus will be obtained or not. In addition, there is a good chance that the content will be changed through deliberation. For this reason, it was pointed out that using the number of approved standards as a numerical indicator would mean setting a target with a high degree of uncertainty.

3.3. Adopted evaluation design

In order to overcome the challenges, a quantitative evaluation system based on numerical targets separated into two distinct groups of plans was introduced (Note 4):
(1) The five-year medium-term plan does not specify numerical targets but only evaluation methods (evaluation indicators).
(2) Annual plans set numerical targets for evaluation indicators.

One feature of this system is that standardization activities can be monitored both on an annual basis and on a medium-term basis. This enables NEDO to set reasonable numerical targets only for research appropriate for standardization, taking into account the content of the R&D results before the start of each year.

In addition, as a target indicator, it was decided that the numerical target would be the "number of standards proposed" rather than the "number of standards established." This is because it is necessary to set a predictable and feasible target in order to realize the policy goal. The policy evaluation here is an assessment of achievement. Therefore, a goal without foreseeability is not an appropriate goal because it is difficult to estimate in advance the necessary amount of policy resources (funds and human resources) for achieving it (i.e., both can be estimated to require an infinite amount), making a comparative cost-benefit analysis impossible (Note 5).

3.4. Sharing of case studies within relevant ministries and agencies to promote standardization activities

In order to accumulate knowledge and best practices such as policy evaluation related to standardization, the "Relevant Governmental Agencies Liaison Meeting on International Standardization" (2007) was established to provide an opportunity to exchange information on the standardization policy initiatives of each ministry and agency. The secretariat of this liaison meeting was the Japanese Industrial Standards Committee (JISC) [10] (Note 6). Since standardization covers a wide range of products and services, multiple ministries and agencies from the Japanese government are involved. For this reason, efforts to both share and solve issues are beneficial. However, at the time of implementation, no appropriate mechanism for sharing information among ministries and agencies existed.

Like Japan, there are many countries where different ministries hold jurisdiction over different industries. Thus, Japan's efforts are considered to be a good international reference example.

4. Summary

In the evaluation of R&D policies and measures, it is essential to develop theories for the sophistication of the methods and also to develop concrete methods that can be implemented. In this sense, in order to ensure the implementation of a system that is appropriate to its purpose and allows for future improvements, background information on the issues and challenges behind evaluation systems that have been introduced is essential. For this reason, I introduced the issues and background behind the first introduction of numerical targets for standardization in the NEDO case. I hope this article will help researchers and policymakers in tackling related issues in the future.


This study was supported by JSPS Grants-in-Aid for Scientific Research (19K01827 PI: Suguru TAMURA).

December 18, 2020
  1. ^ Standards and Certification Unit, Ministry of Economy, Trade and Industry (and, as related divisions, Technology Evaluation and Research Division, Ministry of Economy, Trade and Industry, and Council for Science and Technology Policy, Cabinet Office [both at that time]).
  2. ^ Before NEDO became an incorporated administrative agency in 2003, NEDO was subject to evaluations conducted for R&D policies by the national government (Ministry of Economy, Trade and Industry (METI)) (METI's division in charge was the Technology Evaluation and Research Division at the time). This is the fundamental reason why NEDO's R&D evaluation system has the same implementation framework as the policy evaluation system conducted by the national government. NEDO was subsequently modified from an incorporated administrative agency to a national research and development agency in 2015.
  3. ^ The legal framework for implementing the evaluation of national projects is based on the Government Policy Evaluations Act.
  4. ^ In detail, in addition to the number of proposals, the number of basic plans in which standardization efforts are incorporated was introduced as a numerical indicator from the second plan. The latter is not explained in this article because the latter is not an outcome target that is used to directly evaluate the results of R&D.
  5. ^ It is essential to understand that this is an evaluation of R&D results. By their very nature, research and development activities are activities for which the uncertainty of achieving results is very high. For this reason, an evaluation system for R&D activities has been established specially by the "General Guidelines for National R&D Evaluation" and is operated by the Council for Science, Technology and Innovation.
  6. ^ There are details on this matter in the White Paper on Science and Technology (2008, p.104).
  7. Citation method of this article's contents: Tamura, S. (2021). R&D policy evaluation system at NEDO, a national R&D agency (An example of the introduction of quantitative evaluation indicators for technology standards), RIETI Column.
  8. The contents of this article correspond to the policy contents of Chapter 5 (3) "Strategic Use of International Intellectual Property and Standardization" and Chapter 7 (2) "Reforming National R&D Institutes and Enhancing their Function" of the Fifth Science and Technology Basic Plan (FY2016-2020).
  9. Contact.
  • [1] Edler, J., Georghiou, L., Blind, K., and Uyarra, E. (2012). Evaluating the demand side: New challenges for evaluation. Research Evaluation, 21: 33-47.
  • [2] Tassey, G. (2003). Method for Assessing the Economic Impacts of Government R&D. Gaithersburg, MD: National Institute of Standards & Technology.
  • [3] Cabinet Office (1997). General Guidelines for National R&D Evaluation.
  • [4] Ministry of International Trade and Industry (1997). Technology Evaluation Guidelines.
  • [5] New Energy and Industrial Technology Development Organization (2003). The First Medium-term Plan of NEDO.
  • [6] New Energy and Industrial Technology Development Organization (2008). NEDO 2nd Mid-term Plan.
  • [7] New Energy and Industrial Technology Development Organization (2013). NEDO 3rd Mid-term Plan.
  • [8] New Energy and Industrial Technology Development Organization (2018). NEDO Fourth Medium- and Long-Term Plan.
  • [9] Ministry of Education, Culture, Sports, Science and Technology (2016). Science and Technology Basic Plan.
  • [10] Ministry of Education, Culture, Sports, Science and Technology (2008, p.104). White Paper on Science and Technology.

February 19, 2021