Although many researchers have investigated Japan's so-called lost decades (1990s and 2000s), a real puzzle is why productivity growth has slowed down since the 1990s for such a long time with just a weak recovery. To solve it, much research has been done using micro-level data on firm and establishment activity, which sheds great light on our understanding of the mechanism of the productivity dynamics in the long-lasting Japan's lost decades. However, most of the research only uses the data on medium-sized or large firms. Especially, research on productivity dynamics in the non-manufacturing industry is extremely limited. Against these backgrounds, this paper analyzes the productivity dynamics among the small and medium-sized enterprises (SMEs). Our analysis covers non-manufacturing firms as well as manufacturing, which makes this research unique and contributes to the literature of economic growth. The dataset used for the analysis is a unique, large, and comprehensive firm level panel dataset of the Credit Risk Database. We find that the productivity dynamics among SMEs is very different from that of large firms. In contrast to the case of large firms, reallocation effect, not within effect, is the main driver of the productivity dynamics among SMEs, implying that market mechanism is properly working among them. However, the shutting down of a small number of highly productive firms drives most of the negative exit effect, which implies another malfunctioning aspect of the market mechanism among SMEs.