Does Order Distort Evaluation? Evidence from observational data and field experiment in a large-scale piano competition

This study investigates why “serial order effects”—systematic distortions in judgment caused by the presentation order of candidates—arise in sequential expert evaluations, and whether such effects can be mitigated. Focusing on Japan’s largest piano competition, we combine long-term observational data from 2004–2022 with data from a cluster-randomized controlled trial (RCT) conducted in 2023. The result drawn from the observational data reveals a robust serial order effect: performers who appear earlier in the sequence consistently receive lower scores. This disadvantage is particularly pronounced in competitions with fewer participants and with highly skilled performers. These patterns are difficult to reconcile with fatigue-based explanations and instead suggest a calibration mechanism, whereby judges—motivated to maintain internal consistency—avoid extreme evaluations in the early stages when the overall distribution of performance quality is still uncertain, thereby suppressing initial scores. We then run a RCT to evaluate an information treatment that explicitly informs judges about the existence and magnitude of serial order effects using historical data. The results show no strong evidence that the intervention significantly mitigates order effects overall. However, we find suggestive evidence that the serial order effect is partially mitigated for highly skilled performers. The results suggest the need for institutional or procedural solutions to address order effects in sequential evaluation settings.