Abstract: With the institutionalized development of science and technology (S&T) incentives and rolling out of major awards in China, the fairness and validity of the selection of such awards are of increasing concern for various communities. Eyeing the customary approach of peer review in the selection process, the research questions are: (1) When multiple experts need to score or rank multiple nominees to select the winners, to what extent can the peer review results reflect the real differences of the evaluands? (2) For the judgments of experts of the same group, can we get the same shortlist when scoring or sorting methods are used? If the selection processes of science and technology awards can be regarded as experiments, the scoring results to be examined are the test data. In this way, we can build a two factor ANOVA model to examine the uncontrollable factors such as the level of achievements, expert preferences, and the controllable factors of selection criteria, decision-making rules, and so on. Thus the study builds an evaluation model of reliability based on two-way ANOVA, to conduct a sample analysis of the ratio data and ordinal data generated respectively by anonymous online appraisal and assessment of expert panels in four different disciplines between year 2017 and 2019. The result shows that, summarizing expert scoring data and determining the winners based on the total score of the projects will produce large systematic errors. If the shortlisting is based on majority voting rule, the average reliability of the expert group’s evaluation is only 28.6%, with maximum value of 60% and minimal value of 6.5%, which means the selection of winner depends more on other factors than the project itself. Thus in the previous years, among the 115 award winners, at least 8 winners should have been eliminated. To solve the problem, the study carries on a quantitative evaluation and comparison regarding the selection of awards between different years, disciplines, phases of selection, and methods of selection, suggests that systematic errors can be reduced by methods of rank sum or independent voting of experts to increase the reliability of peer review. Besides, the study also takes reference from the features and latest development of renowned international S&T awards to examine the organization of selection, evaluation indicators, and decision-making procedures of domestic awards, discusses the possibility of canceling oral defense, adopting anonymous network evaluation instead of panel meeting.