Composite indicators play an essential role for benchmarking higher education institutions. One of the main sources of uncertainty building composite indicators and, undoubtedly, the most debated problem in building composite indicators is the weighting schemes (assigning weights to the simple indicators or subindicators) together with the aggregation schemes (final composite indicator formula). Except the ideal situation where weights are provided by the theory, there clearly is a need for improving quality assessment of the final rank linked with a fixed vector of weights. We propose to use simulation techniques to generate random perturbations around any initial vector of weights to obtain robust and reliable ranks allowing to rank universities in a range bracket. The proposed methodology is general enough to be applied no matter the weighting scheme used for the composite indicator. The immediate benefit achieved is a reduction of the uncertainty associated with the assessment of a specific rank which is not representative of the real performance of the university, and an improvement of the quality assessment of composite indicators used to rank. To illustrate the proposed methodology we rank the French and the German universities involved in their respective 2008 Excellence Initiatives.
Aguillo, I, Bar-Ilan, J, Levene, M, Ortega, JL2010Comparing university rankings. Scientometrics85:243–256.
Bastedo, M., & Bowman, N. (2010). College rankings as an interorganizational dependency: Establishing the foundation for strategic and institutional accounts. Research in Higher Education. doi: 10.1007/s11162-010-9185-0.)| false
Centre for Higher Education Development (CHE), UNESCO European Centre for Higher Education (CEPES), Institute for Higher Education Policy (IHEP). (2006). Berlin principles on ranking of higher education institutions, Berlin. Available online at: http://www.che.de/downloads/Berlin/Principles/IREG/TEXTsymbol/534.pdf.)| false
Dill, D, Soo, M2004Is there a global definition of academic quality? A cross-national analysis of university ranking system. Public Policy for Academic QualityUniversity of North CarolinaChapel Hill, NC.
Dill, D, Soo, M2004Is there a global definition of academic quality? A cross-national analysis of university ranking system. Public Policy for Academic QualityUniversity of North CarolinaChapel Hill, NC.)| false
Ding, J., & Qiu, J. (2011). An approach to improve the indicator weights of scientific and technological competitiveness evaluation of Chinese universities. Scientometrics, 86, 285–297. doi: 10.1007/s11192-010-0268-7.)| false
Saisana, M, Saltelli, A, Tarantola, S2005Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Association A168:307–323.
Saisana, M, Saltelli, A, Tarantola, S2005Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Association A168:307–32310.1111/j.1467-985X.2005.00350.x.)| false