More on University Rankings – a paper entitled ‘Should you believe in the Shanghai ranking?’
There have been a few posts here discussing university quality, and especially the various ranking systems available that attempt to measure ‘quality’. Here. via Shanghai_JCB_DB_PV.pdf (application/pdf Object), is a paper which asks the question of whether or not one should believe the in JT method of ranking universities. Below the abstract is a startling quotation on the institutional attribution of the location of the Nobel Prize, and the effect this has on weightings.
Should you believe in the Shanghai ranking? An MCDM view
Jean-Charles Billaut, Denis Bouyssou, Philippe Vincke
This paper proposes a critical analysis of the “Academic Ranking of World Universities”, published every year by the Institute of Higher Education of the Jiao Tong University in Shanghai and more commonly known as the Shanghai ranking. After having recalled how the ranking is built, we first discuss the relevance of the criteria and then analyze the proposed aggregation method. Our analysis uses tools and concepts from Multiple Criteria Decision Making (MCDM). Our main conclusions are that the criteria that are used are not relevant, that the aggregation methodology is plagued by a number of major problems and that the whole exercise suffers from an insufficient attention paid to fundamental structuring issues. Hence, our view is that the Shanghai ranking, in spite of the media coverage it receives, does not qualify as a useful and pertinent tool to discuss the “quality” of academic institutions, let alone to guide the choice of students and family or to promote reforms of higher education systems. We outline the type of work that should be undertaken to offer sound alternatives to the Shanghai ranking.
Attribution of the Nobel Prize to Institutions
(pp. 10-12) Table 3 lists some French Nobel prize winners together with their affliation as indicated by the official Nobel prize web site (http://nobelprize.org/nobel_prizes/). A careful examination of this table reveals many interesting facts. First there are, in this list, institutions that have never existed (except maybe, in medieval times). Sorbonne University, whether or not translated into French, has never existed as such and surely does not presently exist (although a building called Sorbonne does exist in Paris). A second problem lies in multiple affliations. To which institution should go the Nobel Prize of Louis de Broglie. It should go either to Sorbonne University (but we have just seen there is no such institution) or to the Institut Henri Poincare that, for sure, is not a university. There are also many cases in which the Nobel prize is attributed to the right institution, but that institution does not exist any more (Strasbourg University, Nancy University, Toulouse University, Universitte de Paris, University of Grenoble, Ecole municipale de physique et de chimie industrielle). Another intriguing case is the Nobel prize of Georges Charpak who has two affliations ESPC and CERN. Since CERN is not a university, should all the prize go the ESPC or only half of it? (note that ESPC is the new name of the Ecole municipale de physique et de chimie industrielle). This shows, for the case of France, that the correct attribution of each prize requires a very good knowledge of the institutional landscape in each country and to take many “micro-decisions”. Since the authors of the Shanghai ranking do not document any of these micro-decisions, the least that can be said is that these two counting criteria are at best, orders of magnitudes. (emphasis added in italics)
The paper is worth reading, particularly for non-specialists in the ranking and measurement business. And the conclusion? I guess economists shouldn’t believe everything they read!