Agenzia Nazionale di Valutazione del Sistema Universitario e della Ricerca

Ricerca sulla valutazione

In questa sezione si elencano le pubblicazioni di ricerca inerenti i temi della valutazione e delle università curate da dipendenti e collaboratori dell'Agenzia.


Abramo G, Cicero T and D'Angelo C 
(2014). Journal of Informetrics
DOI: 10.1016/j.joi.2013.10.011
Ever more frequently, governments have decided to implement policy measures intended to foster and reward excellence in scientific research. This is in fact the intended purpose of national research assessment exercises. These are typically based on the analysis of the quality of the best research products; however, a different approach to analysis and intervention is based on the measure of productivity of the individual scientists, meaning the overall impact of their entire scientific production over the period under observation. This work analyzes the convergence of the two approaches, asking if and to what measure the most productive scientists achieve highly cited articles; or vice versa, what share of highly cited articles is achieved by scientists that are “non-top” for productivity. To do this we use bibliometric indicators, applied to the 2004–2008 publications authored by academics of Italian universities and indexed in the Web of Science.



Abramo G, Cicero T and D'Angelo CA 
(2015). Journal of Informetrics
DOI: 10.1016/j.joi.2014.11.002
The literature on gender differences in research performance seems to suggest a gap between men and women, where the former outperform the latter. Whether one agrees with the different factors proposed to explain the phenomenon, it is worthwhile to verify if comparing the performance within each gender, rather than without distinction, gives significantly different ranking lists. If there were some structural factor that determined a penalty in performance of female researchers compared to their male peers, then under conditions of equal capacities of men and women, any comparative evaluations of individual performance that fail to account for gender differences would lead to distortion of the judgments in favor of men. In this work we measure the extent of differences in rank between the two methods of comparing performance in each field of the hard sciences: for professors in the Italian university system, we compare the distributions of research performance for men and women and subsequently the ranking lists with and without distinction by gender. The results are of interest for the optimization of efficient selection in formulation of recruitment, career advancement and incentive schemes.



Ancaiani A, Anfossi AF, Barbara A, Benedetto S, Blasi B, Carletti V, Cicero T, Ciolfi A, Costa F, Colizza G, Costantini M, di Cristina F, Ferrara A, Lacatena RM, Malgarini M, Mazzotta I, Nappi CA, Romagnosi S and Sileoni S 
(2015). Research Evaluation 24(3): 242-255
DOI: 10.1093/reseval/rvv008
The Italian Research Evaluation assessment for the period 2004-10 (VQR 2004-10) has analyzed almost 185,000 articles, books, patents, and other scientific outcomes submitted for evaluation by Italian universities and other public research bodies. This article describes the main features of this exercise, introducing its legal framework and the criteria used for evaluation. The innovative methodology that has been used for evaluation, based on a combination of peer review and bibliometric methods, is discussed and indicators for assessing the quality of participating research bodies are derived accordingly. The article also presents the main results obtained at the University level, trying to understand the existing relationship among research quality and University characteristics such as location, dimension, age, scientific specialization, and funding.



Ancaiani A and Fantoni S 
(2014). Melanges De La Casa De Velazquez 44(2): 313-318



Anfossi A, Ciolfi A, Costa F, Parisi G and Benedetto S 
"Large-scale assessment of research outputs through a weighted combination of bibliometric indicators"
(2016). Scientometrics
Link: In press
The paper describes a method to combine the information on the number of citations and the relevance of the publishing journal (as measured by the Impact Factor or similar impact indicators) of a publication to rank it with respect to the world scientific production in the specific subfield. The linear or non-linear combination of the two indicators is represented on the scatter plot of the papers in the specific subfield in order to immediately visualize the effect of a change in weights. The final rank of the papers is therefore obtained by partitioning the two-dimensional space through linear or higher order curves. The procedure is intuitive and versatile since it allows, after adjusting few parameters, an automatic and calibrated assessment at the level of the subfield. The derived evaluation is homogeneous among different scientific domains and can be used to address the quality of research at the departmental (or higher) levels of aggregation. We apply this method, that is designed to be feasible on a scale typical of a national evaluation exercise and to be effective in terms of cost and time, to some instances of the Thomson Reuters Web of Science database and discuss the results in view of what was done recently in Italy for the Evaluation of Research Quality exercise 2004-2010. We show how the main limitations of the bibliometric methodology used in that context can be easily overcome.



Bertocchi G, Gambardella A, Jappelli T, Nappi CA and Peracchi F 
(2015). Research Policy 44(2): 451-466
DOI: 10.1016/j.respol.2014.08.004
 A relevant question for the organization of large-scale research assessments is whether bibliometric evaluation and informed peer review yield similar results. In this paper, we draw on the experience of the panel that evaluated Italian research in Economics, Management and Statistics during the national assessment exercise (VQR) relative to the period 2004-2010. We exploit the unique opportunity of studying a sample of 590 journal articles randomly drawn from a population of 5681 journal articles (out of nearly 12,000 journal and non-journal publications), which the panel evaluated both by bibliometric analysis and by informed peer review. In the total sample we find fair to good agreement between informed peer review and bibliometric analysis and absence of statistical bias between the two. We then discuss the nature, implications, and limitations of this correlation. (C) 2014 Elsevier B.V. All rights reserved.



Blasi B 
DOI: 10.3280/SP2015-002002 


Bonaccorsi A and Cicero T 
(2015). Journal of the Association for Information Science and Technology
DOI: 10.1002/asi.23539



Bonaccorsi A and Cicero T 
"Nondeterministic ranking of departments and universities"
(2016). Journal of Informetrics
Link: In press



Bonaccorsi A, Cicero T, Ferrara A and Malgarini M 
(2015). F1000Res 4: 196
DOI: 10.12688/f1000research.6478.1
 The aim of this paper is to understand whether the probability of receiving positive peer reviews is influenced by having published in an independently assessed, high-ranking journal: we eventually interpret a positive relationship among peer evaluation and journal ranking as evidence that journal ratings are good predictors of article quality. The analysis is based on a large dataset of over 11,500 research articles published in Italy in the period 2004-2010 in the areas of architecture, arts and humanities, history and philosophy, law, sociology and political sciences. These articles received a score by a large number of externally appointed referees in the context of the Italian research assessment exercise (VQR); similarly, journal scores were assigned in a panel-based independent assessment, which involved all academic journals in which Italian scholars have published, carried out under a different procedure. The score of an article is compared with that of the journal it is published in: more specifically, we first estimate an ordered probit model, assessing the probability for a paper of receiving a higher score, the higher the score of the journal; in a second step, we concentrate on the top papers, evaluating the probability of a paper receiving an excellent score having been published in a top-rated journal. In doing so, we control for a number of characteristics of the paper and its author, including the language of publication, the scientific field and its size, the age of the author and the academic status. We add to the literature on journal classification by providing for the first time a large scale test of the robustness of expert-based classification.



Bonaccorsi A, Haddaway P, Hassan S, Cicero T, Secondi L and Setteducati E 
(2013). Policy Breaf of the European Commission. Innovation 4 Growth(10)


Bonaccorsi A, Secondi L, Ancaiani A and Setteducati E 
(2011). 2011 4th International Conference of Education, Research and Innovation (Iceri): 1566-1576
 In the last years several changes have occurred in the university research environment concerning both the funding method and the evaluation of research performance. On one hand, universities have significantly increased the recruitment of research funds from external sources - in addition to the basic research mainly funded by government and institutions' core budget - thus realizing a basis for the interaction between science and industry and for the creation of social and economic returns. On the other hand, the modification in the research funding system lead to consider indicators based on financial data in addition to the classical indicators for evaluating the research performance. However, the use of funding data - including third-party amount of funding - as suitable indicators to describe research performance of university departments and therefore for evaluating the scientific capacity of a university is still a debated issue due to the differences among fields of study and the non-profit nature of universities. In Italy, the newly established National Agency for the Evaluation of Universities and Research Institutes is called to define performance indicators, concerning both the education process and the research activities, to be used for distributing funds to the universities on the basis of their performance. In this context, the increasing attitude of universities to collaborate with external partners and therefore the activation of the wider process of knowledge transfer are two major aspects to carefully and properly consider in the framework of research activities evaluation. This study has a twofold aim. Firstly, after a brief analysis of the evolution of funding sources of research activities in Italy, we will aim to shed light on the definition, the importance and the extent of third party funding in the Italian system of research funding. Secondly, by applying suitable statistical models we will empirically investigate the factors that characterize the departments making research on behalf of third parties as well as the significant features in raising research funding from external sources. The analysis include all Italian university departments and it will be based on a unique data set obtained by combining different sources of data - collected early by the Italian Ministry of Education University and Research - which include information concerning both the structure, also in terms of academic staff, the field of study of the department and the economic context in which each institution operates. Although discussion in the paper focuses on the evaluation process and the use of results obtained as a basis for the definition of a global research evaluation framework, the analysis carried out can also represent an important tool for each university institution regarding the internal processes of decision making and monitoring.



Bonaccorsi A, Secondi L, Setteducati E and Ancaiani A 
(2014). Journal of Technology Transfer 39(2): 169-198
DOI: 10.1007/s10961-012-9268-5
Over the last few years, the emergence of universities' third mission has significantly affected objectives, sources of funding and financing methods, as well as the management, of universities. Although the university-industry relationships have been widely investigated, several interesting theoretical and empirical issues still remain open in the literature. In this paper we construct an original data set, combining financial information with structural and organizational data on Italian University departments, with a twofold aim. First, to describe the importance and the extent of third-party funding in the Italian system of research as well as the pattern of evolution over the last few years. Second, to investigate the factors that influence both the probability and the intensity of the commitment of departments in third-party activities by building a multi-level framework combining factors at individual, departmental, university and territorial levels. The results obtained suggest a number of policy implications for universities and policy makers. On one hand, universities should explicitly recognize the role of dedicated internal organizations and provide training for professional staff capable of acting as value-added intermediaries. On the other hand, if policy makers wish to improve the relationships between universities and external actors, disciplinary differences across departments as well as regional inequalities in growth levels should be carefully considered, giving up a one-size-fits-all approach.



Cicero T, Malgarini M, Nappi CA and Peracchi F 
(2013). MPRA (Munich Personal REPEc Archive)(50470)



Cicero T, Malgarini M and Benedetto S 
(2014). Proceedings of Science and Technology Indicators Conference 2014



Di Cristina F 
(2013). Munus 2: 42



Ferrara A and Bonaccorsi A 
(2016). Research Evaluation
DOI: 10.1093/reseval/rvv048
This article reports on a large-scale exercise of classification of journals in the fields of Humanities and Social Sciences, carried out by the Italian Agency for the Evaluation of Universities and Research Institutes. After discussing at some length the controversies linked with journal classification and its impact, we endeavor to compare such a classification with the scores that individual articles published in the same journals were assigned by completely independent assessors in the same period of time. The data refer to an important subset of disciplines covering History, Philosophy, Geography, Anthropology, Education, and Library Sciences, allowing for comparisons between scientific fields of different sizes, outlooks, and methods. As the controversies surrounding the rating of journals focus on the difference between the container (the journal) and the content (the individual article), we addressed the following research questions: (1) Is journal rating, produced by an expert-based procedure, a good predictor of the quality of articles published in the journal? (2) To what extent different panel of experts evaluating the same journals produce consistent ratings? (3) To what extent the assessment of scientific societies on journal rating is a good predictor of the quality of articles published in journals? (4) Are there systematic biases in the peer review of articles and in the expert-based journal rating? We find that journal rating is a legitimate and robust assessment exercise, as long as it is managed carefully and in a cautious way and used to evaluate aggregates of researchers rather than individual researchers.



Malgarini M, Nappi CA and Torrini R 
(2015). Proceedings of ISSI Conference



Montanaro P and Torrini R 
(2014). Bank of Italy Occasional Paper 219
DOI: 10.2139/ssrn.2489908
 Nel settore della ricerca pubblica l'Italia investe meno della media europea. Se rapportato alle risorse impegnate e ai ricercatori, l'output risulta però elevato e la sua qualità media, condotta presso università ed enti di ricerca, non è molto lontana rispetto a paesi prossimi come la Francia, anche se con difficoltà di affermazione nelle punte più avanzate. Il sistema italiano, assai articolato e frammentato nei soggetti che vi operano e nelle fonti di finanziamento, risente di una scarsa attitudine all'applicazione dei risultati e alla collaborazione con le imprese, che a loro volta investono poco e incontrano difficoltà a collegare la propria attività di ricerca con gli input provenienti dai centri di ricerca pubblica. Il sistema sconta inoltre la mancanza di una chiara strategia che stabilisca gli obiettivi da raggiungere, disegni missioni e modelli organizzativi delle strutture di ricerca coerenti con gli obiettivi individuati e definisca le risorse necessarie al loro raggiungimento. La pressante necessità di un rilancio della capacità innovativa del Paese, infatti, non può prescindere da un sistema della ricerca pubblica adeguatamente finanziato ed efficientemente governato.



Nappi CA and Poggi G 
(2015). Rassegna Italiana di Valutazione 59