Our latest Global Research Report from the Institute for Scientific Information (ISI)™, Unpacking research profiles: Moving beyond metrics, is our second report to guide policy makers and research managers on the value of shifting from a reliance on simplistic metrics of research activity and performance towards an adoption of more visually informative profiles. Report co-author Jonathan Adams explains how these richer, visual profiles can help interpret research and enable better policy and management decision making.
Single-figure metrics such as researcher h-index and average citation impact are often used by time-limited research policy analysts and research managers, despite the availability of data visualizations that better describe the underlying research activity – such as Impact Profiles. Overreliance on simple metrics can mean that users miss essential information and real evidence about research achievements and trajectories that aid interpretation, explain unexpected results and guide future investment.
How profiles reveal the ‘bigger picture’ in research
A key to appreciating the need for rich, visual profiles, rather than simple metrics alone, is to be aware of and acknowledge the underlying statistical distribution of the data.
For example, research activity data are highly skewed with many low values and a few exceptionally high values; the average is typically much higher than the median (or central point). Only a profile of the data will reveal more – the small number of people with relatively high-income, large research groups, prolific publications and highly cited papers.
An exploration of visual data will give invaluable context and enable the reader to avoid falling into the trap laid by the option to rely on simplistic, surface-level metrics which are often used out of their proper context. In most cases, this step is critical to deepen our understanding and aid in making the right decisions.
Developing a holistic view
Drawing on Web of Science™ data, this new report focuses on four key indicators at researcher, journal, institutional and research field levels that can be used to broaden and deepen our perspective when it comes to research management:
- Individuals and their publications
Some level of self-citation is perfectly normal; we consider what is excessive. The issue of excessive self-citation is a growing concern, as suspect publications appear to be proliferating and the validity of research publication statistics is under threat. Since citation rate norms vary between fields, our analysis reveals a characteristic pattern to the spread of self-citation observed among Highly Cited Researchers™, which we can model to identify exceptional ‘outliers’ where self-citing is excessive – while, importantly, accounting for varying citation rates between fields.
- Journals and their characteristics
There are an increasingly wide range of descriptive profiles now available for the more than 21,000 journals indexed in the Web of Science, expanding the information in the annual Journal Citation Reports™ beyond the well-established Journal Impact Factor™. In this report we explore the indicator of national orientation (INO) which can be calculated using the addresses of researchers publishing in and those citing a journal. This indicator offers new perspectives on the role, content and significance of journals, and better informs researchers about the optimal venues for their papers.
- Influence of international collaboration
A single ‘average’ metric obscures proper comparison of the impact of international collaboration and hides the influence of well-cited internationally co-authored papers. We show how Collaborative Citation Impact (Collab-CNCI) can be deconstructed, revealing where impact comes from to enable better interpretation and evaluation of domestic and international activity.
- Thinking forward with Research Fronts
Most bibliometric data are retrospective, but research planners need to know what is currently grabbing attention. Recent papers frequently cited by the same new papers cluster as a Research Front™ and link across a landscape of research domains. Our 2020 Global Research Report, Identifying Research Fronts in the Web of Science, describes the history and development of Research Front technology. With Research Fronts, we can get close to the edge of current research. In doing so, we identify the topics, institutions and researchers making a mark now, thereby better supporting resource management and policy decisions.
The underlying concept has been rigorously tested and proven by both the Chinese Academy of Sciences and the Japanese Science and Technology Agency which analyze Web of Science data to underpin their research policy and planning. In this report, we outline the different approaches that we are considering for visual presentations of Research Fronts.
Advancing research excellence through evidence-informed strategies
Visualization of the publication and citation data ‘unpacks’ the information that is lost when a single-point metric is used as a summary. It can not only reveal new insights but also surface critical additional questions and thereby aid in next steps for research planning and investment. Visualization is not always easy to understand, but it avoids the pitfalls of a simple metric used in the wrong context.
At the Institute for Scientific Information, we are actively working with our customers and partners in academia and government on identifying and developing the presentation, indicators, tables and graphics that will best address their needs. Working together, we deliver trusted content and transformative intelligence to enable them to implement evidence-informed strategies and advance research excellence.
Learn more about our ISI Global Research Report: Unpacking research profiles: Moving beyond metrics
Or check out our other reports in this series here.
