I recently read on LinkedIn how a department head bragged about how many papers his team published in highly ranked journals over the past year. This mentality has to stop because it does not lead to more, but to less quality. In fact, quantity competes with quality. Often “motivated” by the disincentive systems of their universities, many academics waste their time writing lots of papers that no one ever will cite. They should rather invest this time in writing one great paper. This can take years, but is worth doing. For example, Mark Granovetter is a highly acclaimed academic who some consider worthy of the Nobel Prize in Economics. Google Scholar counts almost 150,000 citations of his work. Yet, 70% and 84% of these citations refer to just 2 and 5 of his great papers, respectively. So if you ever sit on an assessment committee or make bonus decisions, do not just count the number of publications in certain journals per year. You might then overlook academic leaders like Granovetter.
Claviate Analytics have recently published their newest InCites Journal Citation Reports. It is great to see that the 2018 impact factors of all but one journals related to supply chain management have increased again, which highlights the rapidly growing relevance of our discipline. Two journals have an impact factor larger than 7: Journal of Operations Management (7.776; +2.9) and Journal of Supply Chain Management (7.125; +1.0). With an impact factor larger than 5, International Journal of Physical Distribution & Logistics Management (5.212; +1.0) has now arrived in the first league of management journals. Other SCM-related journals with high impact factors are: Supply Chain Management: An International Journal (4.296; +0.5), Management Science (4.219; +0.7), International Journal of Operations & Production Management (4.111; +1.2), Journal of Business Logistics (3.171; +0.3) and Journal of Purchasing & Supply Management (3.089; −0.6). But there are even more SCM journals with an impact factor around 2: Manufacturing & Service Operations Management (2.667; +0.9), Operations Research (2.604; +0.3), International Journal of Logistics Management (2.226; +0.5), Production and Operations Management (2.171; +0.4) and Decision Sciences (1.960; +0.3). Although the impact factor is certainly an imperfect proxy of a journal’s quality, I can only hope that rather conservative qualitative rankings, such as the ABS-AJG list, the UT Dallas list or the FT50 list, will finally be adapted to this new reality. This step is urgently needed!
Note: The following text refers to a previous version of the JCR impact factors. A more recent version are the 2018 JCR impact factors (see there).
More than many other management disciplines, SCM has been very successful to professionalize and reinvent itself. A good indication for this development are the 2017 JCR journal impact factors, which have just been released. Many, although not all, impact factors of SCM journals have improved. The journal with the highest impact factor among SCM journals and the seventh-highest one among more than 200 management journals is Journal of Supply Chain Management (6.105). Three SCM journals have impact factors between 4 and 5: Journal of Operations Management (4.899), International Journal of Production Economics (4.407) and International Journal of Physical Distribution & Logistics Management (4.215). IJPDLM is now even the Emerald journal with the highest impact factor, which is a great achievement! Two other SCM journals range between 3 and 4: Supply Chain Management: An International Journal (3.833) and Journal of Purchasing & Supply Management (3.667), but International Journal of Operations & Production Management (2.955) and Journal of Business Logistics (2.891) also come close to 3. Both International Journal of Logistics Management (1.776) and Decision Sciences (1.641) were able to improve their impact factors and get closer to 2. It might take some time until our journals will finally be acknowledged by qualitative rankings such as CABS’s AJG and VHB-JOURQUAL, as such rankings tend to be quite conservative. However, with such high impact factors there should be no doubt anymore that SCM plays in the same league as accounting, marketing and finance.
Three years ago, the Chartered Association of Business Schools (ABS) has released its last ranking of business journals: the Academic Journal Guide (AJG), also known as the “ABS list”. This ABS ranking has become quite influential as the guiding journal ranking across management disciplines in the UK. Although the ranking has been heavily criticized (see my previous post) and more democratically-developed rankings exist (e.g. VHB-JOURQUAL), the ABS ranking has since been adopted by many business schools also in other countries.
Shortly after the publication of the last ABS list, Nature has published ten principles to guide research evaluation, which have since become known as the Leiden Manifesto. One of these principles argues that quantitative evaluation should support qualitative, expert assessment, not replace it. Similarly, the San Francisco Declaration on Research Assessment (DORA), which has been signed by thousands of researchers worldwide, asks “not [to] use journal-based metrics […] as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”. The Community for Responsible Research in Business Management (cRRBM) questions whether “even the academy is being served when faculty members are valued for the quantity and placement of their articles, not for the benefit their research can have for the world”. Indeed, the negative consequences of rankings are well-documented (see, for example, Espeland & Sauder, 2007 and Grant & Kovács, 2018).
Unfortunately, the ABS list has often been used in exactly the way the Leiden Manifesto and DORA wanted to prevent us from doing – with very negative consequences for our discipline. Many SCM researchers feel that the ABS list undervalues the journals of our discipline. For example, Journal of Supply Chain Management, a journal with one of the highest impact factors in management (5.789), is ranked by ABS 2015 as a “3” only and Journal of Business Logistics as a “2”. Comparing our journals to other disciplines, like accounting or marketing, it becomes apparent to me that these journals deserve a “4*” and “4”. Worse even, many SCM journals have suffered from their low ABS rankings, as SCM researchers who took the ABS ranking too seriously, felt they should publish in the higher-ranked journals of other disciplines. AJG 2015 has thus become a self-fulfilling prophecy.
Today, the Academic Journal Guide 2018 has been released. It contains more than 1,500 entries. Unfortunately, AJG 2018 did not adapt the ranks of leading SCM journals. When it comes to empirical journals, only Journal of Operations Management gets a “4*”, while Decision Sciences, Journal of Supply Chain Management and Supply Chain Management: An International Journal get a “3” again. International Journal of Physical Distribution and Logistics Management, Journal of Business Logistics and Journal of Purchasing and Supply Management all still get a “2” only. The fact that the rank of none of these journals has changed indicates that the new ranking does not seem to cover all the changes and improvements our discipline has made in the last couple of years. Only four out of eight journals (four empirically- and analytically-focused journals each) that have been identified by leading SCM schools as the top journals of our discipline (sometimes informally referred to as “the SCM basket of eight”; see The SCM Journal List), got an AJG rank higher than 3. My suggestion would be to ignore AJG 2018, as it does not seem to represent the journals of our discipline properly.
Interestingly, even the Chartered ABS itself just tweeted that the REF2021 sub-panel will ignore their own list, so they might well be aware of its shortcomings:
#REF2021 sub-panel chair: “We will not be using metrics. Our role is to assess originality, significance and rigour. We will not use #AJG2018 to do so.” Robert Blackburn #ARC2018
But there is some hope: This latest iteration of AJG was an interim review, the main purpose being to include new journals. The next iteration, planned to be published in 2021, will then be a major review. This is certainly very late, but better late than never: Careers can depend on such lists. Therefore, our discipline urgently deserves a better quality of the AJG. I hope the team behind the list is aware of their responsibility.
My guest post today comes from Alan McKinnon who for several years has been raising concerns about the academic obsession with journal rankings and low rating of logistics/SCM journals. He has just published a new paper updating his earlier arguments.
In a paper that I wrote five years ago I argued that the development of logistics/supply chain management (SCM) as a discipline was being impaired by the relatively low ranking of specialist journals in this field. I was surprised and heartened by the favourable response I received both from logistics/SCM researchers and academics in other disciplines experiencing a similar problem. I have now returned to the journal ranking debate with a sequel to my original article which reviews recent literature on the subject, analyses new data on the validity of the journal ranking as an indicator of research quality and discusses the recalibration of logistics/SCM journals since 2010/11. The literature challenging the principle, practice and application of journal ranking has been steadily expanding and becoming more critical. Regrettably this is not deterring university managers from basing many recruitment, promotional and resource allocation decisions on the rating of journals. Data generated by the UK government’s assessment of university research (REF) has confirmed that, in the field of business and management, the journal ranking is an unreliable predictor of the quality and impact of an individual journal paper. In this analysis, papers published in lower ranked journals tended to be under-valued, a finding of particular relevance to logistics/SCM journals as they tend to be on the 2nd or 3rd tiers of the major journal lists. Since 2010/11, there has been some overall improvement in the relative standing of these journals, though a couple have been downgraded in the widely-used ABS list. Fortunately the backlash against journal rank “fetishism” has begun with bottom-up campaigns such as DORA and top-down, government-led initiatives in countries such as the UK and Australia aiming to make research assessment fairer, more transparent and more rigorous.
Alan McKinnon is Professor of Logistics in Kühne Logistics University, Hamburg and Professor Emeritus at Heriot-Watt University, Edinburgh. You can find out more about his research and publications at www.alanmckinnon.co.uk and follow him on Twitter @alancmckinnon.
McKinnon, A.C. (2017). Starry-eyed II: The Logistics Journal Ranking Debate Revisited. International Journal of Physical Distribution & Logistics Management, 47 (6). DOI: 10.1108/IJPDLM-02-2017-0097
Note: The following text refers to a previous version of the JCR impact factors. A more recent version are the 2018 JCR impact factors (see there).
Few days ago, Thomson Reuters published the 2015 impact factors of well-known management journals as part of their Journal Citation Reports. Two SCM-related journals have an impact factor of 4 or larger: Journal of Supply Chain Management and Journal of Operations Management. Two other journals have an impact factor between 2.5 and 3: Supply Chain Management: An International Journal and Journal of Purchasing & Supply Management. Journals with an impact factor between 2 and 2.5 are: Journal of Business Logistics, Transportation Research Part E, International Journal of Operations & Production Management, International Journal of Physical Distribution & Logistics Management. Journals with an impact factor between 1.5 and 2 are: Manufacturing & Service Operations Management and Production and Operations Management. Among the journals with an impact factor between 1 and 1.5 is: Decision Sciences. Journals with an impact factor below 1 are: International Journal of Logistics: Research & Applications, International Journal of Logistics Management and Interfaces. However, keep in mind that journal rankings have a downside and should not be the dominating criteria for judging the value of our research. Financial Times has recently decided to include M&SOM rather than JSCM on the FT journal list which indicates that their list is not reliable at all to make SCM faculty decisions. Qualitative rankings such as VHB-JOURQUAL can be a good supplement to quantitative impact factors.
This week, SCM World have published their “SCM World University 100” ranking, which aims to list the best business schools for supply chain talent worldwide. According to this ranking, the top 5 universities worldwide are: (1) Michigan State University, (2) Penn State University, (3) University of Tennessee, (4) Massachusetts Institute of Technology, and (5) Arizona State University. I am very pleased that Copenhagen Business School’s SCM program was selected as one of the top 10 programs in the EMEA region. The ranking is based on survey data collected from more than 2,000 supply chain professionals during the last couple of months. In the survey the participants were asked: “As a marker of supply chain talent, please select your top three universities.” Hereby, the respondents could select from a list of 192 universities that are known to offer supply chain management within their business programs. But be careful: Always keep in mind that there are both intended and unintended consequences of such rankings.
The German Academic Association for Business Research (VHB) has published JOURQUAL 3, a journal ranking based on the ratings of more than 1,100 VHB members. A-graded SCM journals are: Production and Operations Management and Journal of Operations Management. Several SCM journals received the grade B, including: International Journal of Production Economics, International Journal of Physical Distribution & Logistics Management, Journal of Supply Chain Management, Journal of Business Logistics, International Journal of Operations & Production Management, Journal of Purchasing & Supply Management and Supply Chain Management: An International Journal. Finally, International Journal of Logistics: Research and Applications, International Journal of Logistics Management and Logistics Research received the grade C. No SCM journal received the best (A+) or the worst grade (D). Qualitative rankings such as this list or the ABS ranking can be a good supplement to quantitative rankings based on impact factors. Always keep in mind that journal rankings have a downside and should not be the dominating criteria for judging the value of our research.
Note: The following blog post is about an older version of the Academic Journal Guide (“ABS list”). The 2018 AJG is discussed in another blog post (follow this link).
The UK-based Association of Business Schools (ABS) has published its Academic Journal Guide. It is the successor of the often criticized Academic Journal Quality Guide. And this is how the new Guide ranks supply chain management journals: The only grade 4* (“excellent”) journal is: Journal of Operations Management. Other “top journals” (grade 4) are: International Journal of Operations & Production Management and Production and Operations Management. Examples of “highly regarded” journals (grade 3) in the list are: Journal of Supply Chain Management and Supply Chain Management: An International Journal. Some other “well regarded” journals (grade 2) are: International Journal of Physical Distribution & Logistics Management, Journal of Business Logistics and Journal of Purchasing & Supply Management.
Given the low grades of some journals with high impact factors and considering their reputation in our field, I am not convinced of the quality of this new ABS list. For example, in spite of its reputation as a leading SCM journal and its higher impact factor, ABS ranks JBL two (!) grades lower than IJOPM. Another ranking, VHB-JOURQUAL, seems to reflect the theoretical and methodological breadth of our discipline much better – maybe because it is based on the opinions of several hundred business researchers rather than an expert panel like in the case of ABS. However, qualitative rankings like ABS and JOURQUAL can be a good supplement to quantitative rankings based on impact factors.
But always keep in mind that journal rankings have a downside and should not be used as criteria for judging a researcher (they can only be used for judging a journal, in fact). I fear that the new ABS ranking will serve as exactly such a criterion in many business schools now. Isn’t the quality of our own articles a much better criterion than the average quality of all articles published in a journal (including the very bad and very good ones)? But this would require the members of an appointment committee to read what the candidates have actually published – maybe too much of an effort? And, if paradigm shifts often start in low-ranked journals, should our incentive system really prevent us from publishing in journals with ABS ranks below 3?
“Everything has to be measurable!” seems to be the new academic doctrine. In the current issue of BizEd Magazine, Sharon Shinn delivers a detailed description of the problems associated with measuring scholarly impact. She concludes: “Tenure-track faculty continue to absorb responsibilities that take time away from conducting research, but the number of A-journal publications is still the primary metric that is considered in faculty evaluations. And that’s a problem still looking for an answer.” If A-journal publications are the primary metric, this could, for example, also prevent us from writing a good book (writing it can take several years!), even though it might be read all over the world. A key reason why people argue for exactly the kinds of rankings Shinn critiques is that they avoid the need for argument between competing factions with different views. Metrics are really useful in support of decisions. But the problems start when they are made.