Three years ago, the Chartered Association of Business Schools (ABS) has released its last ranking of business journals: the Academic Journal Guide (AJG), also known as the “ABS list”. This ABS ranking has become quite influential as the guiding journal ranking across management disciplines in the UK. Although the ranking has been heavily criticized (see my previous post) and more democratically-developed rankings exist (e.g. VHB-JOURQUAL), the ABS ranking has since been adopted by many business schools also in other countries.
Shortly after the publication of the last ABS list, Nature has published ten principles to guide research evaluation, which have since become known as the Leiden Manifesto. One of these principles argues that quantitative evaluation should support qualitative, expert assessment, not replace it. Similarly, the San Francisco Declaration on Research Assessment (DORA), which has been signed by thousands of researchers worldwide, asks “not [to] use journal-based metrics […] as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”. Indeed, the negative consequences of rankings are well-documented (see, for example, Espeland & Sauder, 2007 and Grant & Kovács, 2018).
Unfortunately, the ABS list has often been used in exactly the way the Leiden Manifesto and DORA wanted to prevent us from doing – with very negative consequences for our discipline. Many SCM researchers feel that the ABS list undervalues the journals of our discipline. For example, Journal of Supply Chain Management, a journal with one of the highest impact factors in management (5.789), is ranked by ABS 2015 as a “3” only and Journal of Business Logistics as a “2”. Comparing our journals to other disciplines, like accounting or marketing, it becomes apparent to me that these journals deserve a “4*” and “4”. Worse even, many SCM journals have suffered from their low ABS rankings, as SCM researchers who took the ABS ranking too seriously, felt they should publish in the higher-ranked journals of other disciplines. AJG 2015 has thus become a self-fulfilling prophecy.
Today, the Academic Journal Guide 2018 has been released. It contains more than 1,500 entries. Unfortunately, AJG 2018 did not adapt the ranks of leading SCM journals. When it comes to empirical journals, only Journal of Operations Management gets a “4*”, while Decision Sciences, Journal of Supply Chain Management and Supply Chain Management: An International Journal get a “3” again. International Journal of Physical Distribution and Logistics Management, Journal of Business Logistics and Journal of Purchasing and Supply Management all still get a “2” only. The fact that the rank of none of these journals has changed indicates that the new ranking does not seem to cover all the changes and improvements our discipline has made in the last couple of years. Only four out of eight journals (four empirically- and analytically-focused journals each) that have been identified by leading SCM schools as the top journals of our discipline (sometimes informally referred to as “the SCM basket of eight”; see The SCM Journal List), got an AJG rank higher than 3. My suggestion would be to ignore AJG 2018, as it does not seem to represent the journals of our discipline properly.
Interestingly, even the Chartered ABS itself just tweeted that the REF2021 sub-panel will ignore their own list, so they might well be aware of its shortcomings:
#REF2021 sub-panel chair: “We will not be using metrics. Our role is to assess originality, significance and rigour. We will not use #AJG2018 to do so.” Robert Blackburn #ARC2018
But there is some hope: This latest iteration of AJG was an interim review, the main purpose being to include new journals. The next iteration, planned to be published in 2021, will then be a major review. This is certainly very late, but better late than never: Careers can depend on such lists. Therefore, our discipline urgently deserves a better quality of the AJG. I hope the team behind the list is aware of their responsibility.
There has been a recent trend in several management disciplines, including supply chain management, to create knowledge by systematically reviewing available literature. So far, however, our discipline lacked a “gold standard” that guides researchers in this endeavor. The Journal of Supply Chain Management has now published our new article, Durach, Kembro & Wieland (2017): A New Paradigm for Systematic Literature Reviews in Supply Chain Management. Our systematic literature review process follows six steps: (1) develop an initial theoretical framework; (2) develop criteria for determining whether a publication can provide information regarding this framework; (3) identify literature through structured and rigorous searches; (4) conduct theoretically driven selection of literature and a relevance test; (5) develop two data extraction structures, integrate data to refine the theoretical framework, and develop narrative propositions; and (6) explain the refined framework and compare it to the initial assumptions. We believe that these best-practice guidelines, although developed for the SCM discipline, can be used as a blueprint also for adjacent management disciplines.
Durach, C.F., Kembro, J. & Wieland, A. (2017). A New Paradigm for Systematic Literature Reviews in Supply Chain Management. Journal of Supply Chain Management, 53 (4), 67-85. DOI: 10.1111/jscm.12145
In a previous post, it was demonstrated that researchers can play two different roles, as they can either build or test theories. An SMJ article by Miller and Tsang (2011), which is titled Testing management theories: Critical realist philosophy and research methods, focuses on the latter role we can play. The authors claim: “Not only do we have a plurality of theories within management research, there is also no consensus about the criteria for evaluating theories.” Taking a critical realist perspective, they advance practical guidance for evaluating management theories by proposing a four-step approach to theory testing. This approach includes (1) identifying the hypothesized mechanisms, (2) testing for the presence of the mechanisms in the empirical setting, (3) testing isolated causal relations, and (4) testing the theoretical system. The authors underline that “steps 2 and 3 have been neglected for the most part”. In sum, a lot can be learnt about theory testing from this brilliant article.
Miller, K., & Tsang, E. (2011). Testing management theories: Critical realist philosophy and research methods. Strategic Management Journal, 32 (2), 139-158 DOI: 10.1002/smj.868