Quality assessment in systematic literature reviews : A software engineering perspective

Context: Quality Assessment (QA) of reviewed literature is paramount to a Systematic Literature Review (SLR) as the quality of conclusions completely depends on the quality of selected literature. A number of researchers in Software Engineering (SE) have developed a variety of QA instruments and also reported their challenges. We previously conducted a tertiary study on SLRs with QA from 2004 to 2013, and reported the findings in 2015.

Objective: With the widespread use of SLRs in SE and the increasing adoption of QA in these SLRs in recent years, it is necessary to empirically investigate whether the previous conclusions are still valid and whether there are new insights to the subject in question using a larger and a more up-to-date SLR set. More importantly, we aim to depict a clear picture of QA used in SLRs in SE by aggregating and distilling good practices, including the commonly used QA instruments as well as the major roles and aspects of QA in research.

Method: An extended tertiary study was conducted with the newly collected SLRs from 2014 to 2018 and the original SLRs from 2004 to 2013 to systematically review the QA used by SLRs in SE during the 15-year period from 2004 to 2018. In addition, this extended study also compared and contrasted the findings of the previous study conducted in 2015.

Results: A total of 241 SLRs between 2004 and 2018 were included, from which we identified a number of QA instruments. These instruments are generally designed to focus on the rationality of study design, the rigor of study execution and analysis, and the credibility and contribution of study findings and conclusions, with the emphasis largely placed on its rigor. The quality data is mainly used for literature selection or as evidence to support conclusions.

Conclusions: QA has received much attention in SE in more recent years and the improvement is evident since the last study in 2015. New findings show that the aims are more concise, the instruments are more diverse and rigorous, and the criteria are more thoughtful.