An Informal Community Survey on Research Synthesis & Systematic Reviews

by Katja Rogers and Katie Seaborn

As part of the alt.chi presentation, but also advertised more widely through social media (Twitter, Mastodon, various Slacks), we are gathering community responses on what HCI expects from systematic reviews: what they can look like and what they can result in.

The survey is available here: https://bit.ly/3oBteKA

The results are based on simple descriptive counts and informal open coding. It should be kept in mind that the kinds of people that responded to it are the kinds of people that generally care about systematic review methodology at least to some extent, one way or another.

We asked respondents i) to report their familiarity with systematic reviews, ii) to describe what makes a review systematic, iii) to rate whether they consider certain steps to be (un-)necessary or best practice, and iv) to indicate potential kinds of knowledge that can result from systematic reviews.

---

i) Respondents

A total of n=48 respondents completed the survey.

Fig 1: The respondents’ reported familiarity with systematic reviews based on a single-selection question.

ii) What Makes A Review “Systematic”

Fig 2: The number of respondents whose answers to this question were coded for one of the below aspects (through informal open coding).

The respondents mentioned aspects relating to the process of conducting a systematic reivew, resulting outcomes, its coverage, as well to what extent it was planned a priori, and was reproducible, and transparently reported.

Process: Following “established” methods or conventions was expected (if a specific one was mentioned, then it was PRISMA - although we note that PRISMA is a reporting guideline and its applicability to HCI is less than ideal). Others expected just a(n otherwise unspecified) degree of rigour, structure, or system, with which a review has to be consistent. Only a couple respondents mentioned specifics like an iterative search strategy development, appropriateness of search databases, or quality of the corpus papers.
There were complaints about lack of clarity regarding what makes reviews systematic, and about a lack of in-depth engagement with findings. There were some mentions of reducing bias, but also mentions of allowing for subjectivity.

Transparency: Transparent reporting was most commonly named or implied (e.g., “well-defined”). There was not necessarily agreement about which aspects should be transparent: paper search and paper selection (screening) were by far most commonly mentioned though. Other aspects mentioned in this vein: goal/research question/scope, analysis/synthesis, results of interim steps, materials, limitations, and contribution.

Coverage: Many mentioned the need to aim for comprehensiveness and coverage of breadth. One mentioned specifically the need to include multiple databases. Specifics were rare: one mentioned the goal of an “ample” corpus size, another a minimum of 50 papers per author, and the final mention described a sweet spot as between 40 (to “summarize” enough) and 200 (to avoid becoming too superficial). However, a couple of respondents argued against the need for comprehensiveness, as long as restrictions to this were well rationalized.

A Priori: Some respondents mentioned the need for pre-planning aspects of the review beforehand. When this was specified, this most commonly referred to the research goal/question, the search strategy, or the screening/selection. Data extraction or protocols were rarely mentioned in this context.

Reproducibility: This was mentioned by several respondents, although what exactly should be reproduced was less clear. Search was mentioned a few times, and for a couple also the data extraction. One respondent stated that the “result” should be reproducible.

Outcomes: Few respondents were specific about the expected outcome from a systematic review, and there seemed to be no agreement about what this should be. A couple viewed this as an overview/summary, others as outlining challenges or new directions, yet others as “going beyond just summarizing”. There were some mentions of results aiming for reliability and validity.

iii) Which Aspects are (Un-)Necessary / Best Practice

Fig 3: Necessity of protocol (i.e., documentation of planned review procedure incl. analysis, specific to the review at hand) seems to be considered high - although we note it is still rarely seen in systematic review papers in HCI.

Fig 4: Necessity of pre-registration is considered less necessary, more of a best practice.

Fig 5: Necessity of screening (i.e., checking papers for relevance prior to their inclusion in analysis) was considered very necessary.

Fig 6: Necessity of double or multi-screening (every paper screened by 2+ people, possibly with tie-breaking) seems considered more of a best practice, although the dependency on the review type went up noticeably.

Fig 7: Necessity of screening (i.e., checking papers for relevance) with formal inclusion/exclusion criteria again seems quite necessary.

Fig 8: Necessity of a formalized data extraction process seems quite necessary but may depend on the review type.

Fig 9: Data extraction with multiple coders (data from every paper extracted by multiple people, possibly with tie-breaking) is considered more of a best practice (with high dependency on the review type).

Fig 10: Necessity of quality appraisal (i.e., some kind of verification step that the selected papers are of sufficient quality to help answer the research question) received fairly mixed responses, with agreement that this depends on the review type. This is a noticeable departure from many other fields’ understanding of necessary systematic review components.

Fig 11: Necessity of quality appraisal with multiple coders (every paper appraised by 2+ people, possibly with tie-breaking) seems more of a best practice, but with similar agreement that it depends on the review type.

Fig 12: Necessity of a formal analysis / synthesis method (e.g., meta-analysis, thematic synthesis, …) seems strong, although with similarly strong dependency on the review type.

iv) Forms of Knowledge That Can Result From a Systematic Review

Fig 13: Respondents were quite positive about the types of knowledge that can result from systematic reviews, reported via a multi-selection question. Respondents most commonly felt confident it can result in a “map / overview of the field”, “a specific answer to a broad research question (e.g., what are the types of X used in research domain Y)”, and “future research directions to progress the field”. This agreement decreased slightly when it came to “a specific answer to a targeted research question (e.g., what is the effect of X in the context of Y on dependent variable Z)” and “novel theory / framework / form of intermediate-level knowledge” but was overall still considered a possible outcome by a comfortable majority.