HCI Systematic Review-lution
A landing page and collection of resources for a movement towards more rigorous and useful systematic reviews in human-computer interaction (HCI).
The Systematic Review-lution: A Manifesto to Promote Rigour and Inclusivity in Research Synthesis
by Katja Rogers and Katie Seaborn
This manifesto advocates for community-driven efforts towards figuring out what rigour in systematic reviews can and should mean in human-computer interaction. It was published in the alt.chi track at CHI 2023.
- ACM Digital Library: doi: 10.1145/3544549.3582733
- arXiv PDF: arXiv:2304.13556
- Pre-recorded presentation video
- Presentation slides from CHI 2023’s alt.chi session (different from pre-recorded version)
We also carried out an informal survey of community opinions alongside this presentation.
Too Long, Didn’t Read:
We see a broad range of differing opinions and expectations when it comes to research synthesis, and more specifically systematic reviews: what counts as a systematic review, what kinds of them exist, what purpose they have, how they should be conducted and reported, what forms of knowledge they produce, and whether they have a place in HCI at all. In the manifesto, we address objections to systematic reviews in general, and map out critical issues in the context of systematic reviews that we see within the academic ecosystem.
Critical Issues Charted in the Manifesto:
- Primary Research Reporting: In line with many previous calls for more transparent and consistent reporting, we believe that primary research reporting should be viewed and improved also from the perspective of supporting research synthesis and systematic reviews.
- Epistemological Diversity in HCI: There are many different ways of knowing in HCI, and that should be reflected in our research synthesis as well. This may require novel forms of synthesis. We expect that this will also be reflected in the types of research synthesis or systematic reviews that make sense in our field.
- Secondary Research Reporting: We see room for improvement when it comes to research synthesis, both in conduct and reporting.
- Publishing Ecosystem (Venues & Subcommittees): There aren’t really any places that explicitly welcome and provide clear expectations for systematic reviews. Update: The very new Journal of Visualization and Interaction explicitly mentions that systematic reviews are welcome, although what this looks like is not clearly defined (yet).
- Infrastructure (Digital Libraries and Tools, e.g., Machine Learning): There is little documentation about the relevance, coverage, and inner workings of HCI’s most commonly used databases. There could also be potential for the use of automated and semi-automated tools in various steps of the systematic review process, but what risks and issues this can introduce is unclear. Finally, living reviews may benefit from HCI expertise; this is something we should leverage and investigate.