In this post I am planning to collect information about how to do a literary review for HCI projects. The idea for this review is not that of a systematic literature review paper. But a more relaxed version of it that can used to report in a related-work section of an HCI/Ubicomp/UIST paper, and yet be illustrative enough of the state of the art within a certain research path. This post is not a definitive guide, it’s rather a collection of thoughts and experiences that I plan to nurture over time.

Systematic Literature Reviews are great tools when embarking on a extended research effort like a PhD or a group project. Very common in other sciences like the natural sciences and medicine, within computer science and engineering I have found only references to them from the field of Software Engineering. And it’s from Kitchenham’s work [1] that I derive the guidelines for this post. Kitchenham’s concludes her report with a list of the most important steps to undertake. In this posts I propose a series of steps based on Kitchenham’s, and discuss what they entail.

1. Research Question
2. Search Strategy
3. Inclusion/Exclusion Criteria
4. Data Collection
5. Data Coding
6. Data Synthesis

The first step is defining the research question. Some might say: well, sometimes you don’t know what the research question is until you’ve delved into the area sometime. My answer is: right, but at that point you aren’t writing your CHI paper yet! Anyway, human-computer interaction is a very wide field and each particular area has developed its own methods and relevant research questions. From my experience, a sub-set of research questions that come up once and again are: 1) what’s the effect of a novel technology? 2) what’s the frequency or rate of a factor like the adoption of a technology? 3) presenting a novel technology and devise the possible scenarios in which it could be used presenting initial user feedback or other advantages like simplicity or robustness over previous options. 4) what’s the cost and risks factors associated with a certain novel technology?

The second step is defining the search strategy. Here, the authors identify the search terms that define search terms of related work, terms that will be looked for in the titles and abstracts of potential papers for the review. Particularly for emerging topics, it’s important to be familiar with the different names used for the same phenomena/problem and include them all in the search. Next, the authors are to define the sources for their search which will include online databases like the ACM digital library and Google Scholar, but also complete proceedings and journals. The advantage of not relaying entirely on the online databases is that scanning proceedings and journals can produce works where the same problem is named differently, and then provide other search terms. Finally, following on established research groups and researchers can show unpublished studies (like technical reports) and the evolution of a topic/problem into novel research spaces.

The third step is the definition of clear inclusion/exclusion criteria: their main goal is to help identify the primary studies in relation to the problem and their actual relevance. Here, researchers can look at the kind of paper (system, review, evaluation, design, etc) and their impact (citation count). I have also found it useful to look at less determinant things like the download count and how many copies are found around the web. These criteria could also include things like whether the study has conclusive results, or its internal validity, etc.

The fourth step is defining what kind of data will be collected from each paper. For human-computer interaction and Ubicomp studies, I have found it useful to collect data about the methods they used, the internal and external validity of their experimental setup and results, and an assessment of the quality of the work. Also, an aggregation of the results (if qualitative) and acceptance/rejection of the proposed hypothesis. Finally, in case the paper presents a novel technology or a descriptive study, it’s useful to collect a description of the technology in terms of design dimensions (and their values) and the main topics discovered in the descriptive study.

The fifth step is coding the collected data in order to identify main lines of research and significant results. This coding can take many shapes starting with a tabulation of the surveyed papers (as we did for the InterruptMe design space paper) or a more grounded-theory-inspired kind of coding.

The final step handles the synthesis of the captured data and its aim is to identify what are the gaps and opportunities given the current state of the art. A particular useful thing to look at is the discussion and future work sections of the covered literature (they must have been looked at during the data collection). In the actual paper-writing, this is where you point out how’s your work different than the existing one (either in focus or contribution).

OK, this is the approach I am taking this year. The reader should remember that this is not a fully systematic review like the ones proposed at [1], but rather a way to cover the state of the art for writing compelling related work sections of CHI/Ubicomp papers. The reader should also remember that this is an iterative process, with several cycles over these steps and with shifts back and forth between them.

[1] B. Kitchenham. Procedures for performing systematic reviews.
Technical report, Keele University and NICTA, 2004.