Home » Guide to Reviewing Papers

Guide to Reviewing Papers

Guide to Reviewing Papers

This page provides guidelines for reviewers responsible for assessing submissions to CHI. 

Key points:

  • Reviewers’ primary criterion for judging a paper is: Does this submission provide a strong contribution to the field of HCI? In detail, reviewers must assess papers across five dimensions outlined by ACM: originality, correctness, novelty, importance, and clarity of exposition. Reviewers are asked to remember that there are many ways a paper can contribute to HCI, and they should review it appropriately. The “Contributions to CHI” article provides more details about what constitutes a strong contribution.
  • Reviewers decide the overall recommendation (Accept with minor revisions, Revise and Resubmit, or Reject) for each submission, along with a written appraisal supporting their recommendation.
  • High-quality CHI reviews are typically about 650 words long. Very short reviews are frustrating for authors, tend to be less constructive, and hurt the review process. Reviews should evaluate the manuscript according to its stated aims and methodology while remaining open to different perspectives and research approaches. Feedback should be specific, actionable, respectful, and focused on improving the manuscript rather than judging its authors. Reviewers should always put themselves in the author’s position: what level of detailed feedback would you like to see for your own work?
  • Reviews have to be detailed, regardless of the reviewer’s assumed outcome. Short positive or short negative reviews without justification will not support decisions about the paper during the PC meeting, especially if there are well-argued counter-reviews.
  • Reviewers are responsible for the content and accuracy of the reviews, including the references cited. This is especially relevant if AI tools based on Large Language Models (LLM) are used, which are prone to producing plausible-sounding but wrong text. Such reviews can negatively reflect on the reviewer and their reputation. The ACM Policy on using AI tools in reviewing is provided here: https://www.acm.org/publications/policies/peer-review-faq.
  • Maintaining confidentiality is crucial in the review process; thus, quoting or disclosing papers under review is not allowed. Copying and pasting content from a manuscript into an LLM may pose risks to confidentiality (again, note the ACM Policy: https://www.acm.org/publications/policies/peer-review-faq). Reviewers must therefore avoid sharing content from unpublished papers with any online tools that may store or process the data in any way that could compromise confidentiality.
  • Reviewers should adhere to the Code of Conduct in Reviewing.

Contributions

The primary criterion for the evaluation of all papers is the submission’s contribution to HCI. In all cases, a CHI paper must make an original research contribution. It is important to recognize that a paper can make a contribution to HCI in many ways, and the review should appropriately reflect this. Please see Guide to a Successful Submission for the associated criteria that you can use to assess different types of contributions.

Paper Lengths

During this year’s CHI review process, the papers’ length should be commensurate with their contribution. There is no arbitrary maximum (or minimum) length to papers. However, clarity and conciseness of writing are considered vital to a high-quality submission. When discussing paper lengths, we work from the main text of the paper – not references, figures, tables, etc. To facilitate judgments on whether or not a paper’s length is commensurate with contribution, authors are required to identify if their paper is:

  • short (less than 5,000 words),
  • long (typically 7,000–8,000 words), or
  • excessively long (more than 12,000 words).

Papers of different lengths are reviewed within the same rigorous review process and, at the highest level, are judged by very similar criteria (i.e., does this paper provide a strong contribution to the field of HCI?). However, it is important as a reviewer to realize that the type of content that is appropriate for a shorter paper is somewhat different from that for a longer paper. A short paper should present brief and focused research contributions that are noteworthy but may not be as comprehensive or provide the same depth of results as a long paper – reviewers should not ask for more, but focus on what is needed for a short contribution.

Authors of excessively long papers (going beyond 12,000 words) are required to justify to reviewers why a shorter contribution is not suitable or not possible. In a very exceptional case, such submissions may be considered for full review, but authors must provide clear strong justification for why submissions are required to be more than 12,000 words. The length of typical, long submissions is expected to be approximately 7,000–8,000 words.

Papers whose lengths are incommensurate with their contributions will be rejected. Papers may be perceived as too long if they are repetitive or verbose, or too short if they omit important details, neglect relevant prior related work, or tamper with formatting rules to save on page count.

CHI encourages shorter, more focused papers, but all papers will be reviewed through the same process.

Prior Publication

Content appearing at CHI has to be original content that has not been published elsewhere and should be new and groundbreaking. Therefore, material previously published in widely disseminated archival publications should not be republished unless the work has been significantly revised, thereby making it original again. Guidelines for determining “significance” of a revision are stated in the ACM Policy on Pre-Publication Evaluation and the ACM Policy on Prior Publication and Simultaneous Submissions. Roughly, a significant revision would contain more than 25% of new content material (i.e., material that offers new insights, new results, etc.) and significantly amplify or clarify the original material. These are subjective measures left to the interpretation and judgment of the reviewers and committee members. Authors are advised to revise well beyond the policy guidelines.

Also note that non-archival venues, such as workshop presentations, posters, and CHI’s own Late-Breaking Work do not count as prior publications with respect to content originality. Note: however, such work by the authors has to be cited nevertheless.  Furthermore, a CHI paper should not be rejected on the grounds that it overlaps with work developed independently that was published after the CHI submission was made, during the review period. In other words, work that an author could not have known about should not count against them.

Replicating Work

The policy on prior publication refers only to re-publication of one’s own work; this does not preclude publication of work that replicates other researchers’ work. Constructive replication can be a significant contribution to human-computer interaction, and a new interpretation or evaluation of previously published ideas can make a good CHI paper. However, for future replications to be possible, the submitted work must include sufficient information. Efforts to include complete, well-organized supplementary material facilitating replication, such as software, analysis code and data, should be rewarded.

Transparency

Lack of transparency in how research results are reported can be grounds for doubting the contribution.

Use of AI tools such as LLMs

There are many online tools reviewers might use for editing and proofreading their reviews. Such use is not prohibited but must be used with care. To maintain the integrity of the review process, reviewers should avoid sharing content from unpublished papers with any online tools that may store or process the data in ways that could compromise confidentiality – it is the reviewer’s responsibility to understand the implications of the use of any such tool and abide accordingly. Tools based on LLMs demonstrate greater commitment to prompt completion rather than accuracy. Reviewers must ensure accuracy and relevance of their reviews and take care after using such tools for editing and proofreading. For more information, see the section Confidentiality of Submissions, Authors, and Reviews of the ACM Peer Review Policy.

References

We highly recommend Ken Hinckley’s thoughtful piece on what excellent reviewing is. If we had any way to enforce this, we would make it “required reading” for CHI reviewers and ACs.

Even with great guidelines like these that we can all agree on, the debate over what makes a good CHI paper has persisted since the CHI conference began. For the interested reader, the papers below address this debate and include references to additional papers on it.

  • Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. CHI ’08. ACM. http://doi.acm.org/10.1145/1357054.1357074
  • Olsen, D. R. 2007. Evaluating user interface systems research. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. UIST ’07. ACM. http://doi.acm.org/10.1145/1294211.1294256
  • Dourish, P. 2006. Implications for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’06, ACM. http://doi.acm.org/10.1145/1124772.1124855
  • Newman, W. 1994. A preliminary analysis of the products of HCI research, using pro forma abstracts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 – 28, 1994). ACM, New York, NY, 278-284. http://doi.acm.org/10.1145/191666.191766
  • Daniel Reed and Ed H. Chi. 2012. Online privacy; replicating research results. Commun. ACM 55, 10 (October 2012). https://doi.acm.org/10.1145/2347736.2347739