Review ratings

PC members and external reviewers can anonymously rate one another’s reviews. We hope this feedback will help reviewers improve the quality of their reviews.

When rating a review, please consider its value for both the program committee and the authors. Helpful reviews are specific, clear, technically focused, and provide direction for the authors’ future work. The rating options are:

• Good review: Thorough, clear, constructive, and gives good ideas for next steps.
• Needs work: The review needs revision. If possible, indicate why using a more-specific rating.
• Too short: The review is incomplete or too terse.
• Too vague: The review has weak or unconvincing arguments or gives little useful direction.
• Too narrow: The review’s perspective seems limited; for instance, it might overly privilege the reviewer’s own work.
• Disrespectful: The review’s tone is unnecessarily aggressive or exhibits bias.
• Not correct: The review misunderstands the paper.

HotCRP reports aggregate ratings for each review. It does not report who gave the ratings, and it never shows review ratings to authors.

To find which of your reviews might need work, simply search for “rate:bad:me”. To find all reviews with positive ratings, search for “rate:good”. You may also search for reviews with specific ratings; for instance, search for “rate:short”.

Settings

Chairs set how ratings work on the review settings page.

Visibility

A review’s ratings are visible to any unconflicted PC members who can see the review, but HotCRP tries to hide ratings from review authors if they could figure out who assigned the rating: if only one PC member could rate a review, then that PC member’s rating is hidden from the review author.