Chair’s guide

Submission time

Follow these steps to prepare to accept submissions.

1. Set up PC member accounts. Many PCs are divided into classes, such as “heavy” and “light”, or “PC” and “ERC”. Mark these classes with user tags. It’s also useful to configure tag colors so that PC member names are displayed differently based on class (for instance, heavy PC member names might appear in bold).

2. Set submission policies, including anonymity.

3. Set submission deadlines. Authors first register, then submit their work, possibly multiple times; they choose for each submitted version whether that version is ready for review. Normally, HotCRP allows authors to update submissions until the deadline, but you can also require that authors “freeze” each submission when it is complete. Only the submission deadline really matters, but HotCRP also supports a separate registration deadline, after which new submissions cannot be started. An optional grace period applies to both deadlines: HotCRP reports the set deadlines, but allows updates post-deadline for the specified time.

4. Set up the submission form, including whether abstracts are required, whether authors check off conflicted PC members (“Collect authors’ PC conflicts with checkboxes”), and whether authors must enter additional non-PC collaborators, which can help detect conflicts with external reviewers (“Collect authors’ other collaborators as text”). The submission form also can include:

   • PDF format checker. This adds a “Check format” link to the Edit screen. Clicking the link checks the submission for formatting errors, such as going over the page limit. Submissions with formatting errors may still be submitted, since the checker itself can make mistakes, but the automated checker leaves cheating authors no excuse.

   • Additional fields such as checkboxes, selectors, freeform text, and uploaded attachments. Checkbox fields might include “Consider this paper for the Best Student Paper award” or “Provide this paper to the European shadow PC.” Attachment fields might include supplemental material. You can search for submissions with or without each field.

   • Topics. Authors can select topics, such as “Applications” or “Network databases,” that characterize their submission’s subject areas. PC members express topics for which they have high, medium, and low interest, improving automatic review assignment. Although explicit preferences (see below) are better than topic-based assignments, busy PC members might not specify their preferences; topic matching lets you do a reasonable job at assigning reviews anyway.

5. Take a look at a submission page to make sure it looks right.

6. Open the site for submissions. Submissions are allowed up to the listed deadline.

Assignments

After the submission deadline has passed:

1. Consider checking the papers for anomalies. Withdraw and/or delete duplicates or update details on the paper pages (via “Edit paper”). Also consider contacting the authors of incomplete submissions, especially if a PDF document was uploaded; sometimes a user will uncheck “The submission is ready for review” by mistake.

2. Check for formatting violations (optional). Search > Download > Format check will download a summary report. Serious errors are also shown on paper pages (problematic PDFs are distinguished by an “X”).

3. Prepare the review form. Take a look at the templates to get ideas.

4. Set review policies and deadlines, including reviewing deadlines, review anonymity, and whether PC members may review any paper (usually “yes” is the right answer).

5. Prepare tracks (optional). Tracks give chairs fine-grained control over PC members’ access rights for individual papers. Example situations calling for tracks include external review committees, PC-paper review committees, and multi-track conferences.

6. Collect review preferences (bids) from the PC. PC members can enter per-paper review preferences (also known as bids) to mark papers they want or don’t want to review. Preferences are integers, typically in the range −20 to 20; the higher the number, the more desired the review assignment. Users can either set their preferences all at once, or (often more convenient) page through the list of submitted papers and set their preferences on the paper pages.

   PC members can enter review preferences as soon as they can see submissions. A “Review preferences” link will automatically appear on the site home page. If desired, you can also collect preferences on unsubmitted abstracts; select “PC can view incomplete submissions until submission deadline”, which allows the PC to see abstracts for registered papers that haven’t yet been submitted.

7. Check for missing conflicts. HotCRP does not automatically confirm all conflicts, such as conflicts indicated by PC members’ “Collaborators and other affiliations.” Use the conflict assignment tool to find and confirm such conflicts.

8. Assign reviews. You can make assignments by paper, by PC member, by uploading an assignment file, or, even easier, automatically. PC review assignments can be “primary” or “secondary”; the difference is that primary reviewers are expected to complete their review, but a secondary reviewer can delegate their review to someone else. You can also assign PC “metareviews”. Unlike normal reviewers, a metareviewer can view all other reviews before submitting their own.

   The assignment pages apply to all submissions by default. You can also assign groups of submissions, such as papers with fewer than three completed reviews.

9. Enable review editing.

Chair conflicts

Chairs and system administrators can access any information stored in the conference system, including reviewer identities for conflicted papers. It is easiest to simply accept such conflicts as a fact of life. Chairs who can’t handle conflicts fairly shouldn’t be chairs. However, HotCRP does offer other mechanisms for conflicted reviews.

First, each paper can be assigned a paper administrator: a PC member who manages that paper’s reviewing and discussion. Use the left-hand side of a submission’s assignment page to enter its administrator. (You may need to “Override conflicts” to access the assignment page.) Paper administrators have full privilege to assign and view reviews for their papers, and can, for example, use the autoassignment tool, but they cannot change conference settings. When a paper has an administrator, chair conflicts cannot be overridden.

Paper administrators make life easy for PC reviewers and greatly restrict conflicted chairs’ access. Usually this suffices. For additional privacy, use review tokens, which are completely anonymous review slots. To create a token, an administrator goes to an assignment page and clicks on “Request review” without entering a name or email address. This reports the token, a short string of letters and numbers such as “9HDZYUB”. Any user who knows the token can enter it on HotCRP’s home page, after which the system lets them view the paper and anonymously modify the corresponding “Jane Q. Public” review. True reviewer identities will not appear in HotCRP’s database or its logs. For even more privacy, the paper administrator could collect offline review forms via email and upload them using review tokens; then even web server access logs store only the administrator’s identity.

Before the meeting

1. Collect authors’ responses to the reviews (optional). Authors’ responses (also called rebuttals) let authors correct reviewer misconceptions before decisions are made. Responses are entered into the system as comments. On the decision settings page, update “Collect responses to the reviews,” then send mail to authors informing them of the response deadline. PC members can still update their reviews up to the review deadline; authors are informed via email of any review changes.

2. Set PC can see review contents to “Yes” (optional). This opens up the reviews to the program committee, allowing everyone to see scores and read reviews for non-conflicted papers. (During most conferences’ review periods, a PC member can see a paper’s reviews only after completing their own review for that paper.)

3. Examine paper scores, either one at a time or en masse, and decide which papers will be discussed. The tags system can group papers and prepare discussion sets. Use search keywords to, for example, find all papers with at least two overall merit ratings of 2 or better.

4. Assign discussion orders using tags (optional). Common discussion orders include sorted by overall ranking (high-to-low, low-to-high, or alternating), sorted by topic, and grouped by PC conflicts. Explicit tag-based orders make it easier for the PC to follow along.

5. Assign discussion leads (optional). Discussion leads are expected to be able to summarize the paper and the reviews. You can assign leads either paper by paper or automatically.

6. Define decision types (optional). By default, HotCRP has two decision types, “accept” and “reject,” but you can add other types of acceptance and rejection, such as “accept as short paper.”

7. The night before the meeting, download all reviews onto a laptop (Download > All reviews) in case the Internet explodes and you can’t reach HotCRP from the meeting place.

At the meeting

1. The meeting tracker can keep the PC coordinated by sharing your browser’s status. Search for papers in whatever order you like (you may want an explicit discussion order), navigate to the first paper in the order, and select “☟” to activate the tracker in that tab. As you use that tab to navigate through the order, its current position is broadcast to all logged-in PC members’ browsers:

   

   Manage multiple trackers and limit PC visibility by shift-clicking “☟” or clicking it again. You can also view the discussion status on the discussion status page.

2. Scribes can capture discussions as comments for the authors’ reference.

3. Paper decisions can be recorded on the paper pages or en masse via search. Use decision settings to expose decisions to PC members if desired.

4. Shepherding (optional). If your conference uses shepherding for accepted papers, you can assign shepherds either paper by paper or automatically.

After the meeting

1. Enter decisions and shepherds if you didn’t do this at the meeting.

2. Give reviewers some time to update their reviews in response to PC discussion (optional).

3. Set “Can authors see decisions?” to Yes.

4. Send mail to authors informing them that reviews and decisions are available. The mail can also contain the reviews and comments themselves.

5. Collect final papers (optional). If you’re putting together the program yourself, it can be convenient to collect final versions using HotCRP. Authors upload final versions just as they did submissions. You can then download all final versions as a .zip archive. (The submitted versions are archived for reference.)