An Improved Review Process
for the SIGCSE Technical Symposium

From "Thousand Islands" to "Strongly Connected Networks" of APCs and Reviewers

Starting this year, we have adopted an improved review process for the SIGCSE Technical Symposium.

We increase the number of Associate Program Chairs (APCs) by a factor of three (from 12 to ~36). By increasing the number of APCs, we achieve a setup similar to most other CS conferences with invited program committee members while maintaining the volunteer reviewers and the associated qualities. We get the best of both worlds, so to speak.

Previously, reviewers made their reviews (in isolation) and then an APC conducted a meta review (in isolation) resulting in a recommendation to the program chairs who made final decisions based on the recommendation from the APC and of course with respect to an overall concern for a broad and coherent technical program of optimal quality. This review model had a number of (well-known) flaws that surfaced as lack of both uniformity, transparency, and robustness.

In the improved review model, reviewers initially make their reviews as previously (in isolation); but now, this first phase is followed by a second phase where the reviewers of a paper and an APC discuss the paper and its reviews. The APC is responsible for facilitating an asynchronous online discussion within EasyChair (the submission system). The outcome is more thorough and consistent reviews of higher quality, and a clear recommendation to the program chairs which has been negotiated among the reviewers and the APC.

The new review process has (at least) two advantages over the old:

1. Improvement of reviews (short-term effect)
   Reviewers are requested to comment on other reviews of the same submission and potentially modify their own review to reflect an improved appreciation (or the opposite) of the submission. This will lead to improved and (in general) more consistent reviews.
2. Improvement of reviewer skills (long-term effect)
   The reading of other reviews and the anonymous discussions will help improve reviewer skills in a constructive and efficient way (in-process).

As it is current practice, APCs (PC members) are personally invited, and a fair percentage is renewed every year.

Reviewers are still volunteers. Reviewers can indicate willingness to serve as APC (PC member) and are eligible if they demonstrate the ability to write, discuss, and refine reviews of high quality.

Regarding the headline From "Thousand Islands" to "Strongly Connected Networks" of APCs and Reviewers: The latter, strongly connected networks, is even better than the "traditional" tree-structured hierarchy of PC members and their local assistants (postdocs and PhD students) practised by many program committees. In the new setup, we harvest the benefits of networks: uniformity (or at least quick convergence towards uniformity), robustness (a "bad functioning" internal node does not have exclusive control over a subtree), and transparency (see, be advised, and learn).

We look forward to experience the new review process in action, to evaluate it, and adjust it if need be, and we hope you'll all embrace it and help make it a success.