The 48th Annual IEEE/ACM International Symposium on Microarchitecture, 2015

Review Model for MICRO 2015

Updated: May 1, 2015


This webpage outlines the review process for MICRO 2015. MICRO 2015 will extend the rebuttal period to three weeks. Authors of highly ranked papers (those that are likely to be discussed at the PC meeting) will be recommended to submit a revised version of the paper based on the suggestions/concerns of the reviewers. The review process remains unchanged for the papers that are unlikely to be discussed at the PC meeting (rank 80+ and no champion), whereby the authors are recommended to submit a rebuttal only. We discuss the timeline for the review process, and describe the details and decisions during each part of the review process.


Introduction

One of the objectives of a good review process is to provide feedback to the authors that allows them improve the quality of their submissions. In the reviewing model currently employed at our conferences, the reviewers provide a list of concerns to the authors, and the authors have only a limited way to respond, in the form of a rebuttal. The tone of rebuttal typically is to clarify the misconceptions of reviewers. However, the authors are explicitly prohibited from adding any new content to the paper, so they are unable to demonstrate how they have specifically addressed the concerns provided by the reviewers. In the proposed review model, we seek to provide the authors a window of three weeks in which they can revise the paper to address the concerns, and incorporate the suggestions, of the reviewers. This is not unlike the review model used in Journals, except that this is a single-shot revision rather than having several rounds of interactions between the reviewers and the authors. Such single-shot revision has been incorporated at top conferences in other fields, such as the International Conference on Very Large Data Bases (VLDB), with great success. The figure below captures the overall review process for MICRO 2015.



The key dates for MICRO 2015 are as follows:
  1. Submission deadline for full paper: May 22 (11:59 pm EDT)
  2. Reviews due from PC/ERC: July 10
  3. Discussion between reviewers: July 13-17
    Determine recommendation for revision/rebuttal: July 18-19
  4. Reviews released to authors: July 20
  5. Deadline for rebuttal and revision: August 10
  6. Reviewers read revision/rebuttal: Aug 11-20 
    Online discussion period: Aug 21-26
  7. PC meeting in Atlanta: Aug 29-30
  8. Final decision communicated to authors: Sep 3


1. Submission Deadline (May 22, 11:59pm EDT) 

The submission deadline for MICRO 2015 is approximately one week earlier than the submission deadline for MICRO 2014, in order to accommodate the proposed three-week revision period. The page limit will be 11 pages for text and unlimited space for references. The submission format that we provide will be identical to the camera-ready version, in order to avoid the time to port from submission version to camera-ready version for the accepted papers.  

2. Review Period (May 28 to July 10)

The PC chair will strive to allocate all reviews within six days of the submission deadline. Each paper will be assigned five reviews: 3 from PC and 2 from ERC. The reviewers will have six week to complete their reviews. The six-week review period is similar to what we currently have for HPCA, so it is not unreasonable to expect reviews in six weeks.

3A. Online Discussion Between Reviewers (July 13-17) 

There is a one-week period between the review submission and releasing the reviews to the authors. The purpose of the pre-rebuttal discussion is that the reviewers can see the other reviews. This will (hopefully) help all the reviewers to build a consensus on the concerns and feedback, and also eliminate potential misunderstanding that a reviewer may have. The reviewers are allowed to change their scores based on the discussion.

3B. Determining the Decision of Rebuttal or Revision (July 18-19) 

All papers should have five reviews by this period. All papers will be rank ordered based on the average score of all the reviews for that paper. To reduce the workload for PC members, only a select few (that are likely to appear at the PC meeting) will be recommended for revision. In addition, a reviewer can champion a paper for revision.  The algorithm for determining the recommendation of revision is shown in the figure below. Note, that we are using a threshold of Top80 only as an example, and we may need to fine-tune the exact threshold based on the runtime information. Similarly, we may consider using a model that ignores the review with minimum score while calculating the average scores for the paper, in order to reduce the impact of an outlier harsh review.

The outcome of this phase is to group the papers into two categories: (Category A) "Invited for Revision" and (Category B) "Recommended for Rebuttal Only". All papers in the Category A will also be allocated a “lead”. The lead will “oversee” the reviews, revision, and discussion for the rest of the process, including the post-response period.


4. Reviews Released to Authors (July 20) 

Along with the reviews the authors will get one of the following recommendation:
(A) Invited for Revision (In addition to Rebuttal)
(B) Recommended for Rebuttal Only

For the “Category A” papers, authors are allowed to incorporate the suggestions from the reviewers and address their concerns. The authors are encouraged to improve the editorial quality of the paper. The authors must highlight the sections where changes are made. The authors are NOT allowed to make any unsolicited additions to the paper. In the rebuttal, the authors should point to the sections in the revised paper where they have addressed the concerns of the reviewers, and summarize the revisions made. For the “Category B” papers, the authors are recommended to submit a rebuttal only.

5. Authors Submit Rebuttal and/or Revision (Aug 10) 

The authors have three weeks to submit their revision and/or rebuttal. Note that revision is optional. So, even if a paper is recommended for revision, the authors may still choose to respond only with a rebuttal (or choose to not respond at all).


6A. Reviewers Read Revision and/or Rebuttal (Aug 11-20) 

The reviewers read the revisions for “Category A” papers, and see if/how their concerns got addressed and provide their final merit score. For “Category B” papers, the PC members read the rebuttal and provide their final merit score.  The reviewers have approximately 10 days to key in their post-response scores.


6B. Online Discussion Before PC Meeting (Aug 21-26) 

The reviewers discuss papers online and revise their scores. By default, papers in Category B are unlikely to be discussed at the PC meeting, unless someone advocates for the paper. The online discussion can promote a paper in Category B to be discussed at the PC meeting. All papers in Category A will get discussed at the PC meeting, however it is understood that the rank of any paper may get lowered if the reviewers reduce their scores based on all available information.

The PC chair will lock the submission website on Aug 27-28, to compile the rank ordered list of all papers based on post response rankings, in preparation for the PC meeting.


7. In-Person PC Meeting in Atlanta (Aug 29-30) 

The PC meeting will be held in Atlanta. We will have a 1.5 days PC meeting. The papers will be discussed in rank ordered arrangement based on the post-response merit scores. The author names will remain shielded throughout the process to maintain an unbiased process. We are also looking into using buzzers for the voting during the PC meeting to ensure a fair, unbiased, and quick voting. 

8. Final Decision to Authors (Sep 3) 

The accept/reject decision will be communicated to the authors on Thursday, Sept 3.

Disclaimer

This is a compile-time list of process and policy decisions, and the policy decisions may need to be fine-tuned based on runtime events. Keeping in the spirit of compiler-microarchitecture interaction of MICRO, the PC chair will strive to develop the best static decisions before the process starts, but override these decisions if the runtime information demands as such. All changes in key decisions will be communicated to the steering committee.

If you have any feedback or suggestions, please email the Program Chair at [email protected].



Frequently Asked Questions/Concerns about the New Review Model: 

1. Do we have enough time to be able to do this?

Yes. The typical review period between submission and rebuttal for MICRO is 11 weeks. For HPCA it is about 7 weeks. So, MICRO enjoys a 4 week longer review period. We have  pulled the submission deadline for MICRO by one week, as the submission deadline for MICRO does not clash with results of a prior conference. We have thus been able to fit the 3-week revision window in almost the regular timeline of MICRO.


2. Isn’t this too much work for the PC members?

We have worked on policy decision to reduce the extra work.  The restriction of limiting the revision to only Top80 papers was done primarily to reduce the added work from revision.   For example, if 300 papers get submitted then roughly one-fourth of the papers will be eligible for revision. For example, out of the stack of 20 papers for the PC member, only 5 will be recommended for revision, on average. We can safely assume that the top 2 or so will not need (a major) revision, so it is really generates into reading the revised version of the remaining 3-4 papers which the PC members have already reviewed once. We believe this is not as much extra work for the PC members, especially if it helps them make up their mind, seeing their concerns have been addressed. Furthermore, these papers will get discussed at the PC meeting, so the extra discussion and time spent on these papers will hopefully lead to more meaningful discussions at the PC meeting.


3. Isn’t this too much work for the authors?

Only the Top 80 papers are invited for revision.  This means that the paper has a pretty good shot at getting accepted, so we are providing an additional opportunity for these papers to make their case by revising their papers in response to the reviewer's concerns.  However, revision is optional, and  the papers will not be penalized if they chose to not submit a revision (or rebuttal for that matter). For the papers that are not in
the Top 80, there is no option for revision (unless there is a champion), so we would recommend a rebuttal only,  so the process remains the same as what we have right now for these papers.


4. Isn’t three weeks too short for the authors to revise their papers?

The duration between decision and camera ready for MICRO 2014 was about 4 weeks. So, if the concern took more than 4 weeks to address, then the authors would not have been able to put it in their camera ready version anyway. So, the 3-week period misses out on the class of concerns that would have been possible in four weeks, but not three weeks, which we are inclined to believe are fairly uncommon.


5. Is there a control on what gets added to the revised version?

We will have guideline stating that the authors are not allowed to include any unsolicited new material in the revised version (for example, it is not okay to write a cache paper first and “revise” it to a branch prediction paper). The only purpose of the revision is to address the questions pointed by the reviewers and possibly to enhance the editorial quality of the paper.


6. Are we encouraging people to submit sloppy papers, given that there is a second shot?

The paper has to be in the PC-meeting discussion range (Top 80) in the pre-revision stage to get a recommendation for revision. For example, if the paper ranks in bottom half of all submitted papers, then this new process will degenerate into what we have right now — the authors submit a rebuttal and the decision will be based on the reviews and rebuttal. We are hopeful that the revised process will encourage the authors to submit the best version of the work, so that they can be in the Top 80, and get a recommendation for revision.


7. “With the proposed scheme authors might spend significant time revising the paper only to have it ultimately rejected, which could be a little disheartening” – PC member

Yes, that is true, and the sentiment would be understandable. It would be important to set the expectations right to avoid/reduce the disappointment. We plan to communicate to the authors that an invitation for revision is not a guarantee that the paper will be accepted. There are about 80 papers invited for revision, and historically about 50-55 papers get accepted. If the authors address the reviewers concerns, then they will increase the likelihood of the papers getting accepted. Even if the paper does not get accepted after revision, the authors will still have a stronger version of the paper to submit to a later conference (e.g. HPCA), so their effort in revision is not wasted. Similarly, if the paper is accepted, then the authors now need to do that much less work for the camera ready version. So, in either case, the effort in revision is not wasted effort -- the effort just gets shifted in time, and hopefully with the added benefit of increasing the likelihood that the paper gets accepted at MICRO 2015.


8. Are we converting our conference into a journal? (“I see this as a bug and not as a feature”)

We are just trying to get the best elements of the review process wherever we can find them, and fit them in our time constraints. Therefore, we are limiting the revision to only the papers are likely to be in PC meeting, and the iterative revision is limited to one round too (inspired in part again from VLDB conference, where after one round of revision the authors get a decision of either an accept, or do not submit till next year). With the proposed change, our conference still remains a conference, but hopefully with a much stronger review process.