Toronto Paper Matching System
The current system
The goal of this system is to help conference chairs with the task of assigning papers to reviewers. Our approach is to build a profile for each reviewer using that reviewer's self-selected list of publications. Using the profile we calculate the suitability of the reviewer to each submitted paper. The system is integrated with the Microsoft conference management system (CMT), which lets it easily access the conference submissions.
The reviewer-paper suitabilities, in addition to other information such as the keywords used in CMT, can then be used to help assign reviewers to papers. The assignment process entails maximizing the total reviewer-paper suitabilities, while satisfying certain constraints, in particular load balancing. Such a constrained assignment does not guarantee that reviewers will be assigned to the papers for which they have the highest scores. The only guarantee is that the assignment will be globally optimal, for some chosen objective, given the scores and constraints set in place by the program chairs.
We ask reviewers to register on our site, and to provide a list of publications, which are used to create your profile for any participating conference. As a reviewer, you have the option of including your full set of publication, or to select a subset which reflects the topics of papers you wish to review for the given conference. The selected papers can include tech reports and notes. Registration on this service also lets you keep a permanent record of your selected publications. You will always be able to log in to the system to update your list of selected publications.
Some conferences also use the service to help select papers on which reviewers and area chairs can bid on. In some cases these are not bids in the standard use of the term, but rather more like queries, in that their aim is to help the system estimate that reviewer's suitability for other submitted papers. For the same reasons outlined above we emphasize that bidding highly on a paper does not guarantee that you will get to review it.
The system is described in details by the following two papers:
Paper describing the system (ICML'13 workshop paper) Paper describing some of the initial models (UAI'11).
This service as well as the underlying software are built by Laurent Charlin and Richard Zemel. This effort is partly funded by participating conferences, and the hosting resources are offered by the Computer Science Department of the University of Toronto.
The idea first came when Richard Zemel was the Co-Program-Chair for NIPS'10. We first developed an offline system that was successfully used both in NIPS'10 and NIPS'11. CVPR'12 also used a restricted version of the system.
We have now built an online version of the system that is integrated with Microsoft's CMT. This new version lets reviewers maintain a permanent profile that can be shared across conferences. ICML in 2012 was the first conference to use the online version of the system and was quickly followed by ECCV, UAI, NIPS, CPVR, ICCV, ICML, AISTATS and others.
As of June 2016 the system has now been used by over 50 conferences and keeps the profiles of 8,000 reviewers.
Using the system
If you would like to use the system for your conference please contact us.