Difference between revisions of "ACL 2017 materials"

From Admin Wiki
Jump to navigation Jump to search
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
Regina Barzilay and Min-Yen Kan
  
 +
For ACL 2017, we pioneered a few innovations that we have documented as part of the [https://chairs-blog.acl2017.org/ PC Chairs Blog].  We highly recommend reading through it and the [https://www.aclweb.org/adminwiki/index.php?title=2017Q3_Reports:_Program_Chairs ACL 2017 report] (the section marked ''Innovations'') as these contain reports about the successful and unsuccessful processes.
 +
 +
In this page, we give links to two supplemental materials that we think will be helpful for other PC chairs to consider:
 +
 +
# START Email templates - these are important as the START system has too generic templates and needs to be customised.  We have recorded all emails sent out and list the templates here which you can adapt for your reuse.
 +
# Template Google Spreadsheets - to work with the downloaded START materials.  Using the spreadsheet tools in START you can download all of the data about submissions, users (reviewers), acceptances, and reviews, which are all important for building the necessary statistics for reporting (and in our case, for disclosing to the general public about transparency)
 +
 +
Also we received a script from the ICML 2017 PC Co-Chairs (Doina Precup and Yee Whye Teh) to help with checking for overlaps.  The Python script calculates a TF.IDF form over the title and abstracts (where available) and does a brute force O(n^2) similarity scoring among all of the submissions from both sources.  You can find the script [[doublecheck_v5.py]] here.
 
== START Email Templates ==
 
== START Email Templates ==
  
Line 25: Line 34:
 
# [[2017_ACL_2017:ACs: Reviewers on responses]]
 
# [[2017_ACL_2017:ACs: Reviewers on responses]]
 
# [[2017_ACL_2017:ACs to Rank]]
 
# [[2017_ACL_2017:ACs to Rank]]
−
# [[2017_ACL_2017:ACL program: Oral Monday]] - this is a sample; we did one for each day of the conference                            
+
## [[https://docs.google.com/spreadsheets/d/1TRXTbO6E5Mc2ZRedkT2OZwEOMqNI2R1f6b4sQqBXW_g/edit#gid=0 Google Doc AC Ranking Template]]
 +
# [[2017_ACL_2017:ACL program: Oral Monday]] - this is a sample; we did one for each day of the conference (We had accept statuses: Oral Monday, Oral Tuesday, Oral Wednesday, Poster Monday, Poster Tuesday, TACL)                           
 
# [[2017_ACL_2017:AC for session chairs]]
 
# [[2017_ACL_2017:AC for session chairs]]
 
# [[2017_ACL_2017:Reminders to all authors]]
 
# [[2017_ACL_2017:Reminders to all authors]]
Line 32: Line 42:
 
# [[2017_ACL_2017:Presenter Information]]
 
# [[2017_ACL_2017:Presenter Information]]
 
# [[2017_ACL_2017:AC Thank you]]
 
# [[2017_ACL_2017:AC Thank you]]
 +
 +
== Google Sheets Templates ==
 +
 +
START also allows downloads of the database tables from which it computes its processes.  These are useful for plugging into Google Sheets for your own checking.  We advise this method because it makes any intermediate computation visible and transparent.
 +
 +
There's even an START API to programmatically download this data afresh, but we didn't use this.  I've added the information from Sebastian Riedel who informed us about this.
 +
 +
<blockquote>''
 +
Whenever I was ACing, I always ended up working on a collaborative Google sheet, and while it was easy to download CSV spreadsheets from START and the populate the sheets with it, the sheets would quickly go stale (for example, because reviewers change their scores), and re-uploading and copying old information in the new upload was very tedious. This plugin addresses this issue, and a few more---for example, it generates publication specific START URLs that make it easy to navigate from the sheet to START. It only became a possibility this year, since the START folks added a web-service interface to the START content.
 +
 +
In any case, the instructions for the plugin are here:
 +
 +
[https://github.com/uclmr/softconf-start-sync https://github.com/uclmr/softconf-start-sync]
 +
 +
It's worthwhile pointing out that the plugin only reads from START, so it can't really corrupt its state in any way.''
 +
</blockquote>
 +
 +
There are some times when you need to share information out of one spreadsheet to another for public dissemination.  You need to duplicate the original spreadsheet to create a separate copy, delete the pertinent, un-anonymized data, and then reduplicate it to discard spreadsheet history.
 +
 +
* [https://docs.google.com/spreadsheets/d/1bV0qmpNGk3BKYSfy037TIaA1zUVOA943aSqdwGTzG0I/edit?usp=sharing Invites to Reviewer Areas ] worksheet
 +
* [https://docs.google.com/spreadsheets/d/1bFYSGm-FZkQN_5D1HSuoQRtO1l8Yhsz2DG1Mnx1m03Y/edit?usp=sharing Reviews to Decisions ] worksheet
 +
* [https://docs.google.com/spreadsheets/d/1feTCYp6fX-xNSgiq0X5hw53zW7CY3hWEmYN3jKE3Moc/edit?usp=sharing Decisions to Program ] Worksheet

Latest revision as of 18:12, 6 September 2017

Regina Barzilay and Min-Yen Kan

For ACL 2017, we pioneered a few innovations that we have documented as part of the PC Chairs Blog. We highly recommend reading through it and the ACL 2017 report (the section marked Innovations) as these contain reports about the successful and unsuccessful processes.

In this page, we give links to two supplemental materials that we think will be helpful for other PC chairs to consider:

  1. START Email templates - these are important as the START system has too generic templates and needs to be customised. We have recorded all emails sent out and list the templates here which you can adapt for your reuse.
  2. Template Google Spreadsheets - to work with the downloaded START materials. Using the spreadsheet tools in START you can download all of the data about submissions, users (reviewers), acceptances, and reviews, which are all important for building the necessary statistics for reporting (and in our case, for disclosing to the general public about transparency)

Also we received a script from the ICML 2017 PC Co-Chairs (Doina Precup and Yee Whye Teh) to help with checking for overlaps. The Python script calculates a TF.IDF form over the title and abstracts (where available) and does a brute force O(n^2) similarity scoring among all of the submissions from both sources. You can find the script doublecheck_v5.py here.

START Email Templates

If you want these templates as a basis to start, you might find it easier to just email support@softconf.com and ask Rich or Paolo to populate these into your START installation from the ACL 2017 START archive.

  1. 2017_ACL_2017:AC Info
  2. 2017_ACL_2017:Update for ACs
    1. 2017_ACL_2017:START Invitation
  3. 2017_ACL_2017:Reminder Information
  4. 2017_ACL_2017:Survey and TPMS Reminder
  5. 2017_ACL_2017:AC reminder for reviewers
  6. 2017_ACL_2017:Invitation the the event, second
  7. 2017_ACL_2017:AC to start checking
  8. 2017_ACL_2017:ACs to start checking, 2nd
    1. 2017_ACL_2017:170208-acl17-inviteGeneric.txt
  9. 2017_ACL_2017:ACs at bidding stage
  10. 2017_ACL_2017:ACs before assignment
  11. 2017_ACL_2017:ACs before assignment, 2
  12. 2017_ACL_2017:TPMS
  13. 2017_ACL_2017:AC, first reminder to reviewers
  14. 2017_ACL_2017:AC 2nd update for discussion period
  15. 2017_ACL_2017:Author Response Period: Inform Authors

Post acceptance correspondence

  1. 2017_ACL_2017:ACs: Reviewers on responses
  2. 2017_ACL_2017:ACs to Rank
    1. [Google Doc AC Ranking Template]
  3. 2017_ACL_2017:ACL program: Oral Monday - this is a sample; we did one for each day of the conference (We had accept statuses: Oral Monday, Oral Tuesday, Oral Wednesday, Poster Monday, Poster Tuesday, TACL)
  4. 2017_ACL_2017:AC for session chairs
  5. 2017_ACL_2017:Reminders to all authors
  6. 2017_ACL_2017:ACs proofcheck reviewers listing
  7. 2017_ACL_2017:AC vet schedule, best papers
  8. 2017_ACL_2017:Presenter Information
  9. 2017_ACL_2017:AC Thank you

Google Sheets Templates

START also allows downloads of the database tables from which it computes its processes. These are useful for plugging into Google Sheets for your own checking. We advise this method because it makes any intermediate computation visible and transparent.

There's even an START API to programmatically download this data afresh, but we didn't use this. I've added the information from Sebastian Riedel who informed us about this.

Whenever I was ACing, I always ended up working on a collaborative Google sheet, and while it was easy to download CSV spreadsheets from START and the populate the sheets with it, the sheets would quickly go stale (for example, because reviewers change their scores), and re-uploading and copying old information in the new upload was very tedious. This plugin addresses this issue, and a few more---for example, it generates publication specific START URLs that make it easy to navigate from the sheet to START. It only became a possibility this year, since the START folks added a web-service interface to the START content.

In any case, the instructions for the plugin are here:

https://github.com/uclmr/softconf-start-sync

It's worthwhile pointing out that the plugin only reads from START, so it can't really corrupt its state in any way.

There are some times when you need to share information out of one spreadsheet to another for public dissemination. You need to duplicate the original spreadsheet to create a separate copy, delete the pertinent, un-anonymized data, and then reduplicate it to discard spreadsheet history.

Retrieved from "https://www.aclweb.org/adminwiki/index.php?title=ACL_2017_materials&oldid=72173"