Difference between revisions of "ACL 2017 materials"
| (28 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
In this page, we give links to two supplemental materials that we think will be helpful for other PC chairs to consider: | In this page, we give links to two supplemental materials that we think will be helpful for other PC chairs to consider: | ||
| + | # START Email templates - these are important as the START system has too generic templates and needs to be customised. We have recorded all emails sent out and list the templates here which you can adapt for your reuse. | ||
# Template Google Spreadsheets - to work with the downloaded START materials. Using the spreadsheet tools in START you can download all of the data about submissions, users (reviewers), acceptances, and reviews, which are all important for building the necessary statistics for reporting (and in our case, for disclosing to the general public about transparency) | # Template Google Spreadsheets - to work with the downloaded START materials. Using the spreadsheet tools in START you can download all of the data about submissions, users (reviewers), acceptances, and reviews, which are all important for building the necessary statistics for reporting (and in our case, for disclosing to the general public about transparency) | ||
| − | + | ||
| + | Also we received a script from the ICML 2017 PC Co-Chairs (Doina Precup and Yee Whye Teh) to help with checking for overlaps. The Python script calculates a TF.IDF form over the title and abstracts (where available) and does a brute force O(n^2) similarity scoring among all of the submissions from both sources. You can find the script [[doublecheck_v5.py]] here. | ||
| + | == START Email Templates == | ||
| + | |||
| + | If you want these templates as a basis to start, you might find it easier to just email support@softconf.com and ask Rich or Paolo to populate these into your START installation from the ACL 2017 START archive. | ||
| + | |||
| + | # [[2017_ACL_2017:AC Info]] | ||
| + | # [[2017_ACL_2017:Update for ACs]] | ||
| + | ## [[2017_ACL_2017:START Invitation]] | ||
| + | # [[2017_ACL_2017:Reminder Information]] | ||
| + | # [[2017_ACL_2017:Survey and TPMS Reminder]] | ||
| + | # [[2017_ACL_2017:AC reminder for reviewers]] | ||
| + | # [[2017_ACL_2017:Invitation the the event, second]] | ||
| + | # [[2017_ACL_2017:AC to start checking]] | ||
| + | # [[2017_ACL_2017:ACs to start checking, 2nd]] | ||
| + | ## [[2017_ACL_2017:170208-acl17-inviteGeneric.txt]] | ||
| + | # [[2017_ACL_2017:ACs at bidding stage]] | ||
| + | # [[2017_ACL_2017:ACs before assignment]] | ||
| + | # [[2017_ACL_2017:ACs before assignment, 2]] | ||
| + | # [[2017_ACL_2017:TPMS]] | ||
| + | # [[2017_ACL_2017:AC, first reminder to reviewers]] | ||
| + | # [[2017_ACL_2017:AC 2nd update for discussion period]] | ||
| + | # [[2017_ACL_2017:Author Response Period: Inform Authors]] | ||
| + | |||
| + | Post acceptance correspondence | ||
| + | # [[2017_ACL_2017:ACs: Reviewers on responses]] | ||
| + | # [[2017_ACL_2017:ACs to Rank]] | ||
| + | ## [[https://docs.google.com/spreadsheets/d/1TRXTbO6E5Mc2ZRedkT2OZwEOMqNI2R1f6b4sQqBXW_g/edit#gid=0 Google Doc AC Ranking Template]] | ||
| + | # [[2017_ACL_2017:ACL program: Oral Monday]] - this is a sample; we did one for each day of the conference (We had accept statuses: Oral Monday, Oral Tuesday, Oral Wednesday, Poster Monday, Poster Tuesday, TACL) | ||
| + | # [[2017_ACL_2017:AC for session chairs]] | ||
| + | # [[2017_ACL_2017:Reminders to all authors]] | ||
| + | # [[2017_ACL_2017:ACs proofcheck reviewers listing]] | ||
| + | # [[2017_ACL_2017:AC vet schedule, best papers]] | ||
| + | # [[2017_ACL_2017:Presenter Information]] | ||
| + | # [[2017_ACL_2017:AC Thank you]] | ||
| + | |||
| + | == Google Sheets Templates == | ||
| + | |||
| + | START also allows downloads of the database tables from which it computes its processes. These are useful for plugging into Google Sheets for your own checking. We advise this method because it makes any intermediate computation visible and transparent. | ||
| + | |||
| + | There's even an START API to programmatically download this data afresh, but we didn't use this. I've added the information from Sebastian Riedel who informed us about this. | ||
| + | |||
| + | <blockquote>'' | ||
| + | Whenever I was ACing, I always ended up working on a collaborative Google sheet, and while it was easy to download CSV spreadsheets from START and the populate the sheets with it, the sheets would quickly go stale (for example, because reviewers change their scores), and re-uploading and copying old information in the new upload was very tedious. This plugin addresses this issue, and a few more---for example, it generates publication specific START URLs that make it easy to navigate from the sheet to START. It only became a possibility this year, since the START folks added a web-service interface to the START content. | ||
| + | |||
| + | In any case, the instructions for the plugin are here: | ||
| + | |||
| + | [https://github.com/uclmr/softconf-start-sync https://github.com/uclmr/softconf-start-sync] | ||
| + | |||
| + | It's worthwhile pointing out that the plugin only reads from START, so it can't really corrupt its state in any way.'' | ||
| + | </blockquote> | ||
| + | |||
| + | There are some times when you need to share information out of one spreadsheet to another for public dissemination. You need to duplicate the original spreadsheet to create a separate copy, delete the pertinent, un-anonymized data, and then reduplicate it to discard spreadsheet history. | ||
| + | |||
| + | * [https://docs.google.com/spreadsheets/d/1bV0qmpNGk3BKYSfy037TIaA1zUVOA943aSqdwGTzG0I/edit?usp=sharing Invites to Reviewer Areas ] worksheet | ||
| + | * [https://docs.google.com/spreadsheets/d/1bFYSGm-FZkQN_5D1HSuoQRtO1l8Yhsz2DG1Mnx1m03Y/edit?usp=sharing Reviews to Decisions ] worksheet | ||
| + | * [https://docs.google.com/spreadsheets/d/1feTCYp6fX-xNSgiq0X5hw53zW7CY3hWEmYN3jKE3Moc/edit?usp=sharing Decisions to Program ] Worksheet | ||
Latest revision as of 18:12, 6 September 2017
Regina Barzilay and Min-Yen Kan
For ACL 2017, we pioneered a few innovations that we have documented as part of the PC Chairs Blog. We highly recommend reading through it and the ACL 2017 report (the section marked Innovations) as these contain reports about the successful and unsuccessful processes.
In this page, we give links to two supplemental materials that we think will be helpful for other PC chairs to consider:
- START Email templates - these are important as the START system has too generic templates and needs to be customised. We have recorded all emails sent out and list the templates here which you can adapt for your reuse.
- Template Google Spreadsheets - to work with the downloaded START materials. Using the spreadsheet tools in START you can download all of the data about submissions, users (reviewers), acceptances, and reviews, which are all important for building the necessary statistics for reporting (and in our case, for disclosing to the general public about transparency)
Also we received a script from the ICML 2017 PC Co-Chairs (Doina Precup and Yee Whye Teh) to help with checking for overlaps. The Python script calculates a TF.IDF form over the title and abstracts (where available) and does a brute force O(n^2) similarity scoring among all of the submissions from both sources. You can find the script doublecheck_v5.py here.
START Email Templates
If you want these templates as a basis to start, you might find it easier to just email support@softconf.com and ask Rich or Paolo to populate these into your START installation from the ACL 2017 START archive.
- 2017_ACL_2017:AC Info
- 2017_ACL_2017:Update for ACs
- 2017_ACL_2017:Reminder Information
- 2017_ACL_2017:Survey and TPMS Reminder
- 2017_ACL_2017:AC reminder for reviewers
- 2017_ACL_2017:Invitation the the event, second
- 2017_ACL_2017:AC to start checking
- 2017_ACL_2017:ACs to start checking, 2nd
- 2017_ACL_2017:ACs at bidding stage
- 2017_ACL_2017:ACs before assignment
- 2017_ACL_2017:ACs before assignment, 2
- 2017_ACL_2017:TPMS
- 2017_ACL_2017:AC, first reminder to reviewers
- 2017_ACL_2017:AC 2nd update for discussion period
- 2017_ACL_2017:Author Response Period: Inform Authors
Post acceptance correspondence
- 2017_ACL_2017:ACs: Reviewers on responses
- 2017_ACL_2017:ACs to Rank
- 2017_ACL_2017:ACL program: Oral Monday - this is a sample; we did one for each day of the conference (We had accept statuses: Oral Monday, Oral Tuesday, Oral Wednesday, Poster Monday, Poster Tuesday, TACL)
- 2017_ACL_2017:AC for session chairs
- 2017_ACL_2017:Reminders to all authors
- 2017_ACL_2017:ACs proofcheck reviewers listing
- 2017_ACL_2017:AC vet schedule, best papers
- 2017_ACL_2017:Presenter Information
- 2017_ACL_2017:AC Thank you
Google Sheets Templates
START also allows downloads of the database tables from which it computes its processes. These are useful for plugging into Google Sheets for your own checking. We advise this method because it makes any intermediate computation visible and transparent.
There's even an START API to programmatically download this data afresh, but we didn't use this. I've added the information from Sebastian Riedel who informed us about this.
Whenever I was ACing, I always ended up working on a collaborative Google sheet, and while it was easy to download CSV spreadsheets from START and the populate the sheets with it, the sheets would quickly go stale (for example, because reviewers change their scores), and re-uploading and copying old information in the new upload was very tedious. This plugin addresses this issue, and a few more---for example, it generates publication specific START URLs that make it easy to navigate from the sheet to START. It only became a possibility this year, since the START folks added a web-service interface to the START content.
In any case, the instructions for the plugin are here:
https://github.com/uclmr/softconf-start-sync
It's worthwhile pointing out that the plugin only reads from START, so it can't really corrupt its state in any way.
There are some times when you need to share information out of one spreadsheet to another for public dissemination. You need to duplicate the original spreadsheet to create a separate copy, delete the pertinent, un-anonymized data, and then reduplicate it to discard spreadsheet history.
- Invites to Reviewer Areas worksheet
- Reviews to Decisions worksheet
- Decisions to Program Worksheet