About ML Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles

Process and Timeline

PAI is launching the ABOUT ML iterative multi-stakeholder process with this initial draft v0 in order to provide a starting place for community response. The goal is to update this draft into future releases and move towards best practices by going through the following phases:

• Understand the latest research
• Combine research theory and results of current practice into testable pilots
• Run pilot tests with PAI Partners and organizations
• Collect data from pilot tests for transparency practices
• Iterate on pilots with the latest research and practice
• When there is sufficient body of evidence for a certain practice, elevate it to a best practice
• Promulgate effective transparency practices

PAI recognizes that this effort can only succeed with input from as broad a set of stakeholders as possible, and will be seeking input not only from our 90+ partners, but also from stakeholders from academia, civil society organizations, companies designing and deploying ML technology, and the general public. We welcome your participation.

The process is modeled after iterative ongoing processes to design internet standards (such as W3C, IETF, WHATWG) and includes a public forum for discussion and a place to submit any proposed changes. We welcome you to join in the public discussion and to submit proposed changes as many times as you’d like.

Public comments will be collected and batch evaluated by the ABOUT ML Steering Committee, a group of ~30 experts, researchers and practitioners recruited from a diverse set of PAI Partner organizations. The Steering Committee will be guiding the process of updating ABOUT ML drafts based on the public comments submitted and new developments in research and practice. They will vote to approve new releases by “rough consensus” commonly used by other multi-stakeholder working groups. They will convene 1-3 times a year, depending on the volume of proposed changes and the velocity of change of research and practice.

To prevent contributors from commenting on an out-of-date draft, the document will be closed to proposed changes for ~2-4 weeks prior to each Steering Committee meeting. This closure will be announced here with at least 2 weeks of notice. The first meeting of the Steering Committee will be on September 28th, 2019, so the v0 draft will be closed for comment from September 14th until the end of the update process.

To ensure that diverse perspectives— especially those from communities historically excluded from technology decision-making—contribute to any ABOUT ML recommendations, PAI is engaging with our Partner, the Tech Policy Lab at the University of Washington to conduct Diverse Voices panels. This method was designed to gather feedback from stakeholders whose perspectives might not otherwise be consulted and to ensure that those perspectives are reflected in the released text. Thus, for any ABOUT ML releases that go through the Diverse Voices process, the panel feedback will be the last edits incorporated before a new release. This also means that each round of Diverse Voices panels will cause public comment on the document to be paused for several months, although the public forum will remain open for discussion during that time. Public comment on the document itself will re-open with the new release of the draft. The first round of Diverse Voices panels for ABOUT ML will be held between October and December 2019. Thus, draft v0 will be closed to public comment from September 14th, 2019 until early 2020, and will reopen upon the release of draft v1 in early 2020. Additional panels will be convened approximately once a year, especially for milestone releases when the ABOUT ML project progresses through the phases outlined above.