About ML Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles

An ongoing multistakeholder initiative to enable responsible AI by increasing transparency and accountability with machine learning system documentation.

About the Project

ABOUT ML (Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles) is a multi-year, multi-stakeholder initiative led by PAI. This initiative aims to bring together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices at scale.

The initiative is an ongoing, iterative process designed to co-evolve with the rapidly advancing field of AI development and deployment. In recognition that documentation is both an artifact and a process, ABOUT ML is structured into an artifact workstream and a process workstream. In 2020, ABOUT ML will produce one resource for each workstream:

Artifact Workstream:

A database of documentation questions, adapted by the domain of the machine learning application

Process Workstream:

A research-based guide to initiating and scaling a documentation pilot.

Documentation for machine learning systems can contribute to responsible AI development by bringing more transparency into “black box” models and by bridging the gap between increasingly pervasive AI ethics principles and day-to-day operations and practice. Documentation can shape practice because by asking the right question at the right time in the AI development process, teams will become more likely to identify potential issues and take appropriate mitigating actions.

Get Involved

Contribute to 2020 work

Our goal for 2020 is to design testable pilots in a multi stakeholder manner. To make this process more tractable, we’ve broken this into two different workstreams. Each button below leads to a key subtask in this process, and we invite you to share your thoughts, comments, and feedback on any that you are interested in.

Process workstream

Help us solve the challenge of how documentation can be created at scale within an organization.

Artifact workstream

Join the debate on what information stakeholders deserve to know about ML systems, and how that information should be presented. 

Future work

Deployed Examples

See real-world deployed examples of ML documentation which can focus on datasets, models, and ML systems. Provide your feedback on these examples as part of ABOUT ML’s public feedback comment process.

To guide ABOUT ML, let the steering committee know what you think of these examples. Which questions are useful? What questions are these examples missing? Is there anything about the format of one of these examples that is effective? Submit a comment below.

Thanks for the submission!

Steering Committee

The ABOUT ML Steering Committee is comprised of around 30 experts, researchers and practitioners recruited from a diverse set of PAI Partner organizations. The Steering Committee guides the process of updating ABOUT ML drafts based on the public comments submitted and new developments in research and practice. They vote to approve new releases by “rough consensus” commonly used by other multi-stakeholder working groups. They convene 1-3 times a year, depending on the volume of proposed changes and velocity of change of research and practice.

To allow for as many diverse perspectives as possible, PAI limits participation in the Steering Committee to up to 2 people per organization, with 1 vote per organization. As needed, PAI will periodically reopen applications to the Steering Committee and recruit more members.

Current Steering Committee members:

Norberto Andrade

Privacy and Public Policy Manager (FACEbook)

Amir Banifatemi

General Manager, Innovation & Growth (XPRIZE)

Rachel Bellamy

Principal Researcher & Manager, Human-AI Collaboration (IBM)

Umang Bhatt

Student Fellow (Leverhulme Centre for the Future of Intelligence)

Rumman Chowdhury

Managing Director 
(Accenture AI)

Jacomo Corbo

Chief Scientist (QuantumBlack)

Daniel First

Associate / Data Scientist (McKinsey / QuantumBlack)

Ben Garfinkel

Research Fellow (Future of Humanity Institute)

Jeremy Gillula

Tech Projects Director 
(EFF)

Brenda Leong

Senior Counsel and Director 
of Strategy (Future of Privacy Forum)

Tyler Liechty

Data Engineer (DeepMind)

Momin M. Malik

Data Scientist 
(Berkman Klein Center)

Lassana Magassa

Graduate Research Associate (Tech Policy Lab/University of Washington)

Meg Mitchell

Researcher, ML Fairness, Ethical AI (Google)

Amanda Navarro

Managing Director (PolicyLink)

Deborah Raji

Tech Fellow 
(AI Now)

Nicole Rigillo

Anthropologist 
(Berggruen Institute)

Andrew Selbst

Postdoctoral Scholar 
(Data & Society)

Ramya Sethuraman

Product Manager (Facebook)

Reshama Shaikh

Board Member, NY Co-Organizer 
(WiMLDS)

Moninder Singh

Research Staff Member 
(IBM)

Amber Sinha

Senior Programme Manager (Centre for Internet and Society)

Michael Spranger

Senior Research Scientist, AI 
Collaboration Office (Sony)

Andrew Strait

Researcher, Ethics and 
Society Team (DeepMind)

Gabriel Straub

Head of Data Science and Architecture (BBC)

Michael Veale

Assistant Professor (UCL)

Hanna Wallach

Senior Principal Researcher (Microsoft)

Adrian Weller

Senior Research Fellow (Leverhulme Centre for the Future of Intelligence)

Abigail Wen

Managing Counsel, Office of the CTO (Intel)

Alexander Wong

Co-Director, Vision and Image Processing (VIP) Research Group (University of Waterloo)

Jennifer Wortman Vaughan

Principal Researcher (Microsoft)

Andrew Zaldivar

Senior Developer Advocate (Google)

ABOUT ML Resources

Access ABOUT ML’s library of materials to learn more about the project, join the discussion, and see the current draft.