About ML Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles

An ongoing multistakeholder initiative to enable responsible AI by increasing transparency and accountability with machine learning system documentation.

About the Project

ABOUT ML (Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles) is a multi-year, multi-stakeholder initiative led by PAI. This initiative aims to bring together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices at scale.

The initiative is an ongoing, iterative process designed to co-evolve with the rapidly advancing field of AI development and deployment. In recognition that documentation is both an artifact and a process, ABOUT ML is structured into an artifact workstream and a process workstream.

Blog post

Read the Project Overview (PDF)

Additional Resources

Get Involved

Start Here
Contribute to 2021 work

Our goal for 2021 is to design testable pilots in a multi stakeholder manner. To make this process more tractable, we’ve broken this into two different workstreams. Each button below leads to a key subtask in this process, and we invite you to share your thoughts, comments, and feedback on any that you are interested in.

Process workstream

Help us solve the challenge of how documentation can be created at scale within an organization.

Artifact workstream

Join the debate on what information stakeholders deserve to know about ML systems, and how that information should be presented. 

Future work

Deployed Examples

See real-world deployed examples of ML documentation which can focus on datasets, models, and ML systems. Provide your feedback on these examples as part of ABOUT ML’s public feedback comment process.

Google, Jigsaw Perspective API

Model-focused

Microsoft’s Face API Transparency Note

API documentation

IBM, FactSheets

Model or System-focused

Google, Model Cards

Model-focused

OpenAI, GPT-2

Model-focused

IBM, Image Caption Generator

Model-focused

Share More Examples

Submit

To guide ABOUT ML, let the steering committee know what you think of these examples. Which questions are useful? What questions are these examples missing? Is there anything about the format of one of these examples that is effective? Submit a comment below.

Thanks for the submission!

Steering Committee

The ABOUT ML Steering Committee is comprised of around 30 experts, researchers and practitioners recruited from a diverse set of PAI Partner organizations. The Steering Committee guides the process of updating ABOUT ML drafts based on the public comments submitted and new developments in research and practice. They vote to approve new releases by “rough consensus” commonly used by other multi-stakeholder working groups. They convene 1-3 times a year, depending on the volume of proposed changes and velocity of change of research and practice.

To allow for as many diverse perspectives as possible, PAI limits participation in the Steering Committee to up to 2 people per organization, with 1 vote per organization. As needed, PAI will periodically reopen applications to the Steering Committee and recruit more members.

Current Steering Committee members:

Himani Agrawal

Data Scientist (AT&T Research Labs); Organizer (Women in Machine Learning and Data Science)

Norberto Andrade

Privacy and Public Policy Manager (Facebook)

Amir Banifatemi

General Manager, Innovation & Growth (XPRIZE)

Rachel Bellamy

Principal Researcher & Manager, Human-AI Collaboration (IBM)

Umang Bhatt

Student Fellow (Leverhulme Centre for the Future of Intelligence)

Jacomo Corbo

Chief Scientist (QuantumBlack)

Daniel First

Associate / Data Scientist (McKinsey / QuantumBlack)

Ben Garfinkel

Research Fellow (Future of Humanity Institute)

Jeremy Gillula

Tech Projects Director 
(EFF)

Jerremy Holland

Director of AI Research (Apple)

Sara Jordan

Senior Counsel, AI & Ethics (Future of Privacy Forum)

Joohyung Lee

Corporate VP / Head of Lab (Samsung)

Brenda Leong

Senior Counsel and Director 
of Strategy (Future of Privacy Forum)

Tyler Liechty

Data Engineer (DeepMind)

Lassana Magassa

Graduate Research Associate (Tech Policy Lab/University of Washington)

Meg Mitchell

AI/ML Researcher

Amanda Navarro

Managing Director (PolicyLink)

Sinead O'Brien

AI/ML Engagement Lead (BBC)

Irina Raicu

Director, Internet Ethics program (Markkula Center for Applied Ethics)

Deborah Raji

Fellow (Mozilla Foundation)

Becca Ricks

Researcher (Mozilla Foundation)

Andrew Selbst

Postdoctoral Scholar 
(Data & Society)

Ramya Sethuraman

Product Manager (Facebook)

Moninder Singh

Research Staff Member 
(IBM)

Spandana Singh

Policy Analyst (Open Technology Institute)

Amber Sinha

Senior Programme Manager (Centre for Internet and Society)

Michael Spranger

Senior Research Scientist, AI 
Collaboration Office (Sony)

Andrew Strait

Associate Director of Research Partnerships (Ada Lovelace Institute)

Hanna Wallach

Senior Principal Researcher (Microsoft)

Adrian Weller

Senior Research Fellow (Leverhulme Centre for the Future of Intelligence)

Abigail Wen

Author & Filmmaker

Alexander Wong

Co-Director, Vision and Image Processing (VIP) Research Group (University of Waterloo)

Jennifer Wortman Vaughan

Senior Principal Researcher (Microsoft)

Andrew Zaldivar

Senior Developer Advocate (Google)

ABOUT ML Resources

Access ABOUT ML’s library of materials to learn more about the project, join the discussion, and see the current draft.