Making Algorithms Public: Reimagining Auditing from Matters of Fact to Matters of Concern

Stakeholders concerned with bias, discrimination, and fairness in algorithmic systems are increasingly turning to audits, which typically apply generalizable methods and formal standards to investigate opaque systems. We discuss four attempts to audit algorithmic systems with varying levels of success-depending on the scope of both the system to be audited and the audit's success criteria. Such scoping is contestable, negotiable, and political, linked to dominant institutions and movements to change them. Algorithmic auditing is typically envisioned as settling "matters-of-fact" about how opaque algorithmic systems behave: definitive declarations that (de)certify a system. However, there is little consensus about the decisions to be automated or about the institutions automating them. We reposition algorithmic auditing as an ongoing and ever-changing practice around "matters-of-concern." This involves building infrastructures for the public to engage in open-ended democratic understanding, contestation, and problem solving-not just about algorithms in themselves, but the institutions and power structures deploying them. Auditors must recognize their privilege in scoping to "relevant" institutional standards and concerns, especially when stakeholders seek to reform or reimagine them.