The danger of predictive algorithms in criminal justice

01How risk assessment algorithms work

A criminal-justice risk tool usually combines inputs—such as age, prior records, court history, or supervision information—and maps them to an outcome observed in historical data. The output may be a score, a category, or a recommendation. It is not a crystal ball; it is a statistical estimate whose meaning depends on the population, the outcome chosen, and the decisions that produced the training data.

That last point is easy to miss. “Recidivism” can mean rearrest, reconviction, or a technical violation, and each reflects policing and supervision choices as well as behavior. A model can be accurate at predicting the recorded outcome while saying much less about the underlying conduct policymakers actually care about.

02The ProPublica COMPAS investigation

In 2016, ProPublica examined the COMPAS system used in parts of the United States and reported racial disparities in false-positive and false-negative rates. The company disputed the analysis, arguing that the score satisfied a different fairness criterion. The ensuing debate became important precisely because multiple mathematical definitions of fairness can conflict when groups have different base rates.

The lesson is not that one article settled every question about COMPAS. It is that a proprietary score can become consequential while outsiders cannot inspect its variables, validation population, thresholds, or error rates. A fairness claim without a clearly defined outcome and an independently reproducible test is not enough.

03Racial bias in predictive policing

Predictive-policing systems often use historical reports, calls for service, arrests, or locations as signals. But those records are not a neutral census of crime. They also record where police were sent, which neighborhoods were watched, who was stopped, and which behaviors were enforced. Feeding those records back into patrol allocation can create a feedback loop: more patrol produces more recorded incidents, which “confirms” the model's focus.

This does not mean every statistical pattern is invented. It means the pattern's pathway into the dataset matters. A system that predicts police activity may be very good at predicting police activity while being marketed as a prediction of public safety. Those are different claims with different ethical stakes.

04The due process problem

A person can often challenge a witness, a document, or a police procedure. Challenging a score is harder when the vendor treats the model as a trade secret, the agency cannot explain its operation, or the defense receives only a label such as “high risk.” Due process requires meaningful notice and an opportunity to contest the reasons that affect detention, bail, sentencing, or supervision.

Opacity is not the only problem. Even a fully disclosed model can be inappropriate if its proxies encode protected characteristics, its data are stale, or its threshold produces unacceptable harms. Due process is therefore about contestability as well as source-code access: someone must be able to explain the decision, correct bad inputs, and obtain review from an accountable human authority.

05Algorithmic transparency and accountability

Transparency begins before procurement. Agencies should publish the purpose of a system, its legal authority, data sources, target outcome, validation results, subgroup performance, threshold choices, and a log of decisions influenced by it. Independent auditors need access to the model or a reliable testing interface, and affected people need a plain-language explanation rather than a vendor brochure.

Accountability also requires monitoring after launch. Error rates can drift when policy changes, policing patterns shift, or the population differs from the development sample. A meaningful governance process includes impact assessments, incident reporting, appeal routes, sunset dates, and the power to pause a tool when it causes unanticipated harm.

06Alternatives to algorithmic justice

The alternative to a risky model is not necessarily intuition. Judges and agencies can use structured, transparent checklists; invest in counsel, treatment, housing, and reminders; reduce unnecessary pretrial detention; and publish aggregate outcomes so communities can evaluate policy. Human decision-making should not be romanticized: people carry bias too, and unreviewable discretion can be as damaging as an opaque score.

Some jurisdictions are also narrowing the role of prediction. A tool may be limited to scheduling support or service referrals rather than detention or punishment. That is a policy choice worth making explicitly: use technology to expand capacity and reduce harm, not to make a contested judgment look inevitable.

07What fair AI in criminal justice looks like

Fair AI starts with a narrow, publicly defensible purpose and a demonstration that automation improves the relevant decision rather than merely accelerating it. It reports uncertainty, disaggregates errors, protects sensitive data, and gives the affected person a usable way to challenge inputs and outcomes. Most importantly, it leaves final authority with institutions that can be held responsible.

There may be contexts where a validated tool helps allocate scarce services. There are also contexts where the risk of laundering unequal enforcement through software is too high. The right question is not whether an algorithm is biased in the abstract. It is whether this system, for this decision, with this data and these safeguards, produces a benefit that survives public scrutiny.