Network access is blocked in this session, so I could not verify URLs live. I’m restricting citations to primary sources whose URLs I’m highly confident in and dropping the borderline ones.
Yes. Algorithms perpetuate human bias, and Cathy O’Neil’s sharper point is that they do it at scale, in secret, and with a veneer of objectivity that a biased person never gets. A model learns from historical data, and historical data records who got hired, arrested, lent to and treated. Train on that record and the model reproduces it, then applies it to millions of people with no appeal. O’Neil, a Harvard-trained mathematician who left a hedge fund after the 2008 crash, named these systems “weapons of math destruction” in her 2016 book. Her test is simple: a model is dangerous when it is opaque, operates at scale, and damages the people it scores. The question is never whether a system is biased, but whether anyone has measured it.
Who is Cathy O’Neil, and why does her argument carry weight?
O’Neil earned a PhD in mathematics at Harvard, taught at Barnard, then built models at D.E. Shaw during the financial crisis. She watched credit-default models treated as truth while they were wrong, and later saw the same deference to models in ad tech. That experience is the spine of Weapons of Math Destruction and of her 2018 TED Radio Hour interview with NPR, where she argued that algorithms are “opinions embedded in code.”
She is not a critic from the sidelines. Through her firm ORCAA she audits live algorithms for companies and regulators, and her TED talk remains the clearest ten-minute explanation of why “the data says so” is not a defense.
How does a “neutral” algorithm end up biased?
Four mechanisms account for nearly every documented case.
- Training data encodes past decisions. A hiring model trained on ten years of résumés learns who was hired, not who was good.
- Proxies smuggle in protected traits. Zip code tracks race. Healthcare spending tracks access to care, not sickness. A model forbidden from using race will find race anyway.
- The objective is chosen by the owner. A model that optimizes for cost, clicks or arrests will sacrifice fairness unless fairness is in the objective.
- Feedback loops confirm the model. Send police where the model predicts crime, record the arrests, retrain. The prediction manufactures its own evidence.
The U.S. National Institute of Standards and Technology catalogues these failure modes, plus the human and institutional biases around them, in NIST Special Publication 1270.
Where has algorithmic bias been documented?
| Sector | Case | What went wrong |
|---|---|---|
| Criminal justice | COMPAS risk scores, ProPublica 2016 | Black defendants who did not reoffend were nearly twice as likely as white defendants to be labeled high risk. |
| Healthcare | Obermeyer et al., Science 2019 | A model used for about 200 million patients a year used cost as a proxy for need. Black patients had to be considerably sicker than white patients to get the same score. |
| Hiring | Amazon’s recruiting tool, Reuters 2018 | Trained on a decade of mostly male applicants, it penalized résumés containing the word “women’s.” Amazon scrapped it. |
| Computer vision | Gender Shades, Buolamwini and Gebru 2018 | Commercial gender classifiers erred on under 1% of light-skinned men and up to 35% of dark-skinned women. |
| Speech | Koenecke et al., PNAS 2020 | Five major speech-to-text systems had roughly double the word error rate for Black speakers compared with white speakers. |
What does this look like inside a video production company?
We do not build scoring models, but we hand finished work to several every week. Three of them bite regularly.
Ad delivery. When we run a recruiting or real-estate spot on Meta or YouTube, the delivery algorithm decides who actually sees it, and it optimizes from the creative itself. We have watched a trades-recruiting video skew heavily male in the first 48 hours with no gender targeting set, because the platform learned that men clicked first. Meta now forces housing, credit and employment campaigns into a restricted category that strips age, gender and zip targeting. That protects the client legally but does not stop delivery skew, so we check the audience breakdown at day two and, if needed, split the creative and cap frequency rather than trust the optimizer.
Auto-captions. Every deliverable gets captions, and the first pass is automatic. The PNAS numbers above match what we see: interviews with Black speakers, non-native English speakers, or anyone on a lavalier in a reverberant room come back with far more errors. We budget 15 to 30 minutes of human caption review per finished minute for interview-heavy pieces, and closer to 5 for scripted voiceover. Shipping the raw auto-caption is how a client’s own employees end up misquoted.
Face detection and auto-reframe. Camera autofocus, editing-software auto-reframe and thumbnail generators all run on face detectors with the same skin-tone gap Gender Shades measured. The fix is on set, not in software: light and expose for the darkest skin in frame, and never let a 9:16 auto-crop go out without a human watching every cut.
What has changed since O’Neil’s 2018 interview?
The argument won. Regulation and standards now assume bias exists and demand proof otherwise. The EU Artificial Intelligence Act entered into force in August 2024 and classifies hiring, credit, education and law-enforcement systems as high risk, with obligations for bias testing, documentation and human oversight phasing in through 2026 and 2027. In the United States, the NIST AI Risk Management Framework, released in January 2023, is the reference most procurement teams now cite, and New York City has required bias audits of automated hiring tools since July 2023.
The tools changed too. The 2018 conversation was about résumé screeners and risk scores. Today the same questions apply to large language models that draft job descriptions, summarize applicants and generate ad copy, and those models inherit bias from a far larger and less documented corpus.
What actually reduces algorithmic bias?
- Audit outcomes, not intentions. Measure error rates and selection rates by group on real decisions. A diverse engineering team helps, but Amazon’s tool failed in a company that knew the risk.
- Interrogate the proxy. Ask what the model actually predicts. “Cost” is not “need” and “clicks” is not “fit.”
- Keep an appeal path. O’Neil’s test for a weapon of math destruction includes the absence of recourse. A human who can override the score is a design requirement, not a courtesy.
- Treat vendors as auditable. If a tool’s maker cannot show group-level test results, assume the results are bad.
If you use any scoring, ranking or delivery system in hiring, lending, advertising or health, pull its decisions for the last quarter and break them down by the groups that matter legally. Where you cannot get that data, read O’Neil’s book and the NIST framework before renewing the contract.
