Seriously thoughtful Weapons of Math Destruction examples

[j.p.s]

Originally Posted by GA Russell

I have seen this statement before (with an "h"!! ), but I don't know what it means.

Can you please give us an example of what a weaponized algorithm does, and its creator's intention?

Thanks!

I am opening this thread here since it has nothing to do with Deals and I am not a member of P&R or debaters and don't care to debate this anyway.

Criminal risk assesment algorithms are one example. They have several noble motivations, such as concious and unconcious human bias reduction, factors considered increase, human error decrease, drudgery decrease, productivity increase, cost reduction, fairness increase, consistancy increase, etc. But reqirements specifiers, managers, and coders are all subject to bad assumptions, ignorance, misunderstanding, miscommunication, errors, bias, malice, and laziness. Typically these come to light when alogorithms generate obvious erroreous results, suspicious results, or results in conflict with the interest of those using the algorithm and so the problems get addressed. When the results are as expected, but actually incorrect, nothing gets looked into and incorrect data do not get discovered and eliminated.

Some things that get high weight are prior records, proximity to high crime areas, and association with criminals. But "good" people get stuck in undesirable neighborhoods because of low income, skin color, or other reasons out of their control. Many end up getting caught in a death spiral of higher and higher computed criminal risk since police use these assesments to select areas to concentrate patrols, which leads to people being stopped for interrogation over and over, fraying nerves, reacting badly, resulting in arrest and more risk points, missing work for court appearances, losing jobs, and even more poverty. The details of the algorithms are often declared proprietary and are kept secret from their victims and even the algorithm customers.

This is all off the top of my head and is quite incomplete and does not do the full explanation justice. The book does a much better job showing the errors and the weponizing as well as many different types of WMDs with their many different types of victims. (Nobody escapes the grasp of several WMDs.)

The weapon part comes in when the algorithm becomes a tool that justifies and enables the weapon wielders to do what they wre already doing with even more justification, blessing, speed, and efficiency. Win, win, win for everybody involved but the victims.

Other examples are loan and job application assesments and credit scores.

[GA Russell]

Thanks, jps!

Are there instances where the algorithm is created to be a weapon, or is it always a case of an innocent algorithm being used inappropriately?

[j.p.s]

Originally Posted by GA Russell

Thanks, jps!

Are there instances where the algorithm is created to be a weapon, or is it always a case of an innocent algorithm being used inappropriately?

You're welcome. I'm not happy with the clarity of my description of the death spiral aspect.

Credit scores had good intentions, but a good chunk of the finanacial applications are designed as weapons from the get go.

Another example that tries really hard to be good but does harm is testing like ACT, SAT, and IQ testing. No one intentionally uses them as weapons but they have their victims. (Also the automation is not the weapon part.)