The dangers of algorithms

“Algorithms are not facts. They are opinions,” said Cathy O’Neil during a talk on the dangers of algorithms titled “Weapons of Math Destruction.”

O’Neil has a doctorate in mathematics and worked as a data scientist creating algorithms until she started to see the problem with them. Her past enables a distinct perspective; when she asks people for the specifics of algorithms, they cannot hide behind convoluted math jargon.

The talk focused on how algorithms are black boxes—neither the company using them nor the people making them take responsibility for how they can wreck lives and reinforce stereotypes.

One of her prominent examples was how Sarah Wysocki—a teacher who taught in D.C.—was fired because her students didn’t learn as predicted by an algorithm. O’Neil pointed out that an algorithm set to judge a student’s performance is unreliable because economic status also plays a major role. One of the slides she showed was a graph with the scores of teachers who taught various subjects, such as seventh and eighth grade math. If the algorithm were correctly analyzing the teacher’s performance, the result would appear consistent, but the graph ended up looking random. O’Neil and her colleagues asked if they could see the algorithm. However, no one knew where the actual math was coming from. O’Neil pointed this out as a prime example of people using an algorithm and not taking responsibility for it.

She also illustrated how many people have no idea of the problems associated with certain algorithms because there is no knowledge of what they are even being judged on. She showed this with the case of a person diagnosed with bipolar disorder, who—when applying for jobs—saw multiple similarities between the tests for his diagnosis and his job application. The person sued, because they thought that they were being rejected based on their mental diagnosis, which is illegal under the Americans with Disabilities Act of 1990.

O’Neil didn’t even discuss any of the social media algorithms or the power of Facebook, Google, or Amazon. She said that if she did, that the talk would go on for way too long. However, she did mention that most algorithms benefit society or are benign.

This talk scared me. I always knew that basically every possible entity was tracking my data exhaust and plugging it into algorithms, but I thought these entities would just provide advertisements that are my “taste” or recommend certain videos, not get me fired from jobs or even prevent me from acquiring them in the first place.

This talk opened my eyes to many of the algorithms that aren’t discussed as frequently and how that can be incredibly dangerous. No one is taking responsibility for them and that’s even scarier for me. The people making the algorithms don’t take responsibility for users of their code and the people using algorithms place responsibility on the creators.

O’Neil furthered my fears when she said that any laws to have oversight would be fought by every single lobbying group, because it is in every industry’s interest to have “black box” algorithms, to allow them to avoid responsibility. O’Neil opened my eyes to the power of algorithms and how dangerous they can be when unchecked.