The decision-making criteria of computer expert systems are often so complex that they are beyond the comprehension of their individual users and creators. For example, computer systems equipped with artificial intelligence can be used for estimating the degree to which somebody is likely to default on her credit. These days, computer programs can also be used for the large-scale monitoring of populations and for attempts at predicting who is more or less likely to become a criminal or a political dissident.
When computers start to decide who is likely to be a threat and who isn’t and neither secret services, law enforcement nor the subjects of surveillance understand how a threat assessment comes about, the shared understanding of what constitutes suspicious behaviour gets lost. Writing in the Intercept Dan Froomkin cites Phillip Rogaway, a professor of computer science at the University of California, Davis:
If the algorithms NSA computers use to identify threats are too complex for humans to understand, Rogaway wrote, “it will be impossible to understand the contours of the surveillance apparatus by which one is judged. All that people will be able to do is to try your best to behave just like everyone else. 1
If people don’t understand the criteria by which they are judged anymore, one can still find it reasonable to use such computer systems. Yet, their “suspicion algorithms” themselves don’t express human reasoning anymore. People become subject to a governance by statistical probabilities instead of human value choices 2. The computers may not rule as they don’t possess true agency yet. Still, humans delegate their assessment of who is an insider and who an outsider, of who is a friend and a potential foe to systems whose calculations are beyond their comprehension. Reasoning about an essentially political decision is transferred to machines 3.
The data is there, the algorithms set in place. An ethics of the data age has yet to emerge.