Any kind of fallacy might occur anywhere, thus also in science and AI research. For the latter, a fallacy will often refer to an AI system as the culprit in the given situation. This article covers three interesting fallacies that are commonly known within the research community. The following, and first, story is an urban legend, that is, a story that has been told many times - and in fact might very likely be wrong [1]:

(…) The Army trained a program to differentiate American tanks from Russian tanks with 100% accuracy. Only later did analysts realized that the American tanks had been photographed on a sunny day and the Russian tanks had been photographed on a cloudy day. The computer had learned to detect brightness. (…)

The snippet is taken from [2] and Figure 1 visualizes the concrete fallacy found in the story.

Figure 1: Urban Legend: The Tank Story. The AI model (depicted as robot, bottom) was provided with image data depicting tanks, russian and U.S. ones. The AI model learned perfectly to decide on the training set, but when promted on new images that have not been observed before, the model failed miserably. It turned out that the model learned to classify the weather consistenly i.e., it decided because of wrong “reasons”, statistics that most easily accomplish the task of discrimination between the data classes.

While the truth of events for the story depicted in aboves figure is questionable, the fallacy is real. Like in any programmable system with sufficient freedom (whether this means Turing-completeness [3] is an interesting but different question), by the programmer unintended behaviors can occur. In this case, the system completes its task perfectly, however, only the exact formulation of the task (i.e., discriminating the given data perfectly) and not the intended formulation of the task (i.e. using the particular features exposed on the images by each of the tanks to discriminate). Thereby, it might even be wrong to call this scenario a fallacy since the system behaves expectedly, making it a fallacy of the human developer. Similar to the tank legend type of fallacy, in the transition from the 19th to 20th centuries a horse named Hans was deemed “smart” because when queried with arithmetic questions, e.g. what is the solution to 5 + 5, then Hans would stomp his hooves 10 times [4]. For an interactive hommage to the Clever Hans story, consider [5]. The conclusion to Hans’s intelligence is the ability to “read the human body language” which gives statistically speaking very predictive cues onto the right solution. Figure 2 shows a historic scene depicting the horse, Hans:


Figure 2: The clever horse named Hans. The image shows Hans in 1909 solving an arithmetic question. Hans is an Orlow-Traber horse, a race from Russia for which breeding began end-19th century.

The story of Clever Hans, opposed to the urban legend on tank classification, is in fact only symbolic for AI research. More specifically, AI systems have shown Clever Hans like behavior [6] and thus this story has become part of common fallacies in AI. Ironically, to improve on future AI systems, data sets motivated by the Clever Hans story were developed [7]. While the two stories covered so far have been fallacies arguably outside of the system of investigation, the following story is indeed a fallacy of that very system. I.e., adversarial attacks are a substantial part of today’s research regarding stability/robustness in learning systems [8]. The Fast Gradient Sign Method (FSGM), originally introduced by [9], is a method that simply adds a miniscule noise gradient to the original input, to produce an image unnoticeably different to the human eye from the original, which then however produces a completely different prediction from the original where the semantic impact can be severe. E.g., a very confident prediction of a panda might now be classified as a gibbon (monkey race) with high confidence, although the human’s perception remains invariant across the two images. Figure 3 illustrates the attack:


Figure 3: Adversial Examples. Figure 1 from [9] that shows an adversarial example for a “panda” image. The neural classifier will subsequently produce a wrong prediction with high confidence.

Adversarial attacks come in different flavors, and new attacks, but also defences, are being discovered regularly. This last fallacy, in adversarials, that we have covered is the most severe of the three in terms of sensitivity while the former two reveal problems more related to the final goal of artificial intelligence: general intelligence. How future fallacies might look like in future systems, or possibly even in current systems yet to be discovered, is left to be seen.

References

Gwern, “The Neural Net Tank Urban Legend”, 2019.
H. Murphy, “Why Stanford Researchers Tried to Create a ‘Gaydar’ Machine”, 2017.
Wikipedia, “Turing completeness”, 2021.
Wikipedia, “Kluger Hans”, 2021.
The Aperiodical, “Ask Clever Hans a question”, 2015.
S. Lapuschkin et al., “Unmasking Clever Hans predictors and assessing what machines really learn”, nature communications, 2019.
J. Johnson et al., “CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning”, arXiv, 2016.
Xu et al., “Adversarial Attacks and Defenses in Images, Graphs and Text: A Review”, arXiv, 2019.
Goodfellow et al., “Explaining and Harnessing Adversarial Examples”, arXiv, 2014.

Fallacies in AI