Geoff Hinton Dismissed The Need For Explainable AI: 8 Experts Explain Why He's Wrong


A heap of questions..Depositphotos enhanced by CogWorld

If the expectation is that automation will be ubiquitous in the next decade, reliance on human judgement will diminish. The promises of Artificial Intelligence are met with cautious optimism as the technology evolves and the environment around it attempts to keep pace. In this nascent period, industry, academia and lawmakers are grappling with the outcomes of these technologies and their impact on our social norms. The domino effect here created by AI will alter all facets of policy, technology and society.

In a recent interview with Wired Geoff Hinton, distinguished computer scientist, Head of Google Brain, and renowned for his work with Artificial Neural Networks, stated this when prompted about AI's eventual role in decision-making:

I’m an expert on trying to get the technology to work, not an expert on social policy. One place where I do have technical expertise that’s relevant is [whether] regulators should insist that you can explain how your AI system works. I think that would be a complete disaster.

People can’t explain how they work, for most of the things they do. When you hire somebody, the decision is based on all sorts of things you can quantify, and then all sorts of gut feelings. People have no idea how they do that. If you ask them to explain their decision, you are forcing them to make up a story.

Neural nets have a similar problem. When you train a neural net, it will learn a billion numbers that represent the knowledge it has extracted from the training data. If you put in an image, out comes the right decision, say, whether this was a pedestrian or not. But if you ask “Why did it think that?” well if there were any simple rules for deciding whether an image contains a pedestrian or not, it would have been a solved problem ages ago.

... [In response to how do we trust systems?] You should regulate them based on how they perform. You run the experiments to see if the thing’s biased, or if it is likely to kill fewer people than a person. With self-driving cars, I think people kind of accept that now. That even if you don’t quite know how a self-driving car does it all, if it has a lot fewer accidents than a person-driven car then it’s a good thing. I think we’re going to have to do it like you would for people: You just see how they perform, and if they repeatedly run into difficulties then you say they’re not so good.

I reached out to notable practitioners in AI/ML industry, government and academia to weigh in on his arguments:  Ann Cavoukian, Rumman Chowdhury, Joy Buolamwini, Karen Bennet, Tim Miller, Heather Roff, Alejandro Saucedo and David Gunning. This article curates this discourse:

Let's break this down:

Hinton: "I’m an expert on trying to get the technology to work, not an expert on social policy. One place where I do have technical expertise that’s relevant is [whether] regulators should insist that you can explain how your AI system works. I think that would be a complete disaster."

For sometime, debate's ensued to determine who is liable for societal or individual harms resulting from algorithmic flaws: The data scientist, the machine or the company? For Dr. Heather Roff, Associate Fellow from the Leverhulme Centre for the Future of Intelligence, University of Cambridge, the responsibility needs to be broadened. She asserts,

This is a dangerous position to take. An expert on technology who feels themselves divorced from social or policy implications does not understand that technology is not value neutral, and that their decisions—even seemingly basic ones on how many gradient descents to take in a system — have socio-political implications. If one thinks they are only Scientists doing Science, but then simultaneously think that regulators should take an interest has fundamentally misunderstood their role as scientists engaging in socially and morally important questions. If your work requires legislation then you should think about that at the design stage… period.

Dr. Rumman Chowdhury, Managing Director and Global Lead for Responsible AI at Accenture agrees with Roff and addresses the very harms that today's AI experimentation exhibit. The fallout is societal. The solution needs to be a holistic collaboration between technology and policy.

We cannot divorce 'making things work' and 'impact on society' when it comes to applied artificial intelligence. Frankly, your AI does not "work" if it is biased, perpetuates social inequality and discrimination, or reinforces unequal power structures. Setting up that delineation is not only dangerous, it sets up a false dichotomy of "tech innovators" versus "regulators." Regulation, whether in the form of social norms, guidelines, or enforceable law, is intended to enable trust and ease adoption of technology in a way that is beneficial to society. Safe innovation is enabled with well designed regulation. 

Hinton: "People can’t explain how they work, for most of the things they do... People have no idea how they do that. If you ask them to explain their decision, you are forcing them to make up a story."

Timothy Miller, Associate Professor in computer science at the University of Melbourne, Australia, whose specializes in explainable AI and human-agent collaboration, disputes Hinton's claim on the limitations of human explanation:

His quoted paragraph is itself an explanation: an explanation of why he has reached the decision that explainability for AI would be a disaster. Is he making up a story about this? I imagine he would claim that he is not and that it is based on careful reasoning. But in reality, it is based on neurons in his brain firing in a particular way that nobody understands. The ability to communicate his reasons to others is a strength of the human brain. Philosopher Daniel Dennett claims that consciousness itself is simply our brain creating an `edited digest’ of our brains inner workers for precisely the purpose of communicating our thoughts and intentions (including explanations) to others.

Hinton: "Neural nets have a similar problem. When you train a neural net, it will learn a billion numbers that represent the knowledge it has extracted from the training data. If you put in an image, out comes the right decision, say, whether this was a pedestrian or not. But if you ask “Why did it think that?” well if there were any simple rules for deciding whether an image contains a pedestrian or not, it would have been a solved problem ages ago."

So if humans are to slowly cede control to autonomous algorithms, it will increasingly become difficult to understand what led to those decisions. Deep Learning requires minimal supervision and with enough training data can identify patterns from the data it accesses. As Hinton notes, eventually it becomes more difficult, even for the architects of this algorithm to understand the specific reasons for those outcomes. Dr. Ann Cavoukian, Distinguished Expert-in-Residence, leading the Privacy by Design Centre of Excellence at Ryerson University and former three-term Privacy Commissioner of Ontario, agrees with this.... somewhat:

...humans can externalize features that define a dog... however with current deep learning algorithms, although they may initially decompose an image into relevant features, and then recompose those features back into an image for categorization, these features are implicit to the algorithm, buried in the myriad numbers of parameter values. Current algorithms cannot externalize these features and use them to explain a decision. What is needed are algorithms that construct “wholes” from previously learned “parts/features” such that the features are also external to the algorithm that is making the decision.

...there is a second process taking place: there is a meta-algorithm in the brain that is able to view the process of decision-making and collect the sequence of features that were involved in the decision, and based on those, output the explanation. Again, this cannot be done with existing deep learning because the features are implicit– in the parameter values. Moreover, any one parameter value may affect features associated with categories other than “dog,” in the above example.

For Heather Roff, comparing humans to neural nets is not a "true equivalence":

...it is false equivalence. We can interrogate and probe human beings as to why they did X or Y. We even claim that we have AI based “lie detectors” to use micro facial expressions to show when someone is being untruthful. So why should we think that AI can save us from our worst selves but also accept that we cannot as humans figure these same things out? Designers must figure out what to measure, what data is important or relevant and the like, and AIs right now are not able to do that themselves. So to claim that humans are inherently opaque and non-transparent and that justifies us using other intelligence that are actually more opaque and inherently nonhuman-like in their reasoning as a justified argument is a false equivalence. Humans have a theory of mind. AIs right now do not. I don’t have a sense of what another being like me may think, if I’m an AI. I DO have that as a human being. And this excuse — as a an attempted justification at using tech that we don’t understand fully — is a red herring.

For the DOD, where precision in aspects of  war require investigation and justification, David Gunning introduces the work being done on explainable ML that will allow future warfighters to "understand, appropriately trust and manage an emerging generation of AI Machine partners" :

There are techniques to explain deep nets: DARPA’s Explainable AI (XAI) program, and a growing community of researchers, are developing techniques that can be used to explain, at least partially, deep nets: (1) there are techniques that can select the training examples that were most influential in a decision; (2) there are techniques to identify the most salient input features used in a decision; (3) there are network dissection techniques that can identify meaning features inside the layers of a deep net that can be used for explanation; and: (4) there are deep learning researchers who are developing deep learning techniques to generate explanations. 

Alejandro Saucedo, Chief Scientist, The Institute for Ethical AI & Machine Learning, partially agrees with Hinton in that it's not possible to open complex systems or models and provide a thorough explanation, however only focusing on the algorithm itself and trying to understand the value of each weight and its interaction with the outcomes is short-sighted.

AI explainability cannot be addressed by solely looking at it as a technological challenge. It requires consideration of the processes, infrastructure and even humans (yes, humans) operating the algorithms themselves.

It is possible to reach a reasonable level of explainability and accountability by ensuring the right touchpoints with domain experts are in place throughout the development and operation of AI systems. Sometimes this may involve a trade-off between explainability and accuracy, but it may be required depending on the critical nature of the project. A reasonable level of explainability can only be achieved through cross-functional collaboration across technology experts, industry practitioners and policy-makers.

At the Institute for Ethical AI, we are empowering industry practitioners to find this reasonable level of explainability with the AI Procurement Framework we released this year. It provides professionals with the tools to evaluate the maturity of their machine learning systems through a checklist which highlights red flags around processes and infrastructure.

Hinton: "You should regulate them based on how they perform. You run the experiments to see if the thing’s biased, or if it is likely to kill fewer people than a person. With self-driving cars, I think people kind of accept that now. That even if you don’t quite know how a self-driving car does it all, if it has a lot fewer accidents than a person-driven car then it’s a good thing. I think we’re going to have to do it like you would for people: You just see how they perform, and if they repeatedly run into difficulties then you say they’re not so good."

Reactive legislation cannot and should not be the panacea for technology going forward. Today, rampant harms as a result of existing bias in systems, unproven technology raise ethical concerns about their deployment. Recidivism, self-driving cars, social bots and facial recognition technologies released into the ether have created a growing movement to determine thresholds for accuracy with a check on legal and societal acceptances before they are commercialized. Joy Buolamwini, MIT Media Lab, Graduate Researcher and Founder, Algorithmic Justice League, who has extensive experience in facial recognition technology has been a strong proponent for government action:

Not only do we need to look at biased outcomes we need to look at bias from design, development, deployment, and governance of AI. Even systems that show decreased of harmful technical bias can be deployed in ways that breach civil liberties and human rights. Take the example of facial analysis systems which my research has shown to have substantial racial and gender bias. Companies have been working to reduce this technical bias, but there still needs to be accountability about how the technology is used and who it is sold to. Given everything we know about facial analysis technology failures it should not be used for lethal applications. Yet companies like Microsoft and Amazon appear poised to apply the technology for fatal military use if left unchecked. Even though the social impact of facial recognition technology is ill understood, companies are equipping law enforcement departments with this technology with no legal oversight or meaningful accountability. We need explanations for irresponsible use of facial analysis technology and a moratorium on life or liberty threatening applications. This is why we launched the Safe Face Pledge to prevent the lethal use and mitigate abuse of facial analysis technology.

In business, justification behind decisions is a standard practice. In the fallout of Enron, the genesis of laws like Sarbanes Oxley were formed for the purpose of protecting the public and the business from fraud or errors. Explainability has been the business norm. It's our system of checks and balances. Ann Cavoukian, who served in policy for three consecutive terms as Privacy Commissioner disagrees with regulation as a result of performance:

I don’t share Geoff’s view of after-the-fact regulation. Technology is simply moving far too fast, and regulations, in this day and age, are a lagging remedy. We must be pre-emptive and proactively build-in explainability. However, to implement useful explainability will require different artificial architectures from the existing ones.

Rumman Chowdhury goes a step further beyond explainability and towards understandable AI for law, business and society:

In talking about "explainability" as a false fix, Hinton is raising a discussion those of us in the ethics and AI space have addressed already. We agree, that explaining the activation functions of nodes, or a technical explanation of decisions, is not useful. What we seek is actionable understanding. Depending on the implementation (whether human or algorithmic), there are decisions that need an explanation, and we work to create understandable and actionable explanations to AI outcomes. This means, not only creating systems of explanations that are useful to humans, but creating systems of addressing and redressing potential issues and harms.

Karen Bennet, Principle at ArCompany (disclosure) and former senior engineering executive at Yahoo!, Redhat and IBM provides a pragmatic perspective that supports Chowdhury's understandable AI, and builds more accountability within data science and engineering:

In all the organizations that I have been working with, there is a disconnect between the people who are building predictive models and those who know how to best serve the organization's objectives. There are laws, standards or regulations that an organization must adhere to, so we do need to solve the problem of making an intelligence system accountable, so it can be audited and trusted to do the right thing. In the financial industry, e.g., before software is deployed live, there is an exhaust checklist that must be adhered to get approval; and one of them is explaining to a non-techie what the software does. It is not enough to just say look at the results. As incidents happen, the judicial system will also require these explanations. I suggest we look at Intelligent systems in (4) categories: Research/Human in Loop; Applied/Human in Loop; Research/Autonomous and Applied/Autonomous, as each has different requirements.

AI adoption comes with trust in our systems, with the belief that humanity's interests are given full consideration in AI's decisions. We're not there yet. In retrospect, this discourse is necessary to bring the issues of technology and all its impacts to the forefront. The division that exists between research and regulation will quickly dissipate as AI slowly wields itself into all facets of our lives.