Explainable AI
Explainability has multiple facets to it. We usually assume explainability as the ability of an AI model to explain the process behind a decision or output.

Explaining Explainable AI

Artificial intelligence is enabling advanced decision-making capabilities. The deep layers of a neural network can mimic the functions of a human mind. When human beings make a decision; they are able to explain how they arrived at it – thought process, the observations, logical thinking, that went into making that decision.

Basic machine algorithms such as decision trees can be easily explained by following the path of the tree. However, complex artificial intelligence or machine learning algorithms that have deep layers are incomprehensible. Data scientists find it difficult to explain why their algorithm gave a particular outcome. Due to a lack of comprehensibility, the end-user finds it equally difficult to trust the machine’s decisions.


Explainability has multiple facets to it. We usually assume explainability as the ability of an AI model to explain the process behind a decision or output. But it isn’t just that! Explainability comprises of the individual models and the larger systems in which they are incorporated. It refers to whether a model’s outcome can be interpreted and whether the entire process and intention around the model can be accounted for.

Therefore, explainability should encompass three key aspects that the AI platform or system should explain:

  • The intent behind how the AI system impacts the users
  • The data sources that are used and how the outcomes are audited
  • How the data inputs result in specific outputs

Why Explainability is necessary

Explainability is driven by the lack of transparency – what is commonly known as the Blackbox problem. Blackbox approach to AI evokes distrust and a general lack of acceptability of the technology. Parallelly there’s a rising focus for legal and privacy aspects that will have a direct impact on AI and ML technology. For example, the European General Data Protection Regulations (GDPR) will make it difficult for businesses to take a Blackbox approach.

Another example is IBM Watson. The world was astounded when it beat the best human players in a game of jeopardy. However, when IBM began marketing their AI technology to hospitals to help detect cancer, it saw a lot of resistance. It was alright if Watson couldn’t provide the reasoning behind a move that allowed it to win the game. But diagnosing cancer wasn’t a game and couldn’t be taken as lightly. Neither the doctors nor the patients were able to trust the technology because it lacked the ability to provide reasons for the results. Even if its results were the same as that of the doctor’s it couldn’t provide a diagnosis.

“IBM’s attempt to promote its supercomputer program to cancer doctors (Watson for Oncology) was a PR disaster. The problem with Watson for Oncology was that doctors simply didn’t trust it.”

Vyacheslav Polonski, Ph.D., UX researcher for Google and founder of Avantgarde Analytics.

Another hotly debated topic is the use of AI for military purposes. Advocates of the lethal autonomous weapon systems (LAWS) claim that using AI will cause less collateral damage. However, despite the availability of large volumes of training data that will help the LAWS distinguish a civilian from a combatant or a non-target from a target, it is highly risky to leave the decision to AI.


Source: Arms Control Association

One of the AI projects of the CIA is AI-enabled drones. The extent of explanation of the AI software for the selection of targets is only 95%. That 5% is left to chance and leaves room open for a lot of controversy and debate on racism, bias, or stereotype issues.

Methods to develop Explainable AI systems

There are two sets of techniques that are used to develop explainable AI (XAI).

Post-hoc methods

Post-hoc is Latin for ‘after this’, a method in which AI models are built normally and explainability is incorporated only during the testing phase.


Local Interpretable Model-Agnostic Explanations (LIME) is a prominent post-hoc technique. It works by influencing the features within a single prediction instance. For example, the technique perturbs features of the pixels of an image of a cat to define which pixel segments contribute the most to the AI model’s classification of that image. Perhaps, the model classified the image of the cat as a dog because the cat had dropping ears.


Source: integrate.ai

In 2015, it was reported that Google Photos labeled images of a black developer and his black friend as “gorillas”. In such a scenario, LIME could be used to mitigate this kind of bias by having a human operator override such biased decisions by evaluating the reasons provided by the algorithm. There are other post-hoc techniques such as Layer-wise Relevance Propagation (LRP) that can also be used to solve such classification errors caused by bias.

Ante-Hoc Methods

Ante-hoc techniques entail baking explainability into a model from the beginning.


Bayesian deep learning (BDL) is a popular ante-hoc technique. It is a great way to add uncertainty handling to the AI model. It helps gauge a neural network’s level of uncertainty about its predictions. By leveraging the hierarchical representation power of deep learning, the architecture can model complex tasks. The idea of BDL is straight forward – instead of having to learn point estimates for each parameter, we learn the parameters of gaussian for each parameter during forward propagation. To learn the parameters, backpropagation is used to make the parameters differentiable.


Source: Towards Data Science

Bayesian Neural Networks can be trained by beginning with flat gaussian distributions for the trainable parameters of the neural networks. For each batch during the training loop, sample the weights according to the current distributions, forward pass with the weights, then back propagate the loss to the parameter. The advantage of this approach is, that for each output we also get the prediction certainty, which will help the user to judge whether to trust the result.

Reversed time attention model, alternatively known as RETAIN, is a model developed by researchers at Georgia-Tech to help doctors understand the AI software’s prediction.

Difference between explainability and accuracy

Explainable AI models are easy to understand but might work as well as expected because they are simple. Most often accurate models work well because they are complicated but aren’t explainable. The biggest issue with explainable AI is whether it can accurately complete the task that it is designed for. So then, where do we draw the line? That depends on the field where the algorithm is being applied and the end-user to whom it will be accountable.


Source: ResearchGate

If the end-user is a technical user who is well acquainted with AI and has complete trust in it, it is preferable to have accurate models over explainable AI models. However, if the end-user is a non-technical user, it is better to choose explainability over accuracy else there is the risk of losing their trust. Alternatively, the requirements of certain industries are different from others. For example, the approach to AI for industries such as banks and insurance companies will be different because they are more prone to legal and ethical rules and laws. This limits their use of Blackbox models.

Closing thoughts

So far, there is only early research and work being done in making deep learning approaches explainable. However, there will be significant progress so that organizations and users can enjoy both accuracy and explainability together. The levels of accuracy and explainability should be determined based on the consequences and outcomes that are derived from the AI platform. Simultaneously, organizations should have enough governance over the AI operations.

Subscribe to our newsletter

whitepaper download

Want to know more about AI but don’t know where to start?

Get a free copy of our whitepaper.

Related articles


Responsible AI

Responsible AI is a framework that registers how an organization is addressing the challenges around artificial intelligence (AI) from an ethical and legal perspective.

Read More »

AI platform for the world’s
data-driven companies